Re: Long running application failed to init containers due to anthentication errors

So just a hunch because we’ve been dealing with something similar. When the failure occurs, has the resource manager also failed over just recently or in the previous 24 hours? One thing to try: catch this exception and manually fail to the new master/resource manager. - Billy Watson On Thu

Re: Regarding containers not launching

Also, is there anything interesting in the yarn scheduler logs? Something about scheduling being skipped? On Wed, Jan 31, 2018 at 05:16 Billy Watson <williamrwat...@gmail.com> wrote: > Ok, and your container settings? > > On Wed, Jan 31, 2018 at 02:38 nishchay malhotra <

Re: Regarding containers not launching

Ok, and your container settings? On Wed, Jan 31, 2018 at 02:38 nishchay malhotra wrote: > yes my job has about 160,000 maps and my cluster not getting fully > utilized around 6000 maps ran for 2 hrs and then I killed the job. At any > point of time only 40

Re: Regarding containers not launching

Is your job able to use more containers, I.e. does your job have tasks waiting or are all tasks in progress? William Watson On Tue, Jan 30, 2018 at 1:56 AM, nishchay malhotra < nishchay.malht...@gmail.com> wrote: > What should I be looking for if my 24-node cluster in not launching enough >

Re: Install a hadoop cluster manager for open source hadoop 2.7.3

Nishant, Sorry about the late reply. You may want to check out https://ambari.apache.org/mail-lists.html to see if the Ambari user list can answer your question better. William Watson Lead Software Engineer J.D. Power O2O

Re: fs.s3a.endpoint not working

me-bucket/ > hadoop fs -ls s3a://some-bucket/ > hadoop distcp /your/favorite/hdfs/data s3a://some-bucket/ > > HTH, > > -- > Caleb Phillips, Ph.D. > Data Scientist | Computational Science Center > > National Renewable Energy Laboratory (NREL) > 15013 Denver West Parkway

Re: fs.s3a.endpoint not working

Stupid question, I assume you're using a URL that starts with s3a and that your custom endpoint supports s3a? William Watson Lead Software Engineer On Thu, Jan 14, 2016 at 1:57 PM, Alexander Pivovarov wrote: > http://www.jets3t.org/toolkit/configuration.html > On Jan 14,

Re: fs.s3a.endpoint not working

One of the threads suggested using the core-site.xml. Did you try putting your configuration in there? One thing I've noticed is that the AWS stuff is handled by an underlying library (I think jets3t in < 2.6 versions, forget what in 2.6+) and when I was trying to mess with stuff and spelunking

Re: can't submit remote job

Netflix Genie is what we use for submitting jobs. William Watson Software Engineer (904) 705-7056 PCS On Mon, May 18, 2015 at 7:07 AM, xeonmailinglist-gmail xeonmailingl...@gmail.com wrote: I also can't find a good site/book that explains well how to submit remote jobs. Also, can anyone

Re: Jr. to Mid Level Big Data jobs in Bay Area

Uh, it's not about being tolerant. It's the wrong forum for it. There's enough chatter on here and there are 50 job boards all over the internetwebs. Please use the proper forums. William Watson Software Engineer (904) 705-7056 PCS On Sun, May 17, 2015 at 9:14 PM, Juan Suero juan.su...@gmail.com

Re: Unable to Find S3N Filesystem Hadoop 2.6

/ From: Billy Watson williamrwat...@gmail.com Reply-To: user@hadoop.apache.org user@hadoop.apache.org Date: Monday, April 20, 2015 at 11:14 AM To: user@hadoop.apache.org user@hadoop.apache.org Subject: Re: Unable to Find S3N Filesystem Hadoop 2.6 We found the correct configs. This post

Unable to Find S3N Filesystem Hadoop 2.6

) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) — Billy Watson -- William Watson Software Engineer (904) 705-7056 PCS

Re: Unable to Find S3N Filesystem Hadoop 2.6

. --- Regards, Jonathan Aquilina Founder Eagle Eye T On 2015-04-20 15:17, Billy Watson wrote: Hi, I am able to run a `hadoop fs -ls s3n://my-s3-bucket` from the command line without issue. I have set some options in hadoop-env.sh to make sure all the S3 stuff for hadoop 2.6 is set up

Re: Unable to Find S3N Filesystem Hadoop 2.6

On Mon, Apr 20, 2015 at 10:26 AM, Jonathan Aquilina jaquil...@eagleeyet.net wrote: One thing I think which i most likely missed completely is are you using an amazon EMR cluster or something in house? --- Regards, Jonathan Aquilina Founder Eagle Eye T On 2015-04-20 16:21, Billy Watson

Re: Unable to Find S3N Filesystem Hadoop 2.6

/property William Watson Software Engineer (904) 705-7056 PCS On Mon, Apr 20, 2015 at 11:13 AM, Billy Watson williamrwat...@gmail.com wrote: Thanks, anyways. Anyone else run into this issue? William Watson Software Engineer (904) 705-7056 PCS On Mon, Apr 20, 2015 at 11:11 AM, Jonathan

Re: Unable to Find S3N Filesystem Hadoop 2.6

iPhone On 20 Apr 2015, at 16:53, Billy Watson williamrwat...@gmail.com wrote: This is an install on a CentOS 6 virtual machine used in our test environment. We use HDP in staging and production and we discovered these issues while trying to build a new cluster using HDP 2.2 which upgrades from