Re: problem with hdfs access in spark job

2014-05-18 Thread Marcin Cylke
On Thu, 15 May 2014 09:44:35 -0700 Marcelo Vanzin wrote: > These are actually not worrisome; that's just the HDFS client doing > its own thing to support HA. It probably picked the "wrong" NN to try > first, and got the "NN in standby" exception, which it logs. Then it > tries the other NN and th

Re: problem with hdfs access in spark job

2014-05-16 Thread Marcelo Vanzin
Hi Marcin, On Wed, May 14, 2014 at 7:22 AM, Marcin Cylke wrote: > - This looks like some problems with HA - but I've checked namenodes during > the job was running, and there > was no switch between master and slave namenode. > > 14/05/14 15:25:44 ERROR security.UserGroupInformation: > Priviled

problem with hdfs access in spark job

2014-05-14 Thread Marcin Cylke
Hi I'm running Spark 0.9.1 on hadoop cluster - cdh4.2.1, with YARN. I have a job, that performs a few transformations on a given file and joins that file with some other. The job itself finishes with success, however some tasks are failed and then after rerun succeeds. During the development