Re: What is the meaning to of 'STATE' in a worker/ an executor?

2015-03-29 Thread Mark Hamstra
A LOADING Executor is on the way to RUNNING, but hasn't yet been registered with the Master, so it isn't quite ready to do useful work. > On Mar 29, 2015, at 9:09 PM, Niranda Perera wrote: > > Hi, > > I have noticed in the Spark UI, workers and executors run on several states, > ALIVE, LOAD

What is the meaning to of 'STATE' in a worker/ an executor?

2015-03-29 Thread Niranda Perera
Hi, I have noticed in the Spark UI, workers and executors run on several states, ALIVE, LOADING, RUNNING, DEAD etc? What exactly are these states mean and what is the effect it has on working with those executor? ex: whether an executor can not be used in the loading state, etc cheers -- Niran

Re: Building spark 1.2 from source requires more dependencies

2015-03-29 Thread Sean Owen
Given that's it's an internal error from scalac, I think it may be something to take up with the Scala folks to really fix. We can just look for workarounds. Try blowing away your .m2 and .ivy cache for example. FWIW I was running on Linux with Java 8u31, latest scala 2.11 AFAIK. On Sun, Mar 29, 2

Re: Building spark 1.2 from source requires more dependencies

2015-03-29 Thread Pala M Muthaia
Sean, I did a mvn clean and then build, it produces the same error. I also did a fresh git clone of spark and invoked the same build command and it resulted in identical error (I also had a colleague do a same thing, lest there was some machine specific issue, and saw the same error). Unless i mis

One corrupt gzip in a directory of 100s

2015-03-29 Thread Dale Richardson
Recently had an incident reported to me where somebody was analysing a directory of gzipped log files, and was struggling to load them into spark because one of the files was corrupted - calling sc.textFiles('hdfs:///logs/*.gz') caused an IOException on the particular executor that was reading