Re: NullPointerException when starting StreamingContext

Which Scala version / Spark release are you using ? Cheers On Wed, Jun 22, 2016 at 8:20 PM, Sunita Arvind wrote: > Hello Experts, > > I am getting this error repeatedly: > > 16/06/23 03:06:59 ERROR streaming.StreamingContext: Error starting the > context, marking it as

Re: Building Spark 2.X in Intellij

It works well with me. You can try reimport it into intellij. On Thu, Jun 23, 2016 at 10:25 AM, Stephen Boesch wrote: > > Building inside intellij is an ever moving target. Anyone have the magical > procedures to get it going for 2.X? > > There are numerous library references

Re: Building Spark 2.X in Intellij

I had some errors like SqlBaseParser class missing, and figured out I needed to get these classes from SqlBase.g4 using antlr4. It works fine now. On Thu, Jun 23, 2016 at 9:20 AM, Jeff Zhang wrote: > It works well with me. You can try reimport it into intellij. > > On Thu, Jun

Re: Improving performance of a kafka spark streaming app

After a bit of effort I moved from a Spark cluster running 1.5.2, to a Yarn cluster running 1.6.1 jars. I'm still setting the maxRPP. The completed batches are no longer showing the number of events processed in the Streaming UI tab . I'm getting around 4k inserts per second in hbase, but I

Re: Silly question about Yarn client vs Yarn cluster modes...

LOL… I hate YARN, but unfortunately I don’t get to make the call on which tools we’re going to use, I just get paid to make stuff work on the tools provided. ;-) Testing is somewhat problematic. You have to really test at some large enough fraction of scale. Fortunately for this issue (YARN

Re: Silly question about Yarn client vs Yarn cluster modes...

Trying to keep the answer short and simple... On Wed, Jun 22, 2016 at 1:19 PM, Michael Segel wrote: > But this gets to the question… what are the real differences between client > and cluster modes? > What are the pros/cons and use cases where one has advantages over

Re: Silly question about Yarn client vs Yarn cluster modes...

Thanks Marcelo, Sounds like cluster mode is more resilient than the client-mode. Does it also depend on the number of Spark nodes involved in choosing which way to go? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Executors killed in Workers with Error: invalid log directory

Hi there, I have been getting a strange error in spark-1.6.1 The job submitted uses only the executor launched on the Master node while the other workers are idle. When I check the errors from the web ui to investigate on the killed executors I see the error: Error: invalid log directory

<    1   2