Re: Re: How to keep a SQLContext instance alive in a spark streaming application's life cycle?

2015-06-12 Thread Tathagata Das
BTW, in Spark 1.4 announced today, I added SQLContext.getOrCreate. So you dont need to create the singleton yourself. On Wed, Jun 10, 2015 at 3:21 AM, Sergio Jiménez Barrio < drarse.a...@gmail.com> wrote: > Note: CCing user@spark.apache.org > > > First, you must check if the RDD is empty: > >

Fwd: Re: How to keep a SQLContext instance alive in a spark streaming application's life cycle?

2015-06-10 Thread Sergio Jiménez Barrio
Note: CCing user@spark.apache.org First, you must check if the RDD is empty: messages.foreachRDD { rdd => if (!rdd.isEmpty) { }} Now, you can obtain the instance of a SQLContext: val sqlContext = SQLContextSingleton.getInstance(rdd.sparkContext)

Re: How to keep a SQLContext instance alive in a spark streaming application's life cycle?

2015-06-09 Thread drarse
Why? I tried this solution and works fine. El martes, 9 de junio de 2015, codingforfun [via Apache Spark User List] < ml-node+s1001560n23218...@n3.nabble.com> escribió: > Hi drarse, thanks for replying, the way you said use a singleton object > does not work > > > > > 在 2015-06-09 16:24:25,"dra