If you create a HiveContext before starting StreamingContext, then
`SQLContext.getOrCreate` in foreachRDD will return the HiveContext you
created. You can just call asInstanceOf[HiveContext] to convert it to
HiveContext.
On Tue, Nov 22, 2016 at 8:25 AM, Dirceu Semighini Filho <
Hi Koert,
Certainly it's not a good idea, I was trying to use SQLContext.getOrCreate
but it will return a SQLContext and not a HiveContext.
As I'm using a checkpoint, whenever I start the context by reading the
checkpoint it didn't create my hive context, unless I create it foreach
microbach.
I
you are creating a new hive context per microbatch? is that a good idea?
On Tue, Nov 22, 2016 at 8:51 AM, Dirceu Semighini Filho <
dirceu.semigh...@gmail.com> wrote:
> Has anybody seen this behavior (see tha attached picture) in Spark
> Streaming?
> It started to happen here after I changed the
Has anybody seen this behavior (see tha attached picture) in Spark
Streaming?
It started to happen here after I changed the HiveContext creation to
stream.foreachRDD {
rdd =>
val hiveContext = new HiveContext(rdd.sparkContext)
}
Is this expected?
Kind Regards,
Dirceu
