Hi Koert,
Certainly it's not a good idea, I was trying to use SQLContext.getOrCreate
but it will return a SQLContext and not a HiveContext.
As I'm using a checkpoint, whenever I start the context by reading the
checkpoint it didn't create my hive context, unless I create it foreach
microbach.
I didn't find a way to use the same hivecontext for all batches.
Does anybody know where can I find how to do this?
2016-11-22 14:17 GMT-02:00 Koert Kuipers <ko...@tresata.com>:
> you are creating a new hive context per microbatch? is that a good idea?
>
> On Tue, Nov 22, 2016 at 8:51 AM, Dirceu Semighini Filho <
> dirceu.semigh...@gmail.com> wrote:
>
>> Has anybody seen this behavior (see tha attached picture) in Spark
>> Streaming?
>> It started to happen here after I changed the HiveContext creation to
>> stream.foreachRDD {
>> rdd =>
>> val hiveContext = new HiveContext(rdd.sparkContext)
>> }
>>
>> Is this expected?
>>
>> Kind Regards,
>> Dirceu
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>
>
