If you create a HiveContext before starting StreamingContext, then
`SQLContext.getOrCreate` in foreachRDD will return the HiveContext you
created. You can just call asInstanceOf[HiveContext] to convert it to
HiveContext.
On Tue, Nov 22, 2016 at 8:25 AM, Dirceu Semighini Filho <
dirceu.semigh...@gmail.com> wrote:
> Hi Koert,
> Certainly it's not a good idea, I was trying to use SQLContext.getOrCreate
> but it will return a SQLContext and not a HiveContext.
> As I'm using a checkpoint, whenever I start the context by reading the
> checkpoint it didn't create my hive context, unless I create it foreach
> microbach.
> I didn't find a way to use the same hivecontext for all batches.
> Does anybody know where can I find how to do this?
>
>
>
>
> 2016-11-22 14:17 GMT-02:00 Koert Kuipers <ko...@tresata.com>:
>
>> you are creating a new hive context per microbatch? is that a good idea?
>>
>> On Tue, Nov 22, 2016 at 8:51 AM, Dirceu Semighini Filho <
>> dirceu.semigh...@gmail.com> wrote:
>>
>>> Has anybody seen this behavior (see tha attached picture) in Spark
>>> Streaming?
>>> It started to happen here after I changed the HiveContext creation to
>>> stream.foreachRDD {
>>> rdd =>
>>> val hiveContext = new HiveContext(rdd.sparkContext)
>>> }
>>>
>>> Is this expected?
>>>
>>> Kind Regards,
>>> Dirceu
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>
>>
>
