Ayan,
Did you get to work the HBase Connection through
Pyspark as well ? I have got the Spark - HBase connection working with
Scala (via HBasecontext). However, but I eventually want to get this
working within a Pyspark code - Would you have some suitable code snippets
or approach so that I can call a Scala class within Pyspark ?
Thanks,
Debu
On Wed, Jun 28, 2017 at 3:18 PM, ayan guha <guha.a...@gmail.com> wrote:
> Hi
>
> Thanks for all of you, I could get HBase connector working. there are
> still some details around namespace is pending, but overall it is working
> well.
>
> Now, as usual, I would like to use the same concept into Structured
> Streaming. Is there any similar way I can use writeStream.format and use
> HBase writer? Or any other way to write continuous data to HBase?
>
> best
> Ayan
>
> On Tue, Jun 27, 2017 at 2:15 AM, Weiqing Yang <yangweiqing...@gmail.com>
> wrote:
>
>> For SHC documentation, please refer the README in SHC github, which is
>> kept up-to-date.
>>
>> On Mon, Jun 26, 2017 at 5:46 AM, ayan guha <guha.a...@gmail.com> wrote:
>>
>>> Thanks all, I have found correct version of the package. Probably HDP
>>> documentation is little behind.
>>>
>>> Best
>>> Ayan
>>>
>>> On Mon, 26 Jun 2017 at 2:16 pm, Mahesh Sawaiker <
>>> mahesh_sawai...@persistent.com> wrote:
>>>
>>>> Ayan,
>>>>
>>>> The location of the logging class was moved from Spark 1.6 to Spark 2.0.
>>>>
>>>> Looks like you are trying to run 1.6 code on 2.0, I have ported some
>>>> code like this before and if you have access to the code you can recompile
>>>> it by changing reference to Logging class and directly use the slf4 Logger
>>>> class, most of the code tends to be easily portable.
>>>>
>>>>
>>>>
>>>> Following is the release note for Spark 2.0
>>>>
>>>>
>>>>
>>>> *Removals, Behavior Changes and Deprecations*
>>>>
>>>> *Removals*
>>>>
>>>> The following features have been removed in Spark 2.0:
>>>>
>>>> - Bagel
>>>> - Support for Hadoop 2.1 and earlier
>>>> - The ability to configure closure serializer
>>>> - HTTPBroadcast
>>>> - TTL-based metadata cleaning
>>>> - *Semi-private class org.apache.spark.Logging. We suggest you use
>>>> slf4j directly.*
>>>> - SparkContext.metricsSystem
>>>>
>>>> Thanks,
>>>>
>>>> Mahesh
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* ayan guha [mailto:guha.a...@gmail.com]
>>>> *Sent:* Monday, June 26, 2017 6:26 AM
>>>> *To:* Weiqing Yang
>>>> *Cc:* user
>>>> *Subject:* Re: HDP 2.5 - Python - Spark-On-Hbase
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> I am using following:
>>>>
>>>>
>>>>
>>>> --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories
>>>> http://repo.hortonworks.com/content/groups/public/
>>>>
>>>>
>>>>
>>>> Is it compatible with Spark 2.X? I would like to use it....
>>>>
>>>>
>>>>
>>>> Best
>>>>
>>>> Ayan
>>>>
>>>>
>>>>
>>>> On Sat, Jun 24, 2017 at 2:09 AM, Weiqing Yang <yangweiqing...@gmail.com>
>>>> wrote:
>>>>
>>>> Yes.
>>>>
>>>> What SHC version you were using?
>>>>
>>>> If hitting any issues, you can post them in SHC github issues. There
>>>> are some threads about this.
>>>>
>>>>
>>>>
>>>> On Fri, Jun 23, 2017 at 5:46 AM, ayan guha <guha.a...@gmail.com> wrote:
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> Is it possible to use SHC from Hortonworks with pyspark? If so, any
>>>> working code sample available?
>>>>
>>>>
>>>>
>>>> Also, I faced an issue while running the samples with Spark 2.0
>>>>
>>>>
>>>>
>>>> "Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging"
>>>>
>>>>
>>>>
>>>> Any workaround?
>>>>
>>>>
>>>>
>>>> Thanks in advance....
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Best Regards,
>>>> Ayan Guha
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Best Regards,
>>>> Ayan Guha
>>>> DISCLAIMER
>>>> ==========
>>>> This e-mail may contain privileged and confidential information which
>>>> is the property of Persistent Systems Ltd. It is intended only for the use
>>>> of the individual or entity to which it is addressed. If you are not the
>>>> intended recipient, you are not authorized to read, retain, copy, print,
>>>> distribute or use this message. If you have received this communication in
>>>> error, please notify the sender and delete all copies of this message.
>>>> Persistent Systems Ltd. does not accept any liability for virus infected
>>>> mails.
>>>>
>>> --
>>> Best Regards,
>>> Ayan Guha
>>>
>>
>>
>
>
> --
> Best Regards,
> Ayan Guha
>
