Hi Thanks for all of you, I could get HBase connector working. there are still some details around namespace is pending, but overall it is working well.
Now, as usual, I would like to use the same concept into Structured Streaming. Is there any similar way I can use writeStream.format and use HBase writer? Or any other way to write continuous data to HBase? best Ayan On Tue, Jun 27, 2017 at 2:15 AM, Weiqing Yang <yangweiqing...@gmail.com> wrote: > For SHC documentation, please refer the README in SHC github, which is > kept up-to-date. > > On Mon, Jun 26, 2017 at 5:46 AM, ayan guha <guha.a...@gmail.com> wrote: > >> Thanks all, I have found correct version of the package. Probably HDP >> documentation is little behind. >> >> Best >> Ayan >> >> On Mon, 26 Jun 2017 at 2:16 pm, Mahesh Sawaiker < >> mahesh_sawai...@persistent.com> wrote: >> >>> Ayan, >>> >>> The location of the logging class was moved from Spark 1.6 to Spark 2.0. >>> >>> Looks like you are trying to run 1.6 code on 2.0, I have ported some >>> code like this before and if you have access to the code you can recompile >>> it by changing reference to Logging class and directly use the slf4 Logger >>> class, most of the code tends to be easily portable. >>> >>> >>> >>> Following is the release note for Spark 2.0 >>> >>> >>> >>> *Removals, Behavior Changes and Deprecations* >>> >>> *Removals* >>> >>> The following features have been removed in Spark 2.0: >>> >>> - Bagel >>> - Support for Hadoop 2.1 and earlier >>> - The ability to configure closure serializer >>> - HTTPBroadcast >>> - TTL-based metadata cleaning >>> - *Semi-private class org.apache.spark.Logging. We suggest you use >>> slf4j directly.* >>> - SparkContext.metricsSystem >>> >>> Thanks, >>> >>> Mahesh >>> >>> >>> >>> >>> >>> *From:* ayan guha [mailto:guha.a...@gmail.com] >>> *Sent:* Monday, June 26, 2017 6:26 AM >>> *To:* Weiqing Yang >>> *Cc:* user >>> *Subject:* Re: HDP 2.5 - Python - Spark-On-Hbase >>> >>> >>> >>> Hi >>> >>> >>> >>> I am using following: >>> >>> >>> >>> --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories >>> http://repo.hortonworks.com/content/groups/public/ >>> >>> >>> >>> Is it compatible with Spark 2.X? I would like to use it.... >>> >>> >>> >>> Best >>> >>> Ayan >>> >>> >>> >>> On Sat, Jun 24, 2017 at 2:09 AM, Weiqing Yang <yangweiqing...@gmail.com> >>> wrote: >>> >>> Yes. >>> >>> What SHC version you were using? >>> >>> If hitting any issues, you can post them in SHC github issues. There are >>> some threads about this. >>> >>> >>> >>> On Fri, Jun 23, 2017 at 5:46 AM, ayan guha <guha.a...@gmail.com> wrote: >>> >>> Hi >>> >>> >>> >>> Is it possible to use SHC from Hortonworks with pyspark? If so, any >>> working code sample available? >>> >>> >>> >>> Also, I faced an issue while running the samples with Spark 2.0 >>> >>> >>> >>> "Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging" >>> >>> >>> >>> Any workaround? >>> >>> >>> >>> Thanks in advance.... >>> >>> >>> >>> -- >>> >>> Best Regards, >>> Ayan Guha >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Best Regards, >>> Ayan Guha >>> DISCLAIMER >>> ========== >>> This e-mail may contain privileged and confidential information which is >>> the property of Persistent Systems Ltd. It is intended only for the use of >>> the individual or entity to which it is addressed. If you are not the >>> intended recipient, you are not authorized to read, retain, copy, print, >>> distribute or use this message. If you have received this communication in >>> error, please notify the sender and delete all copies of this message. >>> Persistent Systems Ltd. does not accept any liability for virus infected >>> mails. >>> >> -- >> Best Regards, >> Ayan Guha >> > > -- Best Regards, Ayan Guha
