Agree with avoid unmanaged jars and publishing to maven cental. As far as I remember, we applied custom patch to control rpc time per request, but I guess Asynchbase 1.7.1 also support it(not sure though). Let me check if we are still rely on custom patch. if we don't need custom patch, I think we should go with https://github.com/jongwook/asynchbase-shaded.
Thanks for your works, Jong Wook, I will update this after check. On Mon, May 2, 2016 at 1:18 PM Jong Wook Kim <[email protected]> wrote: > So I went ahead and made an asynchbase package that shades Google Guava, > and thought it is a good chance to: > > - avoid pulling duplicate Netty from two organizations - io.netty and > org.jboss.netty. > - remove log4j-over-slf4j from the runtime dependency: this assumes that > the application is not using log4j in favor of slf4j, and using something > else like logback. Most Apache hadoop/spark environment unfortunately uses > log4j, and due to its distributed nature it's not easy to switch. Better > not enforce anything on logging implementation as a library and let the > application decide. > > I first started off making a Gradle project that pulls the official > Asynchbase 1.7.1 and relocates packages: > https://github.com/jongwook/asynchbase-shaded > > But then I realized that s2graph was using a custom version of Asynchbase > for RPC timeout, etc., so I ended up making a fork of SteamShon/asynchbase > and used maven-shade-plugin to create the shaded jar: > https://github.com/jongwook/asynchbase > > Running make && make pom.xml && mvn -DskipTests=true install will make a > shaded jar and pom in ~/.m2 provided that protoc 2.5.0 is installed, and > replacing jars at s2core/lib/ with the shaded jar seems to be working well. > > Carrying around unmanaged jars in the repository is not a good idea, so we > need to consider publishing this version of asynchbase to maven central or > to some apache repo. > > Jong Wook > > > > On 21 April 2016 at 10:37, DO YUNG YOON <[email protected]> wrote: > > > Thanks for suggestions Jong Wook. > > > > I think shading would solve version conflict problems. I am not familiar > > with this issue. Jong Wook, can you contribute your knowledge on > shading? I > > think we should make sure discuss on version conflict problems regardless > > native client storage providing. so it would be better to open up > separate > > thread to discuss it. What do you guys think? > > > > As you mentioned, with Native HBase client is all blocking API and What > we > > end up easily would be enclose blocking API with Scala's Future. > > I was wondering if there would be anyone who want to use HBase native > > client rather than asynchbase, but since s2graph public interfaces are > all > > asynchronous, I think there is any benefit to use blocking native client. > > > > So you guys are on not provide native client storage? I am +1 on > providing > > it since it is easy and it is always better to provide more options. > > What others think? > > > > Best Regards > > DOYUNG YOON > > > > On Thu, Apr 21, 2016 at 1:17 PM Hyunsung Jo <[email protected]> > wrote: > > > > > It seems like Elasticsearch had similar problems: > > > https://www.elastic.co/blog/to-shade-or-not-to-shade > > > > > > On Wed, Apr 20, 2016 at 10:36 AM Jong Wook Kim <[email protected]> > wrote: > > > > > > > As per Guava version conflict, we should be able to shade the > > dependency > > > > to another package, maybe with the whole asynchbase together. Guava > > > > versions have been the PITA for many other projects too, and usually > > got > > > > avoided this way. > > > > > > > > If we can avoid the Asynchbase+Guava issue by shading them and the > only > > > > interesting reason left to switch is the benchmark, it might not > worth > > > > going back to the blocking API as it will require a whole new > threading > > > > design. > > > > > > > > Sincerely, > > > > Jong Wook > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > On Apr 19, 2016, at 9:01 PM, DO YUNG YOON <[email protected]> > wrote: > > > > > > > > > > Hi All. > > > > > > > > > > Since implementing storage becomes easier(I believe), I think it is > > > good > > > > to > > > > > have HBaseStroage which use HBase's native client. > > > > > The reason I brought up this is following. > > > > > > > > > > 1. in some environment, especially specific Hadoop and Spark > cluster > > > > > distribution, > > > > > s2core have guava version conflict which comes from asynchbase. > > > > > - Many cases it is necessary to process stream of edges and write > > into > > > > > HBase directly on streaming processing. > > > > > - Currently, there is no way to specify version to avoid above > > > problem. > > > > > With Native HBaseClient, users will be select right version for > there > > > > > environment. > > > > > 2. It would be fun to run benchmark on these two client. > > > > > > > > > > any feedback would be appreciated. > > > > > > > > > > Best Regards. > > > > > DOYUNG YOON > > > > > > > > > >
