To tie-off this thread, this nice feature was just pushed on hbase-connector. See https://github.com/apache/hbase-connectors/tree/master/kafka for how-to. Review and commentary welcome.
Thanks, S On Fri, Aug 3, 2018 at 6:32 AM Hbase Janitor <[email protected]> wrote: > I opened hbase-21002 to start the scripts and assembly. > > Mike > > On Thu, Aug 2, 2018, 19:29 Stack <[email protected]> wrote: > > > Up in https://issues.apache.org/jira/browse/HBASE-20934 I created an > > hbase-connectors repo. I put some form on it using the v19 patch from > > HBASE-15320 "HBase connector for Kafka Connect". It builds and tests > > pass. Here are some remaining TODOs: > > > > * Figure how to do start scripts: e.g. we need to start up the kafka > > proxy. It wants some hbase jars, conf dir, and others on the CLASSPATH > > (Depend on an HBASE_HOME and then source bin/hbase?) > > * Can any of the connectors make-do with the shaded client? > > * Make connectors standalone or have them share conf, bin, etc? > > * Need to do an assembly. Not done. > > * Move over REST and thrift next. Mapreduce after? > > > > The poms could do w/ a review. Hacked them over from hbase-thirdparty. > > > > File issues and apply patches up in JIRA if your up for any of the above. > > > > Thanks, > > S > > > > On Wed, Jul 25, 2018 at 10:46 PM Stack <[email protected]> wrote: > > > > > > > > > On Tue, Jul 24, 2018 at 10:01 PM Misty Linville <[email protected]> > > wrote: > > >> > > >> I like the idea of a separate connectors repo/release vehicle, but > I'm a > > >> little concerned about the need to release all together to update just > > one > > >> of the connectors. How would that work? What kind of compatibility > > >> guarantees are we signing up for? > > >> > > > > > > I hate responses that begin "Good question" -- so fawning -- but, ahem, > > good question Misty (in the literal, not flattering, sense). > > > > > > I think hbase-connectors will be like hbase-thirdparty. The latter > > includes netty, pb, guava and a few other bits and pieces so yeah, > > sometimes a netty upgrade or an improvement on our patch to pb will > require > > us releasing all though we are fixing one lib only. Usually, if bothering > > to make a release, we'll check for fixes or updates we can do in the > other > > bundled components. > > > > > > On the rate of releases, I foresee a flurry of activity around launch > as > > we fill missing bits and address critical bug fixes, but that then it > will > > settle down to be boring, with just the occasional update. Thrift and > REST > > have been stable for a good while now (not saying this is a good thing). > > Our Sean just suggested moving mapreduce to connectors too -- an > > interesting idea -- and this has also been stable too (at least until > > recently with the shading work). We should talk about the Spark connector > > when it comes time. It might not be as stable as the others. > > > > > > On the compatibility guarantees, we'll semver it so if an incompatible > > change in a connector or if the connectors have to change to match a new > > version of hbase, we'll make sure the hbase-connector version number is > > changed appropriately. On the backend, what Mike says; connectors use > HBase > > Public APIs (else they can't be moved to the hbase-connector repo). > > > > > > S > > > > > > > > > > > > > > > > > >> > > >> On Tue, Jul 24, 2018, 9:41 PM Stack <[email protected]> wrote: > > >> > > >> > Grand. I filed https://issues.apache.org/jira/browse/HBASE-20934. > > Let me > > >> > have a go at making the easy one work first (the kafka proxy). Lets > > see how > > >> > it goes. I'll report back here. > > >> > S > > >> > > > >> > On Tue, Jul 24, 2018 at 2:43 PM Sean Busbey <[email protected]> > > wrote: > > >> > > > >> > > Key functionality for the project's adoption should be in the > > project. > > >> > > Please do not suggest we donate things to Bahir. > > >> > > > > >> > > I apologize if this is brisk. I have had previous negative > > experiences > > >> > > with folks that span our communities trying to move work I spent a > > lot > > >> > > of time contributing to within HBase over to Bahir in an attempt > to > > >> > > bypass an agreed upon standard of quality. > > >> > > > > >> > > On Tue, Jul 24, 2018 at 3:38 PM, Artem Ervits < > > [email protected]> > > >> > > wrote: > > >> > > > Why not just donating the connector to http://bahir.apache.org/ > ? > > >> > > > > > >> > > > On Tue, Jul 24, 2018, 12:51 PM Lars Francke < > > [email protected]> > > >> > > wrote: > > >> > > > > > >> > > >> I'd love to have the Kafka Connector included. > > >> > > >> > > >> > > >> @Mike thanks so much for the contribution (and your planned > ones) > > >> > > >> > > >> > > >> I'm +1 on adding it to the core but I'm also +1 on having a > > separate > > >> > > >> repository under Apache governance > > >> > > >> > > >> > > >> On Tue, Jul 24, 2018 at 6:01 PM, Josh Elser <[email protected] > > > > >> > wrote: > > >> > > >> > > >> > > >> > +1 to the great point by Duo about use of non-IA.Public > classes > > >> > > >> > > > >> > > >> > +1 for Apache for the governance (although, I wouldn't care > if > > we > > >> > use > > >> > > >> > Github PRs to try to encourage more folks to contribute), a > > repo > > >> > with > > >> > > the > > >> > > >> > theme of "connectors" (to include Thrift, REST, and the > like). > > Spark > > >> > > too > > >> > > >> -- > > >> > > >> > I think we had suggested that prior, but it could be a mental > > >> > > invention > > >> > > >> of > > >> > > >> > mine.. > > >> > > >> > > > >> > > >> > > > >> > > >> > On 7/24/18 10:16 AM, Hbase Janitor wrote: > > >> > > >> > > > >> > > >> >> Hi everyone, > > >> > > >> >> > > >> > > >> >> I'm the author of the patch. A separate repo for all the > > >> > connectors > > >> > > is > > >> > > >> a > > >> > > >> >> great idea! I can make whatever changes necessary to the > > patch to > > >> > > help. > > >> > > >> >> > > >> > > >> >> I have several other integration type projects like this > > planned. > > >> > > >> >> > > >> > > >> >> Mike > > >> > > >> >> > > >> > > >> >> > > >> > > >> >> On Tue, Jul 24, 2018, 00:03 Mike Drob <[email protected]> > > wrote: > > >> > > >> >> > > >> > > >> >> I would be ok with all of the connectors in a single repo. > > Doing a > > >> > > repo > > >> > > >> >>> per > > >> > > >> >>> connector seems like a large amount of overhead work. > > >> > > >> >>> > > >> > > >> >>> On Mon, Jul 23, 2018, 9:12 PM Clay B. <[email protected]> > wrote: > > >> > > >> >>> > > >> > > >> >>> [Non-binding] > > >> > > >> >>>> > > >> > > >> >>>> I am all for the Kafka Connect(er) as indeed it makes > HBase > > "more > > >> > > >> >>>> relevant" and generates buzz to help me sell HBase > adoption > > in my > > >> > > >> >>>> endeavors. > > >> > > >> >>>> > > >> > > >> >>>> Also, I would like to see a connectors repo a lot as I > would > > >> > > expect it > > >> > > >> >>>> > > >> > > >> >>> can > > >> > > >> >>> > > >> > > >> >>>> make the HBase source and releases more obvious in what is > > >> > > changing. > > >> > > >> Not > > >> > > >> >>>> to distract from Kafka, but Spark has in the past been a > > hang-up > > >> > > and > > >> > > >> >>>> > > >> > > >> >>> seems > > >> > > >> >>> > > >> > > >> >>>> a good fit in such a repo too; as such, I would prefer > > Apache > > >> > over > > >> > > >> >>>> > > >> > > >> >>> GitHub. > > >> > > >> >>> > > >> > > >> >>>> > > >> > > >> >>>> -Clay > > >> > > >> >>>> > > >> > > >> >>>> On Mon, 23 Jul 2018, Andrew Purtell wrote: > > >> > > >> >>>> > > >> > > >> >>>> Would we make a new repo called hbase-connectors and move > > REST, > > >> > > >> >>>>>> > > >> > > >> >>>>> thrift, > > >> > > >> >>> > > >> > > >> >>>> and this new patch there? > > >> > > >> >>>>> > > >> > > >> >>>>> I like this idea. We are already releasing > > hbase-thirdparty like > > >> > > >> this. > > >> > > >> >>>>> > > >> > > >> >>>>> > > >> > > >> >>>>> On Mon, Jul 23, 2018 at 5:47 PM Stack <[email protected]> > > wrote: > > >> > > >> >>>>> > > >> > > >> >>>>> (Thanks for the good discussion) > > >> > > >> >>>>>> > > >> > > >> >>>>>> Where we think 'outside of HBase' would be? > > >> > > >> >>>>>> > > >> > > >> >>>>>> Github seems too 'remote' from project and from Apache? > > Would > > >> > we > > >> > > >> make > > >> > > >> >>>>>> > > >> > > >> >>>>> a > > >> > > >> >>> > > >> > > >> >>>> new > > >> > > >> >>>> > > >> > > >> >>>>> repo called hbase-connectors and move REST, thrift, and > > this new > > >> > > >> patch > > >> > > >> >>>>>> there? > > >> > > >> >>>>>> > > >> > > >> >>>>>> Thanks, > > >> > > >> >>>>>> S > > >> > > >> >>>>>> > > >> > > >> >>>>>> On Mon, Jul 23, 2018 at 3:50 PM Josh Elser < > > [email protected]> > > >> > > >> wrote: > > >> > > >> >>>>>> > > >> > > >> >>>>>> I'm -0 for including this into the main hbase tree. I > > feel like > > >> > > >> we've > > >> > > >> >>>>>>> made a bit of progress in cleaning up our core, and > this > > >> > > strikes me > > >> > > >> >>>>>>> > > >> > > >> >>>>>> as > > >> > > >> >>> > > >> > > >> >>>> a > > >> > > >> >>>> > > >> > > >> >>>>> step in the wrong direction. > > >> > > >> >>>>>>> > > >> > > >> >>>>>>> At the same time, the integration seems nice enough > (for > > the > > >> > > same > > >> > > >> >>>>>>> reasons Andrew points out). Is there a reason this > > couldn't > > >> > > exist > > >> > > >> >>>>>>> outside of HBase (at the ASF or otherwise)? Given a > quick > > >> > > glance at > > >> > > >> >>>>>>> > > >> > > >> >>>>>> the > > >> > > >> >>> > > >> > > >> >>>> patch, it would be quite trivial to keep separate (just > > requires > > >> > > some > > >> > > >> >>>>>>> heavier scripting to get it off the ground that the > HBase > > >> > > scripts > > >> > > >> do > > >> > > >> >>>>>>> setup for). I feel like that will decrease our debt > > while we > > >> > > see if > > >> > > >> >>>>>>> people start using it. Our API should be more than > stable > > >> > > enough to > > >> > > >> >>>>>>> prevent any worry about drift happening from core to > this > > >> > > project. > > >> > > >> >>>>>>> > > >> > > >> >>>>>>> On 7/23/18 6:35 PM, Stack wrote: > > >> > > >> >>>>>>> > > >> > > >> >>>>>>>> We have a very nice contrib sitting up in HBASE-15320 > > which > > >> > > via a > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>> proxy > > >> > > >> >>>> > > >> > > >> >>>>> -- > > >> > > >> >>>>>>> > > >> > > >> >>>>>>>> so minimal dependencies -- adds source and sink for > > Kafka > > >> > > Connect. > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>> It > > >> > > >> >>> > > >> > > >> >>>> is > > >> > > >> >>>>>> > > >> > > >> >>>>>>> nicely contained inside two new hbase-kafka-* modules. > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>>> We good w/ taking on this new feature? > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>>> It looks good to me. Check it out up on HBASE-15320. I > > was > > >> > > going > > >> > > >> to > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>> commit > > >> > > >> >>>>>>> > > >> > > >> >>>>>>>> to tip of branch-2 so it'd show up in hbase-2.2.x > > unless you > > >> > > all > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>> want > > >> > > >> >>> > > >> > > >> >>>> some > > >> > > >> >>>>>>> > > >> > > >> >>>>>>>> backporting action going on. > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>>> S > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>>> > > >> > > >> >>>>>>> > > >> > > >> >>>>>> > > >> > > >> >>>>> > > >> > > >> >>>>> -- > > >> > > >> >>>>> Best regards, > > >> > > >> >>>>> Andrew > > >> > > >> >>>>> > > >> > > >> >>>>> Words like orphans lost among the crosstalk, meaning torn > > from > > >> > > >> truth's > > >> > > >> >>>>> decrepit hands > > >> > > >> >>>>> - A23, Crosstalk > > >> > > >> >>>>> > > >> > > >> >>>>> > > >> > > >> >>>> > > >> > > >> >>> > > >> > > >> >>> > > >> > > >> >> > > >> > > >> > > >> > > > > >> > > > >
