On Fri, Nov 2, 2018 at 7:30 AM Josh Elser <[email protected]> wrote: > Nice stuff, Stack! > > Not me. It Mike's work.
> Two quick questions: > > First, on provenance: this codebase primarily came from Mike Wingert on > https://issues.apache.org/jira/browse/HBASE-15320? Just saw that the > commit came from your email addr -- wasn't sure if that Mike was still > involved (or you took it to completion). > > You talking about the merge done over on the hbase-connectors? Looks like I get blamed for the merge -- if I do a show on the merge commit, it is nothing but the merge note "Merge pull request #3 from hbasejanitor/master" where hbasejanitor is Mike's handle -- but then the merge note is followed by Mikes' work, properly attributed to him. I did not pay close attention to this aspect of how git boxing does it. Seems fine to me. What you think? > Second, I assume this new Git repo had all of the normal email-hooks set > up. Do you know where they are being sent (dev, commit, or issues)? I'm > also assuming that this is a Gitbox repo -- are we OK with pull-requests > to this repo (as well as operator-tools) but still create a Jira issue? > > Yep, gitbox. It has whatever infra set it up as. Back and forth was dumped into HBASE-21002 (We changed this config on hbase-operator-tools config to not do this? I should look). Regards pull requests, etc., email configs., etc., we are in experimental mode around all this stuff trying to figure it out so suggestions/help/exercising the possibilities are all welcome. Thanks, S > - Josh > > On 10/31/18 6:43 PM, Stack wrote: > > To tie-off this thread, this nice feature was just pushed on > > hbase-connector. See > > https://github.com/apache/hbase-connectors/tree/master/kafka for how-to. > > Review and commentary welcome. > > > > Thanks, > > S > > > > On Fri, Aug 3, 2018 at 6:32 AM Hbase Janitor <[email protected]> > wrote: > > > >> I opened hbase-21002 to start the scripts and assembly. > >> > >> Mike > >> > >> On Thu, Aug 2, 2018, 19:29 Stack <[email protected]> wrote: > >> > >>> Up in https://issues.apache.org/jira/browse/HBASE-20934 I created an > >>> hbase-connectors repo. I put some form on it using the v19 patch from > >>> HBASE-15320 "HBase connector for Kafka Connect". It builds and tests > >>> pass. Here are some remaining TODOs: > >>> > >>> * Figure how to do start scripts: e.g. we need to start up the kafka > >>> proxy. It wants some hbase jars, conf dir, and others on the CLASSPATH > >>> (Depend on an HBASE_HOME and then source bin/hbase?) > >>> * Can any of the connectors make-do with the shaded client? > >>> * Make connectors standalone or have them share conf, bin, etc? > >>> * Need to do an assembly. Not done. > >>> * Move over REST and thrift next. Mapreduce after? > >>> > >>> The poms could do w/ a review. Hacked them over from hbase-thirdparty. > >>> > >>> File issues and apply patches up in JIRA if your up for any of the > above. > >>> > >>> Thanks, > >>> S > >>> > >>> On Wed, Jul 25, 2018 at 10:46 PM Stack <[email protected]> wrote: > >>>> > >>>> > >>>> On Tue, Jul 24, 2018 at 10:01 PM Misty Linville <[email protected]> > >>> wrote: > >>>>> > >>>>> I like the idea of a separate connectors repo/release vehicle, but > >> I'm a > >>>>> little concerned about the need to release all together to update > just > >>> one > >>>>> of the connectors. How would that work? What kind of compatibility > >>>>> guarantees are we signing up for? > >>>>> > >>>> > >>>> I hate responses that begin "Good question" -- so fawning -- but, > ahem, > >>> good question Misty (in the literal, not flattering, sense). > >>>> > >>>> I think hbase-connectors will be like hbase-thirdparty. The latter > >>> includes netty, pb, guava and a few other bits and pieces so yeah, > >>> sometimes a netty upgrade or an improvement on our patch to pb will > >> require > >>> us releasing all though we are fixing one lib only. Usually, if > bothering > >>> to make a release, we'll check for fixes or updates we can do in the > >> other > >>> bundled components. > >>>> > >>>> On the rate of releases, I foresee a flurry of activity around launch > >> as > >>> we fill missing bits and address critical bug fixes, but that then it > >> will > >>> settle down to be boring, with just the occasional update. Thrift and > >> REST > >>> have been stable for a good while now (not saying this is a good > thing). > >>> Our Sean just suggested moving mapreduce to connectors too -- an > >>> interesting idea -- and this has also been stable too (at least until > >>> recently with the shading work). We should talk about the Spark > connector > >>> when it comes time. It might not be as stable as the others. > >>>> > >>>> On the compatibility guarantees, we'll semver it so if an incompatible > >>> change in a connector or if the connectors have to change to match a > new > >>> version of hbase, we'll make sure the hbase-connector version number is > >>> changed appropriately. On the backend, what Mike says; connectors use > >> HBase > >>> Public APIs (else they can't be moved to the hbase-connector repo). > >>>> > >>>> S > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>> > >>>>> On Tue, Jul 24, 2018, 9:41 PM Stack <[email protected]> wrote: > >>>>> > >>>>>> Grand. I filed https://issues.apache.org/jira/browse/HBASE-20934. > >>> Let me > >>>>>> have a go at making the easy one work first (the kafka proxy). Lets > >>> see how > >>>>>> it goes. I'll report back here. > >>>>>> S > >>>>>> > >>>>>> On Tue, Jul 24, 2018 at 2:43 PM Sean Busbey <[email protected]> > >>> wrote: > >>>>>> > >>>>>>> Key functionality for the project's adoption should be in the > >>> project. > >>>>>>> Please do not suggest we donate things to Bahir. > >>>>>>> > >>>>>>> I apologize if this is brisk. I have had previous negative > >>> experiences > >>>>>>> with folks that span our communities trying to move work I spent a > >>> lot > >>>>>>> of time contributing to within HBase over to Bahir in an attempt > >> to > >>>>>>> bypass an agreed upon standard of quality. > >>>>>>> > >>>>>>> On Tue, Jul 24, 2018 at 3:38 PM, Artem Ervits < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>> Why not just donating the connector to http://bahir.apache.org/ > >> ? > >>>>>>>> > >>>>>>>> On Tue, Jul 24, 2018, 12:51 PM Lars Francke < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> I'd love to have the Kafka Connector included. > >>>>>>>>> > >>>>>>>>> @Mike thanks so much for the contribution (and your planned > >> ones) > >>>>>>>>> > >>>>>>>>> I'm +1 on adding it to the core but I'm also +1 on having a > >>> separate > >>>>>>>>> repository under Apache governance > >>>>>>>>> > >>>>>>>>> On Tue, Jul 24, 2018 at 6:01 PM, Josh Elser <[email protected] > >>> > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 to the great point by Duo about use of non-IA.Public > >> classes > >>>>>>>>>> > >>>>>>>>>> +1 for Apache for the governance (although, I wouldn't care > >> if > >>> we > >>>>>> use > >>>>>>>>>> Github PRs to try to encourage more folks to contribute), a > >>> repo > >>>>>> with > >>>>>>> the > >>>>>>>>>> theme of "connectors" (to include Thrift, REST, and the > >> like). > >>> Spark > >>>>>>> too > >>>>>>>>> -- > >>>>>>>>>> I think we had suggested that prior, but it could be a mental > >>>>>>> invention > >>>>>>>>> of > >>>>>>>>>> mine.. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 7/24/18 10:16 AM, Hbase Janitor wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi everyone, > >>>>>>>>>>> > >>>>>>>>>>> I'm the author of the patch. A separate repo for all the > >>>>>> connectors > >>>>>>> is > >>>>>>>>> a > >>>>>>>>>>> great idea! I can make whatever changes necessary to the > >>> patch to > >>>>>>> help. > >>>>>>>>>>> > >>>>>>>>>>> I have several other integration type projects like this > >>> planned. > >>>>>>>>>>> > >>>>>>>>>>> Mike > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Jul 24, 2018, 00:03 Mike Drob <[email protected]> > >>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I would be ok with all of the connectors in a single repo. > >>> Doing a > >>>>>>> repo > >>>>>>>>>>>> per > >>>>>>>>>>>> connector seems like a large amount of overhead work. > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Jul 23, 2018, 9:12 PM Clay B. <[email protected]> > >> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> [Non-binding] > >>>>>>>>>>>>> > >>>>>>>>>>>>> I am all for the Kafka Connect(er) as indeed it makes > >> HBase > >>> "more > >>>>>>>>>>>>> relevant" and generates buzz to help me sell HBase > >> adoption > >>> in my > >>>>>>>>>>>>> endeavors. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Also, I would like to see a connectors repo a lot as I > >> would > >>>>>>> expect it > >>>>>>>>>>>>> > >>>>>>>>>>>> can > >>>>>>>>>>>> > >>>>>>>>>>>>> make the HBase source and releases more obvious in what is > >>>>>>> changing. > >>>>>>>>> Not > >>>>>>>>>>>>> to distract from Kafka, but Spark has in the past been a > >>> hang-up > >>>>>>> and > >>>>>>>>>>>>> > >>>>>>>>>>>> seems > >>>>>>>>>>>> > >>>>>>>>>>>>> a good fit in such a repo too; as such, I would prefer > >>> Apache > >>>>>> over > >>>>>>>>>>>>> > >>>>>>>>>>>> GitHub. > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -Clay > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, 23 Jul 2018, Andrew Purtell wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Would we make a new repo called hbase-connectors and move > >>> REST, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> thrift, > >>>>>>>>>>>> > >>>>>>>>>>>>> and this new patch there? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I like this idea. We are already releasing > >>> hbase-thirdparty like > >>>>>>>>> this. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Jul 23, 2018 at 5:47 PM Stack <[email protected]> > >>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> (Thanks for the good discussion) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Where we think 'outside of HBase' would be? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Github seems too 'remote' from project and from Apache? > >>> Would > >>>>>> we > >>>>>>>>> make > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> a > >>>>>>>>>>>> > >>>>>>>>>>>>> new > >>>>>>>>>>>>> > >>>>>>>>>>>>>> repo called hbase-connectors and move REST, thrift, and > >>> this new > >>>>>>>>> patch > >>>>>>>>>>>>>>> there? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> S > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, Jul 23, 2018 at 3:50 PM Josh Elser < > >>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm -0 for including this into the main hbase tree. I > >>> feel like > >>>>>>>>> we've > >>>>>>>>>>>>>>>> made a bit of progress in cleaning up our core, and > >> this > >>>>>>> strikes me > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> as > >>>>>>>>>>>> > >>>>>>>>>>>>> a > >>>>>>>>>>>>> > >>>>>>>>>>>>>> step in the wrong direction. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> At the same time, the integration seems nice enough > >> (for > >>> the > >>>>>>> same > >>>>>>>>>>>>>>>> reasons Andrew points out). Is there a reason this > >>> couldn't > >>>>>>> exist > >>>>>>>>>>>>>>>> outside of HBase (at the ASF or otherwise)? Given a > >> quick > >>>>>>> glance at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> the > >>>>>>>>>>>> > >>>>>>>>>>>>> patch, it would be quite trivial to keep separate (just > >>> requires > >>>>>>> some > >>>>>>>>>>>>>>>> heavier scripting to get it off the ground that the > >> HBase > >>>>>>> scripts > >>>>>>>>> do > >>>>>>>>>>>>>>>> setup for). I feel like that will decrease our debt > >>> while we > >>>>>>> see if > >>>>>>>>>>>>>>>> people start using it. Our API should be more than > >> stable > >>>>>>> enough to > >>>>>>>>>>>>>>>> prevent any worry about drift happening from core to > >> this > >>>>>>> project. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 7/23/18 6:35 PM, Stack wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> We have a very nice contrib sitting up in HBASE-15320 > >>> which > >>>>>>> via a > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> proxy > >>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> so minimal dependencies -- adds source and sink for > >>> Kafka > >>>>>>> Connect. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> It > >>>>>>>>>>>> > >>>>>>>>>>>>> is > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> nicely contained inside two new hbase-kafka-* modules. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> We good w/ taking on this new feature? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> It looks good to me. Check it out up on HBASE-15320. I > >>> was > >>>>>>> going > >>>>>>>>> to > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> commit > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> to tip of branch-2 so it'd show up in hbase-2.2.x > >>> unless you > >>>>>>> all > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> want > >>>>>>>>>>>> > >>>>>>>>>>>>> some > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> backporting action going on. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> S > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Words like orphans lost among the crosstalk, meaning torn > >>> from > >>>>>>>>> truth's > >>>>>>>>>>>>>> decrepit hands > >>>>>>>>>>>>>> - A23, Crosstalk > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>> > >> > > >
