[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568608#comment-16568608 ]
stack commented on HBASE-15320: ------------------------------- Took another look at the patch as it looks after landing in hbase-connectors. Some things you might consider going forward Mike: * it starts up a barebones region server that just receives replication events On above, it seems like we do not shutdown core regionserver services... the RS we start up is a full-featured instance. There are switches we could set to make it so it does not start Admin and Client services... just the sink for replication. A bit of a description on how the thing works, what landed eventually, would be sweet as a release note and as addition to README over in hbase-connectors... so folks can easily figure how to get this nice new functionality going. The RS we start catches the replication stream and then forwards to Kafka topics? * it allows you to specify various rules to route the replication messages or drop them The sink RS reads these out of conf dir, right? Thanks. > HBase connector for Kafka Connect > --------------------------------- > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication > Reporter: Andrew Purtell > Assignee: Mike Wingert > Priority: Major > Labels: beginner > Fix For: 3.0.0 > > Attachments: 15320.master.16.patch, 15320.master.16.patch, > HBASE-15320.master.1.patch, HBASE-15320.master.10.patch, > HBASE-15320.master.11.patch, HBASE-15320.master.12.patch, > HBASE-15320.master.14.patch, HBASE-15320.master.15.patch, > HBASE-15320.master.2.patch, HBASE-15320.master.3.patch, > HBASE-15320.master.4.patch, HBASE-15320.master.5.patch, > HBASE-15320.master.6.patch, HBASE-15320.master.7.patch, > HBASE-15320.master.8.patch, HBASE-15320.master.8.patch, > HBASE-15320.master.9.patch, HBASE-15320.pdf, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)