Re: how to ingest a database with a Kafka Connect cluster in parallel?

2017-01-04 Thread Yuanzhe Yang
ntially useful properties (e.g. ordering). To split at this level you'd > have to do something like have each task be responsible for a subset of > rowids in the database. > > -Ewen > > On Tue, Jan 3, 2017 at 1:24 PM, Yuanzhe Yang wrote: > > > Hi Ewen, > > > > Than

Re: how to ingest a database with a Kafka Connect cluster in parallel?

2017-01-03 Thread Yuanzhe Yang
, the connector will only generate a single task. If you add more > tables to the whitelist then you'll see more tasks in the status API > output. > > -Ewen > > On Tue, Jan 3, 2017 at 4:03 AM, Yuanzhe Yang wrote: > > > Hi all, > > > > I am trying t

how to ingest a database with a Kafka Connect cluster in parallel?

2017-01-03 Thread Yuanzhe Yang
Hi all, I am trying to run a Kafka Connect cluster to ingest data from a relational database with jdbc connector. I have been investigating many other solutions including Spark, Flink and Flume before using Kafka Connect, but none of them can be used to ingest relational databases in a clusterabl