Re: Accumulating data in Kafka Connect source tasks

2016-01-29 Thread Randall Hauch
On January 28, 2016 at 7:07:02 PM, Ewen Cheslack-Postava (e...@confluent.io) wrote: Randall,  Great question. Ideally you wouldn't need this type of state since it  should really be available in the source system. In your case, it might  actually make sense to be able to grab that information

Re: Accumulating data in Kafka Connect source tasks

2016-01-29 Thread Ewen Cheslack-Postava
On Fri, Jan 29, 2016 at 7:06 AM, Randall Hauch wrote: > On January 28, 2016 at 7:07:02 PM, Ewen Cheslack-Postava ( > e...@confluent.io) wrote: > > Randall, > > Great question. Ideally you wouldn't need this type of state since it > should really be available in the source

Re: Accumulating data in Kafka Connect source tasks

2016-01-29 Thread James Cheng
> On Jan 29, 2016, at 7:06 AM, Randall Hauch wrote: > > On January 28, 2016 at 7:07:02 PM, Ewen Cheslack-Postava (e...@confluent.io) > wrote: > Randall, > > Great question. Ideally you wouldn't need this type of state since it > should really be available in the source system.

Re: Accumulating data in Kafka Connect source tasks

2016-01-28 Thread James Cheng
> On Jan 28, 2016, at 5:06 PM, Ewen Cheslack-Postava wrote: > > Randall, > > Great question. Ideally you wouldn't need this type of state since it > should really be available in the source system. In your case, it might > actually make sense to be able to grab that

Re: Accumulating data in Kafka Connect source tasks

2016-01-28 Thread Ewen Cheslack-Postava
Randall, Great question. Ideally you wouldn't need this type of state since it should really be available in the source system. In your case, it might actually make sense to be able to grab that information from the DB itself, although that will also have issues if, for example, there have been

Re: Accumulating data in Kafka Connect source tasks

2016-01-28 Thread Randall Hauch
Rather than leave this thread so open ended, perhaps I can narrow down to what I think is the best approach. These accumulations are really just additional information from the source that don’t get written to the normal topics. Instead, each change to the accumulated state can be emitted as

Accumulating data in Kafka Connect source tasks

2016-01-27 Thread Randall Hauch
I’m creating a custom Kafka Connect source connector, and I’m running into a situation for which Kafka Connect doesn’t seem to provide a solution out of the box. I thought I’d first post to the users list in case I’m just missing a feature that’s already there. My connector’s SourceTask