Kafka connect FieldPartitioner with scheduled rotation

2019-02-03 Thread Manu Jacob
Hi, I want to use s3 connect by portioning with FieldPartitioner and partition.field.name set to a non timestamp based field. I want to commit and flush based on both size and time. I am getting an exception when I use the option "rotate.schedule.interval.ms". Is it possible to rotate it with

Statestore restoration & scaling questions - possible KIP as well.

2019-02-03 Thread Adam Bellemare
Hey Folks I have a few questions around the operations of stateful processing while scaling nodes up/down, and a possible KIP in question #4. Most of them have to do with task processing during rebuilding of state stores after scaling nodes up. Scenario: Single node/thread, processing 2 topics (1