RE: kafka as recovery only source

2018-02-07 Thread Sofer, Tovi
: user@flink.apache.org; Tzu-Li (Gordon) Tai <tzuli...@apache.org> Subject: Re: kafka as recovery only source Hi Tovi, I've been thinking about this idea. It might be possible, but I think you have to implement a custom source for this. I don't think it would work to have the JMSConsume

Re: kafka as recovery only source

2018-02-07 Thread Fabian Hueske
Hi Tovi, I've been thinking about this idea. It might be possible, but I think you have to implement a custom source for this. I don't think it would work to have the JMSConsumer, KafkaSink, and RecoverySource in separate operators because otherwise it would not be possible to share the Kafka

kafka as recovery only source

2018-02-06 Thread Sofer, Tovi
Hi group, I wanted to get your suggestion on how to implement two requirements we have: * One is to read from external message queue (JMS) at very fast latency * Second is to support zero data loss, so that in case of restart and recovery, messages not checkpointed (and not