Re: WindowFunction to push data from Kafka to S3

2017-01-16 Thread Ufuk Celebi
Can you check the log files of the TaskManagers and JobManager? There is no obvious reason that the collection should not work. On another note: the rolling file sink might be what you are looking for. https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/connectors/filesystem_sink.htm

WindowFunction to push data from Kafka to S3

2017-01-13 Thread Samra Kasim
Hi, I am reading messages off a Kafka Topic and want to process the messages through Flink and save them into S3. It was pointed out to me that stream processing of the Kafka data won't be saved to S3 because S3 doesn't allow data to be appended to a file, so I want to convert the Kafka stream int