Re: StreamingFileSink rolling callback Inbox

2019-09-12 Thread Kostas Kloudas
Hi Anton, First of all, there is this PR https://github.com/apache/flink/pull/9581 that may be interesting to you. Second, I think you have to keep in mind that the hourly bucket reporting will be per-subtask. So if you have parallelism of 4, each of the 4 tasks will report individually that

StreamingFileSink rolling callback Inbox

2019-09-09 Thread Anton Parkhomenko
Hello, I’m writing a Flink job that reads heterogenius (one row contains several types that need to be partitioned downstream) data from AWS Kinesis and writes to S3 directory structure like s3://bucket/year/month/day/hour/type, this all works great with StreamingFileSink in Flink 1.9, but