Watermark on keyed stream
Hi everyone, it seems Flink only supports global watermarks currently which is a problem for my use case. Many sensors send data which might be buffered for days in upstream systems before arriving at the Flink job. The job keys the stream by sensor. If other sensors send values in the meantime, the global watermark is advanced and buffered data that arrives late is dropped. How could the issue be solved? I guess it would be possible to calculate the watermark manually and add it to a wrapper object, but I am not sure how to correctly implement windowing (tumbling window) then. Thank you in advance for any ideas. Regards, Nick
Get stream of rejected data from Elasticsearch6 sink
Hi all, is it possible to further process data that could not be persisted by the Elasticsearch6 sink without breaking checkpointing? As I understand, the onFailure callback can't be used to forward rejected data into a separate stream. I would like to extend the sink if this use case is not covered yet. What would be a reasonable approach that matches Flink's overall architecture? As a new Flink user, my first intuition was to check if it is possible to create a side output from the Elasticsearch6 sink. Does it make sense to support sink side outputs? Thank you for your advice. Regards, Nick
