Re: Flink source error handling

Hi, This consumer should not be used. This only occurs in really old and no longer supported Flink versions. You should really upgrade to a newer version of Flink and use the KafkaSource. Best regards, Martijn On Fri, Jun 9, 2023 at 11:05 AM Anirban Dutta Gupta < anir...@indicussoftware.com> wr

Announcing the Community Over Code 2023 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Halifax, Nova Scotia October 7-10, 2023. The call for presentations is open now through July 13, 2023. I am one of the co-chairs for the stream

Async I/O: preserve stream order for requests on key level

I need to make some slower external requests in parallel, so Async I/O helps greatly with that. However, I also need to make the requests in a certain order per key. Is that possible with Async I/O? The documentation[1] talks about preserving the stream order of results, but it doesn't discuss the

Flink source error handling

Hello, We are using "FlinkKafkaConsumer011" as a Kafka source consumer for Flink. Please guide on how to implement error handling mechanism for the following: 1. If the subscription to the Kafka topic gets lost, Kafka connection gets disconnected. In this case, is there any mechanism of re-sub

RE: External Feedback Loop

You can find the actual code for the previous example here (under Example 2): - https://github.com/ging/fiware-cosmos-orion-flink-connector-examples#example-2-complete-orion-scenario-with-docker-compose Salva On 2023/06/09 07:28:17 Salva Alcántara wrote: > I want to implement a job which feeds s

RE: External Feedback Loop

A related example I've found: https://ngsi-ld-tutorials.readthedocs.io/en/latest/big-data-flink.html#feedback-loop-persisting-context-data Salva On 2023/06/09 07:28:17 Salva Alcántara wrote: > I want to implement a job which feeds sink messages back into a source, > effectively introducing a cyc

RE: External Feedback Loop

Just in case it helps... Attached please find the overall scheme depicted. Salva On 2023/06/09 07:28:17 Salva Alcántara wrote: > I want to implement a job which feeds sink messages back into a source, > effectively introducing a cycle. For simplicity, consider the following > kafka-based enrichm

External Feedback Loop

I want to implement a job which feeds sink messages back into a source, effectively introducing a cycle. For simplicity, consider the following kafka-based enrichment scenario: - A source for the "facts" - A source for the "dimensions" - A sink for the "dimensions" (this creates a loop) - A co-pro