you cannot use Futures between operators as objects are seralialized and
possibly sent through the cluster immediatly. Right now there is no
straight forward way in Flink to do async calls. However, there is a
discussion going on which you might wanna join . As far as I know,
the current solution is to create a FlatMap function manually which
manages the async calls and emits .
I hope that helps.
Am 16/09/16 um 13:16 schrieb Albert Gimenez:
I just started learning Flink (using Scala) recently, and I developed a job
that, in short, does this steps:
- Reads json messages from Kafka
- Enriches the messages, reading data from Cassandra (using Phantom
- Puts the enriched messages back to another Kafka topic.
The job looks like this:
.map(MyService.enrichMessage _) // Returns Option
The "enrichMessage" method is where I'm using Phantom DSL to query
Cassandra, and I would like to return a Future, but I can't figure out a way
to do it right now, so I'm using "Await" to force the resolution and return
a result. Is there a way to use a Future here?
I do have a second job that is updating the data in Cassandra, and since I
don't need to sink, I can have my map to return the Futures, and everything
happens asynchronously. I would like to know if it's possible to have a
similar behavior when I want to use a Sink (so, sink to Kafka as the Futures
BTW, I'm using Flink 1.1.2 with Scala 2.11.
Thanks a lot for your help!
This email has been checked for viruses by Avast antivirus software.
Freundliche Grüße / Kind Regards
Follow me: @twalthr