Hi Raghu,
Great work. I have some questions/remarks that I will add in the PR
(especially on the consumers and partition).
Thanks !
Regards
JB
On 03/18/2016 05:55 PM, Raghu Angadi wrote:
Hi Bill,
We have fairly well tested patch for KafkaIO (pr #121
<https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/121>). It
will be merged soon. The example there keeps track of top hashtags in 10
minute sliding window and writes the results to another Kafka topic.
Please try it if you can. It is well tested on Google Cloud Dataflow. I
have not run it using Flink runner.
Raghu.
On Fri, Mar 18, 2016 at 9:46 AM, Kostas Kloudas
<[email protected] <mailto:[email protected]>> wrote:
Hello Bill,
This is a known limitation of the Flink Runner.
There is a JIRA issue for that
https://issues.apache.org/jira/browse/BEAM-127
A wrapper for Flink sinks will come soon and as Beam evolves,
a more Beam-y solution will come as well.
Kostas
On Mar 18, 2016, at 5:23 PM, William McCarthy
<[email protected] <mailto:[email protected]>> wrote:
Hi,
I’m trying to write a proof-of-concept which takes messages from
Kafka, transforms them using Beam on Flink, then pushes the
results onto a different Kafka topic.
I’ve used the KafkaWindowedWordCountExample as a starting point,
and that’s doing the first part of what I want to do, but it
outputs to text files as opposed to Kafka. FlinkKafkaProducer08
looks promising, but I can’t figure out how to plug it into the
pipeline. I was thinking that it would be wrapped with an
UnboundedFlinkSink, or some such, but that doesn’t seem to exist.
Any advice or thoughts on what I’m trying to do?
I’m running the latest incubator-beam (as of last night from
Github), Flink 1.0.0 in cluster mode and Kafka 0.9.0.1, all on
Google Compute Engine (Debian Jessie).
Thanks,
Bill McCarthy
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
