Hi Jason, I'd suggest to start with [1] and [2] for getting the basics of a Flink program. The DataStream API basically wires operators together with streams so that whatever stream gets out of one operator is the input of the next. By connecting both functions to the same Kafka stream source, your program results in this:
Kafka --> Function1
|
--> Function2
where both functions receive all elements the previous stream offers
(elements are broadcasted). If you want the two functions to work on
different elements, you could add a filter before each function:
DataStream<SomeObject> inputStream = ... got a kafka stream
inputStream.filter(...).process( Function1 );
inputStream.filter(...).process( Function2 );
or split the stream (see [3] for available operators).
I'm no expert on Kafka though, so I can't give you an advise on the
performance - I'd suggest to create some small benchmarks for your setup
since this probably depends on the cluster architecture and the
parallelism of the operators and the number of Kafka partitions.
Maybe Gordon (cc'd) can give some more insights.
Regards
Nico
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/concepts/programming-model.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/index.html
On 12/01/18 21:57, Jason Kania wrote:
> Hi,
>
> I have a question that I have not resolved via the documentation,
> looking in the "Parallel Execution", "Streaming" and the "Connectors"
> sections. If I retrieve a kafka stream and then call the process
> function against it in parallel, as follows, does it consume in some
> round robin fashion between the two process calls or is each element
> coming out of the kafka connector consumed by both processors in parallel?
>
> DataStream<SomeObject> inputStream = ... got a kafka stream
> inputStream.process( Function1 );
> inputStream.process( Function2 );
>
> If it possible to consume in parallel by pointing at the single stream,
> is it typically slower or faster than having two kafka streams with
> different group ids?
>
> If not documented elsewhere, this would be good to cover since it is
> fundamental.
>
> Thanks,
>
> Jason
signature.asc
Description: OpenPGP digital signature
