Hi,
I have been evaluating performance comparison of pipelines created through
StreamPipes and also manually with Flink and Kafka. I have noticed a weird
performance down-grade in case of StreamPipes.
I have two setups:
1. Pipeline created manually using Flink and Kafka (without StreamPipes)
2. Pipeline created using StreamPipes.
* Main data source is added to StreamPipes using Kafka Adaptor
* Using this data source in StreamPipes pipeline definition.
The problem is that in case of StreamPipes created pipeline, the performance
(number of events consumed per second) in Flink is very low (near to 500 events
/ sec), but in case of manual Setup 1, the Flink is able to consume ~10,000
events / sec from Kafka. See image below:
[cid:[email protected]]
The pipeline in StreamPipes is:
[Diagram Description automatically generated]
In case of StreamPipes, I see that when a new data source (Kafka Adapter) is
connected, it replicates the events to a new Kafka Topic which is then used by
the created pipeline. Can that be a reason for performance down-grade? Or do
you thing something is wrong with my setup or missing anything?
Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu
[cid:[email protected]]
i.A. Muhammad Faizan
Student
Corporate R&D
DENSO AUTOMOTIVE Deutschland GmbH
Phone: +49 8165 944 201 / NiceNet: 5033-201 /
[email protected]<mailto:[email protected]> /
www.denso.com<http://www.denso.com/>
Located at: Eching Office, Freisinger Str. 21-23, 85386 Eching, Bayern, Germany
Geschäftsführer: Yuji Ishizuka, Yoshio Nakano, Kazuoki Matsugatani, Taro Tabata
Registergericht München: HR B 72576, VAT-ID No.: DE 129426275, St.-Nr.
115/124/30084
This e-mail message is intended only for the use of the named recipient(s). The
information contained
therein may be confidential or privileged, and its disclosure or reproduction
is strictly prohibited. If you
are not the named recipient, please return it immediately to its sender at the
above address and destroy
it. Environmental preservation and harmony with society is one of DENSO's four
management principles.
Please consider the environment before printing this e-mail.
