Hi Muhammad, Well, I don’t think the version diffs of Flink have such an impact. Do you have more fine-grained evaluation figures, e.g. whats the throughput of the SP Connect Kafka adapter?
The Connect Kafka adapter is the first entry point into SP and thus determines the max throughput after all. Similarly for every other subsequent processor up to the sink. In addition, de/serialization is always quite expensive, especially as we do this in every intermittent processor step as we consume/produce events from/to Kafka. Patrick > Am 21.06.2021 um 11:06 schrieb Muhammad Faizan <m.fai...@eu.denso.com>: > > Hi Dominik, > > The events were thingified JSON in both cases and I was using the default > Kafka batch size. The configurations were quite the same, only the difference > was Flink version: StreamPipes works with Flink 1.9.1 and my other setup was > with Flink 1.11.1. > > > Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu > > i.A. Muhammad Faizan > Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH > Phone: +49 8165 944 201 / NiceNet: 5033-201 / m.fai...@eu.denso.com / > www.denso.com > > > > -----Original Message----- > From: Dominik Riemer <rie...@apache.org> > Sent: Friday, June 18, 2021 9:43 PM > To: dev@streampipes.apache.org > Subject: Re: Performance Issues > > Hi Muhammad, > > thanks for reporting this - this seems to be quite a downgrade and we should > investigate this. > I haven't used the flink wrapper for a while now but will look into this as > soon as possible. > > Were both tests similar in terms of their configuration? Which Kafka settings > did you use in your standalone setting (e.g., batch size) and how did you > serialize events in this setting? > > Dominik > > > On 2021/06/17 10:18:25, Muhammad Faizan <m.fai...@eu.denso.com> wrote: >> Hi, >> >> I have been evaluating performance comparison of pipelines created through >> StreamPipes and also manually with Flink and Kafka. I have noticed a weird >> performance down-grade in case of StreamPipes. >> >> I have two setups: >> >> 1. Pipeline created manually using Flink and Kafka (without StreamPipes) >> 2. Pipeline created using StreamPipes. >> * Main data source is added to StreamPipes using Kafka Adaptor >> * Using this data source in StreamPipes pipeline definition. >> >> The problem is that in case of StreamPipes created pipeline, the performance >> (number of events consumed per second) in Flink is very low (near to 500 >> events / sec), but in case of manual Setup 1, the Flink is able to consume >> ~10,000 events / sec from Kafka. See image below: >> >> [cid:image003.jpg@01D76372.EED22510] >> >> The pipeline in StreamPipes is: >> [Diagram Description automatically generated] >> >> >> In case of StreamPipes, I see that when a new data source (Kafka Adapter) is >> connected, it replicates the events to a new Kafka Topic which is then used >> by the created pipeline. Can that be a reason for performance down-grade? Or >> do you thing something is wrong with my setup or missing anything? >> >> >> Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu >> >> [cid:image005.png@01D76372.EED22510] >> i.A. Muhammad Faizan >> Student >> Corporate R&D >> DENSO AUTOMOTIVE Deutschland GmbH >> >> Phone: +49 8165 944 201 / NiceNet: 5033-201 / >> m.fai...@eu.denso.com<mailto:m.fai...@eu.denso.com> / >> www.denso.com<http://www.denso.com/> >> Located at: Eching Office, Freisinger Str. 21-23, 85386 Eching, >> Bayern, Germany >> Geschäftsführer: Yuji Ishizuka, Yoshio Nakano, Kazuoki Matsugatani, >> Taro Tabata Registergericht München: HR B 72576, VAT-ID No.: DE >> 129426275, St.-Nr. 115/124/30084 >> >> This e-mail message is intended only for the use of the named >> recipient(s). The information contained therein may be confidential or >> privileged, and its disclosure or reproduction is strictly prohibited. >> If you are not the named recipient, please return it immediately to its >> sender at the above address and destroy it. Environmental preservation and >> harmony with society is one of DENSO's four management principles. >> Please consider the environment before printing this e-mail. >>
