Hi, Btw, now I'm working on upgrading the flink-wrapper version to latest (1.13.1).
Thanks, Xin Patrick Wiener <[email protected]> 于2021年6月22日周二 下午5:36写道： > Hi Muhammad, > > Well, I don’t think the version diffs of Flink have such an impact. Do you > have more fine-grained evaluation figures, e.g. whats the throughput of the > SP Connect Kafka adapter? > > The Connect Kafka adapter is the first entry point into SP and thus > determines the max throughput after all. Similarly for every other > subsequent processor up to the sink. > In addition, de/serialization is always quite expensive, especially as we > do this in every intermittent processor step as we consume/produce events > from/to Kafka. > > Patrick > > > Am 21.06.2021 um 11:06 schrieb Muhammad Faizan <[email protected]>: > > > > Hi Dominik, > > > > The events were thingified JSON in both cases and I was using the > default Kafka batch size. The configurations were quite the same, only the > difference was Flink version: StreamPipes works with Flink 1.9.1 and my > other setup was with Flink 1.11.1. > > > > > > Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu > > > > i.A. Muhammad Faizan > > Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH > > Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected] / > www.denso.com > > > > > > > > -----Original Message----- > > From: Dominik Riemer <[email protected]> > > Sent: Friday, June 18, 2021 9:43 PM > > To: [email protected] > > Subject: Re: Performance Issues > > > > Hi Muhammad, > > > > thanks for reporting this - this seems to be quite a downgrade and we > should investigate this. > > I haven't used the flink wrapper for a while now but will look into this > as soon as possible. > > > > Were both tests similar in terms of their configuration? Which Kafka > settings did you use in your standalone setting (e.g., batch size) and how > did you serialize events in this setting? > > > > Dominik > > > > > > On 2021/06/17 10:18:25, Muhammad Faizan <[email protected]> wrote: > >> Hi, > >> > >> I have been evaluating performance comparison of pipelines created > through StreamPipes and also manually with Flink and Kafka. I have noticed > a weird performance down-grade in case of StreamPipes. > >> > >> I have two setups: > >> > >> 1. Pipeline created manually using Flink and Kafka (without > StreamPipes) > >> 2. Pipeline created using StreamPipes. > >> * Main data source is added to StreamPipes using Kafka Adaptor > >> * Using this data source in StreamPipes pipeline definition. > >> > >> The problem is that in case of StreamPipes created pipeline, the > performance (number of events consumed per second) in Flink is very low > (near to 500 events / sec), but in case of manual Setup 1, the Flink is > able to consume ~10,000 events / sec from Kafka. See image below: > >> > >> [cid:[email protected]] > >> > >> The pipeline in StreamPipes is: > >> [Diagram Description automatically generated] > >> > >> > >> In case of StreamPipes, I see that when a new data source (Kafka > Adapter) is connected, it replicates the events to a new Kafka Topic which > is then used by the created pipeline. Can that be a reason for performance > down-grade? Or do you thing something is wrong with my setup or missing > anything? > >> > >> > >> Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu > >> > >> [cid:[email protected]] > >> i.A. Muhammad Faizan > >> Student > >> Corporate R&D > >> DENSO AUTOMOTIVE Deutschland GmbH > >> > >> Phone: +49 8165 944 201 / NiceNet: 5033-201 / > >> [email protected]<mailto:[email protected]> / > >> www.denso.com<http://www.denso.com/> > >> Located at: Eching Office, Freisinger Str. 21-23, 85386 Eching, > >> Bayern, Germany > >> Geschäftsführer: Yuji Ishizuka, Yoshio Nakano, Kazuoki Matsugatani, > >> Taro Tabata Registergericht München: HR B 72576, VAT-ID No.: DE > >> 129426275, St.-Nr. 115/124/30084 > >> > >> This e-mail message is intended only for the use of the named > >> recipient(s). The information contained therein may be confidential or > >> privileged, and its disclosure or reproduction is strictly prohibited. > >> If you are not the named recipient, please return it immediately to its > sender at the above address and destroy it. Environmental preservation and > harmony with society is one of DENSO's four management principles. > >> Please consider the environment before printing this e-mail. > >> > > -- Thanks, Xin
