Hi Karthik This appears to be a common challenge related to a slow-consuming situation. Those with relevant experience in addressing such matters should be capable of providing assistance.
Thanks and regards, Gowtham S On Fri, 15 Sept 2023 at 23:06, Giannis Polyzos <ipolyzos...@gmail.com> wrote: > Hi Karthick, > > on a high level seems like a data skew issue and some partitions have way > more data than others? > What is the number of your devices? how many messages are you processing? > Most of the things you share above sound like you are looking for > suggestions around load distribution for Kafka. i.e number of partitions, > how to distribute your device data etc. > It would be good to also share what your flink job is doing as I don't see > anything mentioned around that.. are you observing back pressure in the > Flink UI? > > Best > > On Fri, Sep 15, 2023 at 3:46 PM Karthick <ibmkarthickma...@gmail.com> > wrote: > >> Dear Apache Flink Community, >> >> >> >> I am writing to urgently address a critical challenge we've encountered >> in our IoT platform that relies on Apache Kafka and real-time data >> processing. We believe this issue is of paramount importance and may have >> broad implications for the community. >> >> >> >> In our IoT ecosystem, we receive data streams from numerous devices, each >> uniquely identified. To maintain data integrity and ordering, we've >> meticulously configured a Kafka topic with ten partitions, ensuring that >> each device's data is directed to its respective partition based on its >> unique identifier. This architectural choice has proven effective in >> maintaining data order, but it has also unveiled a significant problem: >> >> >> >> *One device's data processing slowness is interfering with other devices' >> data, causing a detrimental ripple effect throughout our system.* >> >> To put it simply, when a single device experiences processing delays, it >> acts as a bottleneck within the Kafka partition, leading to delays in >> processing data from other devices sharing the same partition. This issue >> undermines the efficiency and scalability of our entire data processing >> pipeline. >> >> Additionally, I would like to highlight that we are currently using the >> default partitioner for choosing the partition of each device's data. If >> there are alternative partitioning strategies that can help alleviate this >> problem, we are eager to explore them. >> >> We are in dire need of a high-scalability solution that not only ensures >> each device's data processing is independent but also prevents any >> interference or collisions between devices' data streams. Our primary >> objectives are: >> >> 1. *Isolation and Independence:* We require a strategy that guarantees >> one device's processing speed does not affect other devices in the same >> Kafka partition. In other words, we need a solution that ensures the >> independent processing of each device's data. >> >> >> 2. *Open-Source Implementation:* We are actively seeking pointers to >> open-source implementations or references to working solutions that address >> this specific challenge within the Apache ecosystem or any existing >> projects, libraries, or community-contributed solutions that align with our >> requirements would be immensely valuable. >> >> We recognize that many Apache Flink users face similar issues and may >> have already found innovative ways to tackle them. We implore you to share >> your knowledge and experiences on this matter. Specifically, we are >> interested in: >> >> *- Strategies or architectural patterns that ensure independent >> processing of device data.* >> >> *- Insights into load balancing, scalability, and efficient data >> processing across Kafka partitions.* >> >> *- Any existing open-source projects or implementations that address >> similar challenges.* >> >> >> >> We are confident that your contributions will not only help us resolve >> this critical issue but also assist the broader Apache Flink community >> facing similar obstacles. >> >> >> >> Please respond to this thread with your expertise, solutions, or any >> relevant resources. Your support will be invaluable to our team and the >> entire Apache Flink community. >> >> Thank you for your prompt attention to this matter. >> >> >> Thanks & Regards >> >> Karthick. >> >
