Hi folks,
During the review of PR #8593, @reiabreu raised an important point
regarding how Storm handles inbound traffic at the executor level.
Currently, both the control plane and the data plane share the same inbound
JCQueue ("receive-queue"). This layout creates a clear conflict under heavy
load: high-volume data plane traffic can saturate the queue, directly
starving or delaying critical control plane signals (like ticks, flushes,
or acker tuples). These two types of traffic have completely different
requirements: the data plane prioritizes throughput and can tolerate
backpressure, while the control plane requires predictable, bounded latency
to keep the topology healthy. Moreover, control signals are often what
allows the system to recover from saturation in the first place, so they
shouldn't be blocked by a data backlog.To address and properly evaluate this, I have opened an issue: https://github.com/apache/storm/issues/8816 As this change touches a core component of the executor architecture, I would like to invite everyone to use this issue as a collective brainstorming space. Ideally, we want a solution that improves responsiveness under load without introducing regression in raw throughput or breaking backward compatibility. I have shared a very basic initial idea there, implementing two separate queues to decouple the planes, but like any approach, it comes with its own pros and cons. If you have alternatives, insights, or suggestions with fewer trade-offs, please jump into the discussion on the GitHub issue. I'd love to get your architectural insights before moving forward with a formal proposal or PR. Thanks, - Gianluca
