GGraziadei opened a new issue, #8816:
URL: https://github.com/apache/storm/issues/8816
This is a follow-up extracted from the review discussion on #8593 . During
that review it was pointed out that the control plane and the data plane
currently share the same executor receive queue, and that under heavy load the
producer-side buffering of data tuples makes the control plane less responsive.
This issue tracks that problem on its own so we can discuss and evaluate the
options properly.
## Two different traffic types
A Storm executor multiplexes two fundamentally different kinds of traffic
onto its **single** inbound
`JCQueue` ("receive-queue"):
- **Data plane**: the actual business tuples flowing between components.
High volume, bursty, and the
traffic that backpressure is meant to throttle.
- **Control plane (system streams)**: low-volume, coordination tuples that
keep the topology healthy and responsive.
Storm reserves the `__` prefix for these internal streams. The full set:
| Stream ID | Constant | Defined in | Purpose |
|-----------|----------|------------|---------|
| `__system` | `SYSTEM_STREAM_ID` | `StormCommon.java` | System stream
(startup / coordination). |
| `__tick` | `SYSTEM_TICK_STREAM_ID` | `Constants.java` | Tick tuples,
drive tick-based logic and timeouts. |
| `__flush` | `SYSTEM_FLUSH_STREAM_ID` | `Constants.java` | Flush tuples ,
drive the batch flush. |
| `__metrics` | `METRICS_STREAM_ID` | `Constants.java` | Metrics data
stream. |
| `__metrics_tick` | `METRICS_TICK_STREAM_ID` | `Constants.java` | Metrics
tick, drives metrics collection / telemetry. |
| `__ack_init` | `ACKER_INIT_STREAM_ID` | `Acker.java` | Acker:
anchor/init for a new tuple tree. |
| `__ack_ack` | `ACKER_ACK_STREAM_ID` | `Acker.java` | Acker: ack
acknowledgements. |
| `__ack_fail` | `ACKER_FAIL_STREAM_ID` | `Acker.java` | Acker: failure
notifications. |
| `__ack_reset_timeout` | `ACKER_RESET_TIMEOUT_STREAM_ID` | `Acker.java` |
Acker: reset the tuple-tree timeout. |
| `__heartbeat` | `HEARTBEAT_STREAM_ID` | `ShellBolt.java` |
Multilang/shell component heartbeats. |
| `__eventlog` | `EVENTLOGGER_STREAM_ID` | `StormCommon.java` | Event
logger stream (debug/eventlogging). |
These have opposite requirements from the data plane: the data plane wants
throughput and is *expected* to be throttled under load. The control plane, on
the other hand, does not need the *lowest* possible latency, what it needs is a
delivery guarantee with bounded, jitter-free latency. In other words, control
tuples must be delivered within a predictable, stable time bound that does not
degrade as the data plane saturates. A control signal that arrives a little
later but always within a known bound is far more useful than one that is
usually fast but occasionally stalls behind a data backlog: the coordination
logic (timeouts, ticks, the flush feedback loop) depends on consistency, not
raw speed.
## Current behavior: they share the same queue
There is one `JCQueue` per executor, and every inbound tuple, data and
control alike, goes through it. Note that the flush tuple already uses
`tryPublishDirect` to bypass producer-side batching, but it is still appended
to the same MPSC `recvQueue`, so it is delivered FIFO behind whatever data
backlog is already enqueued.
## Why this is a problem
Because the two planes share one FIFO queue, data-plane ingestion directly
degrades control-plane responsiveness: **the data plane can starve or delay the
control plane**, which is the opposite of what we want. Control-plane traffic
should be insulated from data-plane buffering and saturation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
