Sampreeth Reddy B created CASSSIDECAR-510:
---------------------------------------------
Summary: Replace Netty GlobalTrafficShapingHandler with userspace
byte-counting rate limiter so file downloads can use kernel sendfile(2)
Key: CASSSIDECAR-510
URL: https://issues.apache.org/jira/browse/CASSSIDECAR-510
Project: Sidecar for Apache Cassandra
Issue Type: Improvement
Components: Performance
Reporter: Sampreeth Reddy B
Assignee: Sampreeth Reddy B
h3. Description
h4. Problem:
[{{HttpServerOptionsProvider.apply(SidecarConfiguration)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/server/HttpServerOptionsProvider.java]
unconditionally calls
{{options.setTrafficShapingOptions(buildTrafficShapingOptions(…))}} on every
sidecar startup:
{quote}options.setTrafficShapingOptions(buildTrafficShapingOptions(serviceConf.trafficShapingConfiguration()));{quote}
Only install the shaper when a cap is configured does not exist. The
{{buildTrafficShapingOptions(…)}} helper is invoked with the live config values
even when the operator sets
{quote}{{{}inboundGlobalBandwidthBytesPerSecond=0{}}},
{{{}outboundGlobalBandwidthBytesPerSecond=0{}}}, and
{{{}peakOutboundGlobalBandwidthBytesPerSecond=0{}}},
{quote}
the returned {{TrafficShapingOptions}} object is non-null, so Vert.x's
{{HttpServerImpl}} adds a Netty {{GlobalTrafficShapingHandler}} to every
connection's pipeline anyway.
Installing _any_ {{io.netty.handler.traffic.AbstractTrafficShapingHandler}} on
a Netty pipeline has one critical side effect that is not documented in
Vert.x's public API:
{{io.vertx.core.net.impl.ConnectionBase#supportsFileRegion()}} returns
{{false}} whenever the pipeline contains an
{{{}AbstractTrafficShapingHandler{}}}. That return value is checked inside
Vert.x's {{AsyncFileImpl#doSendFile(…)}} and inside
{{Http1xServerResponse#sendFile(…)}} when it returns {{{}false{}}}, Vert.x
silently replaces the {{DefaultFileRegion}} zero-copy path with a userspace
{{{}read(){}}}-into-{{{}ByteBuf{}}} + {{write()}} loop.
Net effect: every live-migration download and every SSTable stream on every
release of the sidecar is served through userspace copy, not kernel
{{sendfile(2)}}, regardless of whether the operator asked for bandwidth
throttling.
h4. Proposed Change
Remove the Netty {{GlobalTrafficShapingHandler}} entirely and move byte-level
rate enforcement into the existing per-file streaming loop as a new
{{SidecarRateLimiter}} injected binding whose unit is bytes per second (the
existing class supports arbitrary permit units it's just a wrapper around
Guava's {{{}SmoothBurstyRateLimiter{}}}).
High-level design:
# Delete the pipeline shaper. In {{{}HttpServerOptionsProvider.apply(…){}}},
drop the {{options.setTrafficShapingOptions(…)}} call. The
{{TrafficShapingConfiguration}} interface is kept for one release as a
deprecated no-op (log a WARN at startup if any non-zero value is set, pointing
operators at the new config section).
# New byte-counting limiter binding.
#
{{{}{}}}Chunk the sendfile call. Replace the single
{{response.sendFile(filename, fileLength, range)}} call inside
[{{FileStreamer.acquireAndSend(…)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/utils/FileStreamer.java]
with a chunked loop (default 256 KiB chunk, configurable):
# Keep {{SidecarRateLimiter}} ("StreamRequestRateLimiter") for admission
control. It still has a legitimate role rejecting new file-stream requests
when the server is saturated at the connection-count level, which is
semantically different from slow this stream down. Rename the binding to
{{"StreamAdmissionRateLimiter"}} to reduce confusion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
