hongzhi-gao opened a new pull request, #19244:
URL: https://github.com/apache/iotdb/pull/19244
## Problem
IoTConsensus V1 replicates uncompressed log payloads through
`syncLogEntries`. Large payloads consume network bandwidth and can increase
replication backlog.
This PR adds configurable log payload compression while retaining the
existing batching, RPC, application, and synchronization-index advancement
paths. Deployment requires all peers to run the same supporting version and
compression configuration. Rolling upgrades and per-peer capability negotiation
are outside this PR's scope.
## Design and implementation
### Preserve batching; encode individual buffers
The compression unit is each ByteBuffer in `TLogEntry.data`. An original
consensus request from the in-memory queue normally produces one buffer. WAL
backfill can collect multiple WAL records with the same consensus index and
produce multiple buffers in one entry.
Each encoded entry carries the actual codec, uncompressed length, and CRC32C
for every buffer. Compressed and raw buffers may coexist; entries containing
only raw buffers omit compression metadata.
```text
In-memory queue / WAL backfill
|
Existing batching
|
Encode individual buffers; retain the original Batch
|
Existing syncLogEntries RPC
|
Validate and decode the entire request
|
Existing deserialization, application, ACK, and sync-index advancement
```
Per-buffer encoding preserves the existing queue/WAL boundaries, avoids
whole-request concatenation and splitting, allows individual benefit checks,
and bounds temporary workspace by the block being processed. Its tradeoffs are
additional descriptors/checksums and the inability to exploit redundancy across
buffers.
An earlier agent-based real-cluster experiment compared per-buffer and
whole-request compression. Whole-request compression did not show a consistent
end-to-end advantage and consumed more sender CPU. That experiment used
different batching settings and is supporting design evidence only; the final
native implementation benchmarks below evaluate per-buffer encoding.
### Configuration and adaptive fallback
New startup settings:
```properties
data_region_iot_log_compression=NONE
data_region_iot_log_compression_min_size=4096
```
Supported codecs are `NONE`, `LZ4`, `SNAPPY`, and `ZSTD`; Zstd uses level 1.
The default remains `NONE`. After all peers have been upgraded, operators can
enable compression consistently across the cluster, for example:
```properties
data_region_iot_log_compression=LZ4
data_region_iot_log_compression_min_size=4096
```
Before enabling this setting, payloads use the original uncompressed format.
After enabling it, eligible buffers are compressed; buffers smaller than 4096
bytes, buffers without a net size benefit including metadata, and buffers for
which temporary memory cannot be reserved are still sent raw. For example, a 2
KiB buffer stays raw under this configuration. The receiver decodes the
transmitted payload before passing the original bytes to the existing
application path.
LZ4 is the recommended initial choice when enabling compression because it
balances encoding cost and payload reduction. Zstd-1 is an alternative when
bandwidth reduction is the priority. This recommendation is not a claim that
LZ4 is fastest for every workload. The 4 KiB threshold is configurable and was
not selected through an exhaustive threshold search.
### Memory, retries, and failures
The sender creates an encoded copy without modifying the original Batch or
shared ByteBuffer positions. Temporary workspace is reserved non-blockingly per
buffer; only the resulting encoded data remains reserved until the RPC
callback. Reservations are also released on synchronous send/borrow failure.
Retries encode from the original Batch.
Original backlog accounting remains based on uncompressed size. Encoded
copies use the existing consensus memory pool. A failed reservation falls back
to raw data without preventing other buffers from being compressed.
The receiver validates declared sizes and decodes/checksums the entire
request before entering the existing deserialization/application path. A decode
failure returns an explicit compression error. The sender retries without
completing the Batch or advancing the synchronization index; a later Batch ACK
cannot advance past the failed prefix.
The existing RPC frame limit bounds the total decoded payload of a
compressed request. This is a per-request limit, not a new global
receive-memory quota. Decoding remains enabled even when the local sending
configuration is `NONE`.
No WAL or TsFile format is changed. Optional Thrift fields alone do not make
compressed requests safe for old receivers; homogeneous versions remain an
operational requirement.
Compression/decompression timers, raw/encoded payload counters, and
memory-fallback counters are included.
## Performance evaluation
### Hardware and workload
The final native binaries were tested with stock iot-benchmark on three
storage servers. DataNode processes did not use an experimental agent; a
client-side timing observer recorded benchmark boundaries.
All three servers have the following hardware. The CPU, memory, NIC, and
kernel were recorded with the benchmark; the disk inventory was additionally
verified when preparing this PR.
| Component | Configuration per server |
|---|---|
| CPU | Intel Core i7-10700 @ 2.90 GHz; 1 socket, 8 physical cores, 16
logical CPUs |
| Memory | 32 GiB installed; approximately 31.2 GiB reported by Linux |
| Network | 1 Gbit/s Ethernet, MTU 1500 |
| Benchmark storage | Seagate ST10000NM001G 10 TB HDD, approximately 9.1
TiB; `/data` on ext4 |
| Additional installed storage | Samsung SSD 970 EVO Plus 250 GB NVMe,
approximately 232.9 GiB |
| OS | Ubuntu 20.04.6 LTS; Linux 5.4.0-195-generic |
| Item | Configuration |
|---|---|
| Topologies | Two replicas: one writing node and one receiving node; three
replicas: one writing node and two receiving nodes |
| Placement | One ConfigNode; benchmark driver co-located with the writing
DataNode |
| DataNode memory | 1536 MiB heap, 512 MiB direct memory; initial consensus
budget approximately 154 MiB |
| Replication | Default 16 MiB batch limit; up to five in-flight Batches per
peer |
| Writes | Tree model, aligned writes, one DataRegion, 8 devices × 24
measurements, four writing clients |
| Data per phase | 2048 requests; 12,582,912 points |
| Profiles | Mixed six-type data, random DOUBLE, and 64-character TEXT |
Two replica counts × four modes × three profiles × three repetitions
produced 72 phases. Mode order was randomized within each topology/repetition.
Each new cluster ran mixed, numeric, and text workloads in that order, without
a separate warm-up. Other jobs on the shared servers remained running.
End-to-end timing starts when the first client begins writing and ends when
native `waitSyncLogComplete` confirms that the source synchronization frontier
has caught up (`searchIndex == minSyncIndex`). It includes writes, batching,
WAL backfill, encoding, transport, application, and retries.
The performance tables below exclude complete phases that logged a WAL-roll
wait timeout; they do not subtract 30 seconds from measured times. All other
normal waits remain included. This filters out a specific existing WAL-backfill
delay: when the required records are still in the active WAL file, the iterator
may wait up to 30 seconds before forcing a roll.
### End-to-end time
Seconds; entries show **mean [minimum, maximum]**. A single remaining
observation is shown directly.
| Replicas | Profile | NONE | LZ4 | SNAPPY | Zstd-1 |
|---|---|---:|---:|---:|---:|
| 2 | Mixed six types | 6.00 [4.10, 9.13] | 5.40 [3.93, 7.89] | 8.24 [5.15,
11.34] | 5.19 [4.16, 7.23] |
| 2 | DOUBLE | 3.84 [2.33, 4.69] | 2.36 [1.90, 2.82] | 2.74 [2.60, 2.84] |
2.64 [1.67, 4.58] |
| 3 | Mixed six types | — | 4.16 | 11.40 | 9.47 [5.41, 16.37] |
| 3 | DOUBLE | 5.00 | 2.54 [1.93, 3.14] | 2.51 [2.49, 2.53] | 4.34 [3.21,
5.36] |
“—” means no observation remains after filtering. All TEXT phases triggered
the WAL wait timeout, so TEXT timings are omitted under this reporting rule.
For two-replica mixed writes, LZ4 and Zstd-1 had slightly lower mean times
than NONE. All three compression modes had lower mean times for two-replica
DOUBLE writes. LZ4 and Snappy also had lower observed times for three-replica
DOUBLE writes; Zstd-1 did not have a uniform timing advantage.
Filtering leaves unequal sample sizes and different retained repetitions,
with some cells containing only one observation. These are descriptive results,
not a stable algorithm ranking or a strictly paired speedup comparison.
### Server load and payload reduction: NONE versus LZ4
The following comparison uses two-replica mixed writes without WAL wait
timeouts. Time, CPU consumption, and payload ratio are means; RSS is the
largest sampled value for any individual DataNode across the included phases.
| Metric | Compression disabled (NONE) | LZ4 enabled | Observed change |
|---|---:|---:|---:|
| End-to-end time | 6.00 s | 5.40 s | Approximately 10.0% lower |
| Total DataNode CPU consumption | 17.30 CPU-s | 17.42 CPU-s | Approximately
0.7% higher |
| Maximum individual DataNode RSS | 1159 MiB | 1030 MiB | 129 MiB lower in
these samples |
| Log payload size, relative to original | 100% | 13.0% | Approximately
87.0% lower |
For this workload, enabling LZ4 reduced log payload by approximately 87%,
reduced mean end-to-end time by approximately 10%, and left total DataNode CPU
consumption approximately unchanged.
CPU-s sums the CPU time of all DataNodes during the end-to-end window. It
includes writing, replication, compression, decompression, and application; it
is neither isolated codec cost nor instantaneous CPU utilization. Compression
adds computation while reducing transmitted data, so overall CPU consumption
reflects the net effect.
Encoding introduces additional copies and temporary workspace. The lower
sampled RSS does not establish that compression saves memory: backlog,
in-flight requests, and GC timing also affect process memory.
Payload ratios include adaptive raw fallbacks but exclude Thrift descriptors
and TCP overhead. Therefore, the payload reduction is not an identical
reduction in total network traffic. Per-node CPU, RSS, heap, GC, disk, and
network measurements were recorded.
## Validation and limitations
Executed checks:
- 56 focused tests passed with no failures, errors, or skips. Coverage
includes codec round trips, real bidirectional asynchronous Thrift,
whole-request rejection on corruption, length limits, per-buffer memory
fallback, reservation cleanup, and synchronization-index gap protection.
- Distribution and dependency reactor `clean install` passed with tests
skipped.
- Full-reactor English and Chinese `test-compile` passed.
- Scoped `spotless:check` passed before submission.
- All 72 native benchmark phases completed with zero client failures and
caught-up synchronization frontiers.
- All 288 query-verification logs passed, covering counts, numeric SUM,
TEXT/BOOLEAN first/last values, time ranges, and queries after stopping the
writing source.
Commands used for build/test validation:
```bash
mvn test -o -pl iotdb-core/consensus -am
-Dtest=CompressedReplicationTest,LogCompressionTest,LogDispatcherCompressionTest,DispatchLogHandlerTest,IoTConsensusMemoryManagerTest,IoTConsensusRecoveryTest,IoTConsensusServerImplTest,LogDispatcherThreadMetricsTest,SyncStatusTest
-Dsurefire.failIfNoSpecifiedTests=false -Ddevelocity.off=true
mvn clean install -o -pl distribution -am -DskipTests -Ddevelocity.off=true
mvn test-compile -o -DskipTests -Ddevelocity.off=true
mvn test-compile -o -DskipTests -P with-zh-locale -Ddevelocity.off=true
mvn spotless:check -o -pl
iotdb-core/consensus,iotdb-core/datanode,iotdb-core/node-commons,iotdb-protocol/thrift-consensus
-Ddevelocity.off=true
```
Limitations:
- Query entry points may route to another replica, so the native query
checks are not per-replica, per-point checksums.
- The filtered performance tables do not represent end-to-end behavior
including WAL wait timeouts. TEXT timings and a complete three-replica mixed
comparison are unavailable under this filter.
- iot-benchmark used `WORKLOAD_BUFFER_SIZE=100`, which reuses buffered
input. Real write/WAL paths were exercised with synthetic data; compression
ratios and speedups are not universal production guarantees.
- Power-loss recovery, table-model performance, and single-buffer stress
near the RPC frame limit were not tested.
## Key changed classes
- `LogCompression`: per-buffer encoding, metadata validation, bounded
decoding, checksums, and reservation ownership.
- `LogDispatcher` / `DispatchLogHandler`: encoded RPC attempts, cleanup,
retries, and compression-error handling.
- `IoTConsensusRPCServiceProcessor`: whole-request validation/decompression
before application.
- `IoTConsensusConfig` / `IoTDBDescriptor` / `DataRegionConsensusImpl`:
startup configuration and wiring.
- `iotconsensus.thrift`: codec descriptors and explicit compression-error
responses.
This PR has:
- [x] Been self-reviewed.
- [x] Documented the new settings in the configuration template.
- [x] Added or updated focused unit/component tests.
- [x] Been tested in real two- and three-replica IoTDB clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]