hongzhi-gao opened a new pull request, #19244:
URL: https://github.com/apache/iotdb/pull/19244

   ## Problem
   
   IoTConsensus V1 replicates uncompressed log payloads through 
`syncLogEntries`. Large payloads consume network bandwidth and can increase 
replication backlog.
   
   This PR adds configurable log payload compression while retaining the 
existing batching, RPC, application, and synchronization-index advancement 
paths. Deployment requires all peers to run the same supporting version and 
compression configuration. Rolling upgrades and per-peer capability negotiation 
are outside this PR's scope.
   
   ## Design and implementation
   
   ### Preserve batching; encode individual buffers
   
   The compression unit is each ByteBuffer in `TLogEntry.data`. An original 
consensus request from the in-memory queue normally produces one buffer. WAL 
backfill can collect multiple WAL records with the same consensus index and 
produce multiple buffers in one entry.
   
   Each encoded entry carries the actual codec, uncompressed length, and CRC32C 
for every buffer. Compressed and raw buffers may coexist; entries containing 
only raw buffers omit compression metadata.
   
   ```text
   In-memory queue / WAL backfill
                 |
          Existing batching
                 |
   Encode individual buffers; retain the original Batch
                 |
        Existing syncLogEntries RPC
                 |
   Validate and decode the entire request
                 |
   Existing deserialization, application, ACK, and sync-index advancement
   ```
   
   Per-buffer encoding preserves the existing queue/WAL boundaries, avoids 
whole-request concatenation and splitting, allows individual benefit checks, 
and bounds temporary workspace by the block being processed. Its tradeoffs are 
additional descriptors/checksums and the inability to exploit redundancy across 
buffers.
   
   An earlier agent-based real-cluster experiment compared per-buffer and 
whole-request compression. Whole-request compression did not show a consistent 
end-to-end advantage and consumed more sender CPU. That experiment used 
different batching settings and is supporting design evidence only; the final 
native implementation benchmarks below evaluate per-buffer encoding.
   
   ### Configuration and adaptive fallback
   
   New startup settings:
   
   ```properties
   data_region_iot_log_compression=NONE
   data_region_iot_log_compression_min_size=4096
   ```
   
   Supported codecs are `NONE`, `LZ4`, `SNAPPY`, and `ZSTD`; Zstd uses level 1. 
The default remains `NONE`. After all peers have been upgraded, operators can 
enable compression consistently across the cluster, for example:
   
   ```properties
   data_region_iot_log_compression=LZ4
   data_region_iot_log_compression_min_size=4096
   ```
   
   Before enabling this setting, payloads use the original uncompressed format. 
After enabling it, eligible buffers are compressed; buffers smaller than 4096 
bytes, buffers without a net size benefit including metadata, and buffers for 
which temporary memory cannot be reserved are still sent raw. For example, a 2 
KiB buffer stays raw under this configuration. The receiver decodes the 
transmitted payload before passing the original bytes to the existing 
application path.
   
   LZ4 is the recommended initial choice when enabling compression because it 
balances encoding cost and payload reduction. Zstd-1 is an alternative when 
bandwidth reduction is the priority. This recommendation is not a claim that 
LZ4 is fastest for every workload. The 4 KiB threshold is configurable and was 
not selected through an exhaustive threshold search.
   
   ### Memory, retries, and failures
   
   The sender creates an encoded copy without modifying the original Batch or 
shared ByteBuffer positions. Temporary workspace is reserved non-blockingly per 
buffer; only the resulting encoded data remains reserved until the RPC 
callback. Reservations are also released on synchronous send/borrow failure. 
Retries encode from the original Batch.
   
   Original backlog accounting remains based on uncompressed size. Encoded 
copies use the existing consensus memory pool. A failed reservation falls back 
to raw data without preventing other buffers from being compressed.
   
   The receiver validates declared sizes and decodes/checksums the entire 
request before entering the existing deserialization/application path. A decode 
failure returns an explicit compression error. The sender retries without 
completing the Batch or advancing the synchronization index; a later Batch ACK 
cannot advance past the failed prefix.
   
   The existing RPC frame limit bounds the total decoded payload of a 
compressed request. This is a per-request limit, not a new global 
receive-memory quota. Decoding remains enabled even when the local sending 
configuration is `NONE`.
   
   No WAL or TsFile format is changed. Optional Thrift fields alone do not make 
compressed requests safe for old receivers; homogeneous versions remain an 
operational requirement.
   
   Compression/decompression timers, raw/encoded payload counters, and 
memory-fallback counters are included.
   
   ## Performance evaluation
   
   ### Hardware and workload
   
   The final native binaries were tested with stock iot-benchmark on three 
storage servers. DataNode processes did not use an experimental agent; a 
client-side timing observer recorded benchmark boundaries.
   
   All three servers have the following hardware. The CPU, memory, NIC, and 
kernel were recorded with the benchmark; the disk inventory was additionally 
verified when preparing this PR.
   
   | Component | Configuration per server |
   |---|---|
   | CPU | Intel Core i7-10700 @ 2.90 GHz; 1 socket, 8 physical cores, 16 
logical CPUs |
   | Memory | 32 GiB installed; approximately 31.2 GiB reported by Linux |
   | Network | 1 Gbit/s Ethernet, MTU 1500 |
   | Benchmark storage | Seagate ST10000NM001G 10 TB HDD, approximately 9.1 
TiB; `/data` on ext4 |
   | Additional installed storage | Samsung SSD 970 EVO Plus 250 GB NVMe, 
approximately 232.9 GiB |
   | OS | Ubuntu 20.04.6 LTS; Linux 5.4.0-195-generic |
   
   | Item | Configuration |
   |---|---|
   | Topologies | Two replicas: one writing node and one receiving node; three 
replicas: one writing node and two receiving nodes |
   | Placement | One ConfigNode; benchmark driver co-located with the writing 
DataNode |
   | DataNode memory | 1536 MiB heap, 512 MiB direct memory; initial consensus 
budget approximately 154 MiB |
   | Replication | Default 16 MiB batch limit; up to five in-flight Batches per 
peer |
   | Writes | Tree model, aligned writes, one DataRegion, 8 devices × 24 
measurements, four writing clients |
   | Data per phase | 2048 requests; 12,582,912 points |
   | Profiles | Mixed six-type data, random DOUBLE, and 64-character TEXT |
   
   Two replica counts × four modes × three profiles × three repetitions 
produced 72 phases. Mode order was randomized within each topology/repetition. 
Each new cluster ran mixed, numeric, and text workloads in that order, without 
a separate warm-up. Other jobs on the shared servers remained running.
   
   End-to-end timing starts when the first client begins writing and ends when 
native `waitSyncLogComplete` confirms that the source synchronization frontier 
has caught up (`searchIndex == minSyncIndex`). It includes writes, batching, 
WAL backfill, encoding, transport, application, and retries.
   
   The performance tables below exclude complete phases that logged a WAL-roll 
wait timeout; they do not subtract 30 seconds from measured times. All other 
normal waits remain included. This filters out a specific existing WAL-backfill 
delay: when the required records are still in the active WAL file, the iterator 
may wait up to 30 seconds before forcing a roll.
   
   ### End-to-end time
   
   Seconds; entries show **mean [minimum, maximum]**. A single remaining 
observation is shown directly.
   
   | Replicas | Profile | NONE | LZ4 | SNAPPY | Zstd-1 |
   |---|---|---:|---:|---:|---:|
   | 2 | Mixed six types | 6.00 [4.10, 9.13] | 5.40 [3.93, 7.89] | 8.24 [5.15, 
11.34] | 5.19 [4.16, 7.23] |
   | 2 | DOUBLE | 3.84 [2.33, 4.69] | 2.36 [1.90, 2.82] | 2.74 [2.60, 2.84] | 
2.64 [1.67, 4.58] |
   | 3 | Mixed six types | — | 4.16 | 11.40 | 9.47 [5.41, 16.37] |
   | 3 | DOUBLE | 5.00 | 2.54 [1.93, 3.14] | 2.51 [2.49, 2.53] | 4.34 [3.21, 
5.36] |
   
   “—” means no observation remains after filtering. All TEXT phases triggered 
the WAL wait timeout, so TEXT timings are omitted under this reporting rule.
   
   For two-replica mixed writes, LZ4 and Zstd-1 had slightly lower mean times 
than NONE. All three compression modes had lower mean times for two-replica 
DOUBLE writes. LZ4 and Snappy also had lower observed times for three-replica 
DOUBLE writes; Zstd-1 did not have a uniform timing advantage.
   
   Filtering leaves unequal sample sizes and different retained repetitions, 
with some cells containing only one observation. These are descriptive results, 
not a stable algorithm ranking or a strictly paired speedup comparison.
   
   ### Server load and payload reduction: NONE versus LZ4
   
   The following comparison uses two-replica mixed writes without WAL wait 
timeouts. Time, CPU consumption, and payload ratio are means; RSS is the 
largest sampled value for any individual DataNode across the included phases.
   
   | Metric | Compression disabled (NONE) | LZ4 enabled | Observed change |
   |---|---:|---:|---:|
   | End-to-end time | 6.00 s | 5.40 s | Approximately 10.0% lower |
   | Total DataNode CPU consumption | 17.30 CPU-s | 17.42 CPU-s | Approximately 
0.7% higher |
   | Maximum individual DataNode RSS | 1159 MiB | 1030 MiB | 129 MiB lower in 
these samples |
   | Log payload size, relative to original | 100% | 13.0% | Approximately 
87.0% lower |
   
   For this workload, enabling LZ4 reduced log payload by approximately 87%, 
reduced mean end-to-end time by approximately 10%, and left total DataNode CPU 
consumption approximately unchanged.
   
   CPU-s sums the CPU time of all DataNodes during the end-to-end window. It 
includes writing, replication, compression, decompression, and application; it 
is neither isolated codec cost nor instantaneous CPU utilization. Compression 
adds computation while reducing transmitted data, so overall CPU consumption 
reflects the net effect.
   
   Encoding introduces additional copies and temporary workspace. The lower 
sampled RSS does not establish that compression saves memory: backlog, 
in-flight requests, and GC timing also affect process memory.
   
   Payload ratios include adaptive raw fallbacks but exclude Thrift descriptors 
and TCP overhead. Therefore, the payload reduction is not an identical 
reduction in total network traffic. Per-node CPU, RSS, heap, GC, disk, and 
network measurements were recorded.
   
   ## Validation and limitations
   
   Executed checks:
   
   - 56 focused tests passed with no failures, errors, or skips. Coverage 
includes codec round trips, real bidirectional asynchronous Thrift, 
whole-request rejection on corruption, length limits, per-buffer memory 
fallback, reservation cleanup, and synchronization-index gap protection.
   - Distribution and dependency reactor `clean install` passed with tests 
skipped.
   - Full-reactor English and Chinese `test-compile` passed.
   - Scoped `spotless:check` passed before submission.
   - All 72 native benchmark phases completed with zero client failures and 
caught-up synchronization frontiers.
   - All 288 query-verification logs passed, covering counts, numeric SUM, 
TEXT/BOOLEAN first/last values, time ranges, and queries after stopping the 
writing source.
   
   Commands used for build/test validation:
   
   ```bash
   mvn test -o -pl iotdb-core/consensus -am 
-Dtest=CompressedReplicationTest,LogCompressionTest,LogDispatcherCompressionTest,DispatchLogHandlerTest,IoTConsensusMemoryManagerTest,IoTConsensusRecoveryTest,IoTConsensusServerImplTest,LogDispatcherThreadMetricsTest,SyncStatusTest
 -Dsurefire.failIfNoSpecifiedTests=false -Ddevelocity.off=true
   mvn clean install -o -pl distribution -am -DskipTests -Ddevelocity.off=true
   mvn test-compile -o -DskipTests -Ddevelocity.off=true
   mvn test-compile -o -DskipTests -P with-zh-locale -Ddevelocity.off=true
   mvn spotless:check -o -pl 
iotdb-core/consensus,iotdb-core/datanode,iotdb-core/node-commons,iotdb-protocol/thrift-consensus
 -Ddevelocity.off=true
   ```
   
   Limitations:
   
   - Query entry points may route to another replica, so the native query 
checks are not per-replica, per-point checksums.
   - The filtered performance tables do not represent end-to-end behavior 
including WAL wait timeouts. TEXT timings and a complete three-replica mixed 
comparison are unavailable under this filter.
   - iot-benchmark used `WORKLOAD_BUFFER_SIZE=100`, which reuses buffered 
input. Real write/WAL paths were exercised with synthetic data; compression 
ratios and speedups are not universal production guarantees.
   - Power-loss recovery, table-model performance, and single-buffer stress 
near the RPC frame limit were not tested.
   
   ## Key changed classes
   
   - `LogCompression`: per-buffer encoding, metadata validation, bounded 
decoding, checksums, and reservation ownership.
   - `LogDispatcher` / `DispatchLogHandler`: encoded RPC attempts, cleanup, 
retries, and compression-error handling.
   - `IoTConsensusRPCServiceProcessor`: whole-request validation/decompression 
before application.
   - `IoTConsensusConfig` / `IoTDBDescriptor` / `DataRegionConsensusImpl`: 
startup configuration and wiring.
   - `iotconsensus.thrift`: codec descriptors and explicit compression-error 
responses.
   
   This PR has:
   
   - [x] Been self-reviewed.
   - [x] Documented the new settings in the configuration template.
   - [x] Added or updated focused unit/component tests.
   - [x] Been tested in real two- and three-replica IoTDB clusters.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to