-1 (binding)
I hit two ConfigNode memory bugs on the current edge version, both with heap
dumps and crash logs. Under an ordinary workload -- one data pipe plus
continuous writes, all defaults -- they OOM a ConfigNode with a modest heap,
and the second one turns the recovery path (the ratis snapshot) into the
crash site.
Setup: ConfigNode + DataNode in one 224 MB heap (SerialGC), one table-model
realtime pipe to a second IoTDB, 1 device x 100..1000 sensors at 1 Hz.
Names below are from 2.0.11-SNAPSHOT source.
1. Pipe heartbeat makes steady-state progress updates grow the ConfigNode
ratis log without bound
- DataNodes report pipe meta (incl. progress) every
pipe_heartbeat_interval_seconds_for_collecting_pipe_meta (default 3 s).
- Under continuous writes progress changes every round, so
PipeHeartbeatParser submits a PipeHandleMetaChangeProcedure each time
(~20/min measured).
- Each procedure state transition goes through
ConfigProcedureStore.update() -> consensus write, plus one full
PipeHandleMetaChangePlan with every pipe's attributes. Measured ~8.4
consensus writes per procedure, ~5.5 ratis LogEntryProto/s. Independent
of data volume -- 1x100 and 1x1000 sensors grow identically.
- Ratis keeps LogSegment.records and entryCache on heap and only frees
them on purge after a snapshot. Defaults:
config_node_ratis_snapshot_trigger_threshold = 400,000 entries,
config_node_ratis_periodic_snapshot_interval = 24 h. At 5.5/s the count
trigger is ~20 h away; ~330 B/entry resident => ~7 MB/h, ~130 MB before
the first purge.
Evidence: defaults, 1x100 -> OOM at 15 h 59 m; jmap -histo:live showed
LogEntryProto growing linearly (176k -> 285k over 5.3 h). With heartbeat
lowered to 1 s -> OOM at 8 h 34 m, exactly when the index reached the 400k
trigger. MAT on that dump: SegmentedRaftLogCache retains 95.6 MB (44 % of
heap), 439,659 TermIndex instances = the index span since last snapshot.
The whole DataNode write path was 7.6 MB; no pipe event class in the top
entries. Only data/confignode/consensus/*/current/log_* grows on disk.
2. PartitionInfo uses a fixed 32 MB buffer to take and load its snapshot
private static final int PARTITION_TABLE_BUFFER_SIZE = 32 * 1024 * 1024;
processTakeSnapshot(): new BufferedOutputStream(fos,
PARTITION_TABLE_BUFFER_SIZE)
processLoadSnapshot(): new BufferedInputStream(...,
PARTITION_TABLE_BUFFER_SIZE)
The serialized partition table in the test was 1.7 KB. When the 400k
trigger finally fired, StateMachineUpdater took the snapshot; ClusterInfo,
NodeInfo, ClusterSchemaInfo succeeded, then:
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>
at
o.a.i.confignode.persistence.partition.PartitionInfo.processTakeSnapshot
at
o.a.i.confignode.persistence.executor.ConfigPlanExecutor.takeSnapshot
at
o.a.i.confignode.consensus.statemachine.ConfigRegionStateMachine.takeSnapshot
at org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot
hs_err: tenured 152,960 K / 152,960 K. A zero-byte
partition_info.bin-<uuid>
was left in sm/.tmp.2_439299/. The snapshot is the only thing that can
purge
the log, it runs exactly when the heap is fullest, and it demands a 32 MB
contiguous block. Without CrashOnOutOfMemoryError it just fails, the log
keeps growing, every retry fails the same way, and the node dies in a
full-GC storm anyway. Any ConfigNode within ~32 MB of full at snapshot
time hits this regardless of issue 1.
Confirmation: same workload, clean data dir, with
snapshot_trigger_threshold=5000, periodic_snapshot_interval=1800 and
heartbeat=30 s has run 15 h so far without incident (29 snapshots on
schedule, old gen oscillates 60-97 % on the flush cycle and recovers,
9 full GCs total). Two caveats: heartbeat=30 s raised steady-state pipe
latency from ~220 ms to ~620-830 ms, so tuning is not a shippable answer;
and even with those settings purge only frees closed segments -- the open
segment (config_node_ratis_log_segment_size_max_in_byte = 24 MB, ~236k
entries) stays fully resident, so at the default 3 s heartbeat a small heap
still fills that segment in ~12 h.
There are already two prs to fix these two, [1] has been merged, [2] is
still under review and will be merged soon.
IMO, since 2.0.11 is the first time we release edge version, we need to
make it more stable and wait for the above two prs being merged and
cherry-pick them into rc/2.0.11.
Best regards,
---------------
Yuan Tian
On Thu, Sep 3, 2026 at 9:23 AM Haonan Hou <[email protected]> wrote:
> Hi all,
>
> Apache IoTDB 2.0.11 has been staged under [2] and it’s time to vote
> on accepting it for release. All Maven artifacts are available under [1].
> Voting will be open for 72hr.
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
>
> Release tag: v2.0.11
> Hash for the release tag: 9883d933c8ab701b49bdcc7f93a2bd70608e9f96
>
> Before voting +1, PMC members are required to download
> the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also
> verifying that the package meets the requirements of the ASF policy
> on releases. [3]
>
> [ ] +1 accept (indicate what you validated)
> [ ] -1 reject (explanation required)
>
>
> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1201
> [2] https://dist.apache.org/repos/dist/dev/iotdb/2.0.11/rc1
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4]
> https://dist.apache.org/repos/dist/dev/iotdb/2.0.11/rc1/RELEASE_NOTES.md
> [5] https://dist.apache.org/repos/dist/dev/iotdb/KEYS
>
> Best regards,
> Haonan Hou
>
