OK, let me do RC2 release after the bug fixed. BR, Haonan
On 2026/09/04 02:33:51 Yuan Tian wrote: > -1 (binding) > > I hit two ConfigNode memory bugs on the current edge version, both with heap > dumps and crash logs. Under an ordinary workload -- one data pipe plus > continuous writes, all defaults -- they OOM a ConfigNode with a modest heap, > and the second one turns the recovery path (the ratis snapshot) into the > crash site. > > Setup: ConfigNode + DataNode in one 224 MB heap (SerialGC), one table-model > realtime pipe to a second IoTDB, 1 device x 100..1000 sensors at 1 Hz. > Names below are from 2.0.11-SNAPSHOT source. > > 1. Pipe heartbeat makes steady-state progress updates grow the ConfigNode > ratis log without bound > > - DataNodes report pipe meta (incl. progress) every > pipe_heartbeat_interval_seconds_for_collecting_pipe_meta (default 3 s). > - Under continuous writes progress changes every round, so > PipeHeartbeatParser submits a PipeHandleMetaChangeProcedure each time > (~20/min measured). > - Each procedure state transition goes through > ConfigProcedureStore.update() -> consensus write, plus one full > PipeHandleMetaChangePlan with every pipe's attributes. Measured ~8.4 > consensus writes per procedure, ~5.5 ratis LogEntryProto/s. Independent > of data volume -- 1x100 and 1x1000 sensors grow identically. > - Ratis keeps LogSegment.records and entryCache on heap and only frees > them on purge after a snapshot. Defaults: > config_node_ratis_snapshot_trigger_threshold = 400,000 entries, > config_node_ratis_periodic_snapshot_interval = 24 h. At 5.5/s the count > trigger is ~20 h away; ~330 B/entry resident => ~7 MB/h, ~130 MB before > the first purge. > > Evidence: defaults, 1x100 -> OOM at 15 h 59 m; jmap -histo:live showed > LogEntryProto growing linearly (176k -> 285k over 5.3 h). With heartbeat > lowered to 1 s -> OOM at 8 h 34 m, exactly when the index reached the 400k > trigger. MAT on that dump: SegmentedRaftLogCache retains 95.6 MB (44 % of > heap), 439,659 TermIndex instances = the index span since last snapshot. > The whole DataNode write path was 7.6 MB; no pipe event class in the top > entries. Only data/confignode/consensus/*/current/log_* grows on disk. > > 2. PartitionInfo uses a fixed 32 MB buffer to take and load its snapshot > > private static final int PARTITION_TABLE_BUFFER_SIZE = 32 * 1024 * 1024; > processTakeSnapshot(): new BufferedOutputStream(fos, > PARTITION_TABLE_BUFFER_SIZE) > processLoadSnapshot(): new BufferedInputStream(..., > PARTITION_TABLE_BUFFER_SIZE) > > The serialized partition table in the test was 1.7 KB. When the 400k > trigger finally fired, StateMachineUpdater took the snapshot; ClusterInfo, > NodeInfo, ClusterSchemaInfo succeeded, then: > > java.lang.OutOfMemoryError: Java heap space > at java.io.BufferedOutputStream.<init> > at > o.a.i.confignode.persistence.partition.PartitionInfo.processTakeSnapshot > at > o.a.i.confignode.persistence.executor.ConfigPlanExecutor.takeSnapshot > at > o.a.i.confignode.consensus.statemachine.ConfigRegionStateMachine.takeSnapshot > at org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot > > hs_err: tenured 152,960 K / 152,960 K. A zero-byte > partition_info.bin-<uuid> > was left in sm/.tmp.2_439299/. The snapshot is the only thing that can > purge > the log, it runs exactly when the heap is fullest, and it demands a 32 MB > contiguous block. Without CrashOnOutOfMemoryError it just fails, the log > keeps growing, every retry fails the same way, and the node dies in a > full-GC storm anyway. Any ConfigNode within ~32 MB of full at snapshot > time hits this regardless of issue 1. > > Confirmation: same workload, clean data dir, with > snapshot_trigger_threshold=5000, periodic_snapshot_interval=1800 and > heartbeat=30 s has run 15 h so far without incident (29 snapshots on > schedule, old gen oscillates 60-97 % on the flush cycle and recovers, > 9 full GCs total). Two caveats: heartbeat=30 s raised steady-state pipe > latency from ~220 ms to ~620-830 ms, so tuning is not a shippable answer; > and even with those settings purge only frees closed segments -- the open > segment (config_node_ratis_log_segment_size_max_in_byte = 24 MB, ~236k > entries) stays fully resident, so at the default 3 s heartbeat a small heap > still fills that segment in ~12 h. > > There are already two prs to fix these two, [1] has been merged, [2] is > still under review and will be merged soon. > > IMO, since 2.0.11 is the first time we release edge version, we need to > make it more stable and wait for the above two prs being merged and > cherry-pick them into rc/2.0.11. > > > Best regards, > --------------- > Yuan Tian > > On Thu, Sep 3, 2026 at 9:23 AM Haonan Hou <[email protected]> wrote: > > > Hi all, > > > > Apache IoTDB 2.0.11 has been staged under [2] and it’s time to vote > > on accepting it for release. All Maven artifacts are available under [1]. > > Voting will be open for 72hr. > > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > > are required to pass. > > > > Release tag: v2.0.11 > > Hash for the release tag: 9883d933c8ab701b49bdcc7f93a2bd70608e9f96 > > > > Before voting +1, PMC members are required to download > > the signed source code package, compile it as provided, and test > > the resulting executable on their own platform, along with also > > verifying that the package meets the requirements of the ASF policy > > on releases. [3] > > > > [ ] +1 accept (indicate what you validated) > > [ ] -1 reject (explanation required) > > > > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1201 > > [2] https://dist.apache.org/repos/dist/dev/iotdb/2.0.11/rc1 > > [3] https://www.apache.org/dev/release.html#approving-a-release > > [4] > > https://dist.apache.org/repos/dist/dev/iotdb/2.0.11/rc1/RELEASE_NOTES.md > > [5] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > > > Best regards, > > Haonan Hou > > >
