Hi haonan, I've already cherry-picked those two commits into rc/2.0.11, I think rc2 is ready.
Best regards, -------------------- Yuan Tian On Fri, Sep 4, 2026 at 10:37 AM Haonan Hou <[email protected]> wrote: > OK, let me do RC2 release after the bug fixed. > > BR, > Haonan > > On 2026/09/04 02:33:51 Yuan Tian wrote: > > -1 (binding) > > > > I hit two ConfigNode memory bugs on the current edge version, both with > heap > > dumps and crash logs. Under an ordinary workload -- one data pipe plus > > continuous writes, all defaults -- they OOM a ConfigNode with a modest > heap, > > and the second one turns the recovery path (the ratis snapshot) into the > > crash site. > > > > Setup: ConfigNode + DataNode in one 224 MB heap (SerialGC), one > table-model > > realtime pipe to a second IoTDB, 1 device x 100..1000 sensors at 1 Hz. > > Names below are from 2.0.11-SNAPSHOT source. > > > > 1. Pipe heartbeat makes steady-state progress updates grow the ConfigNode > > ratis log without bound > > > > - DataNodes report pipe meta (incl. progress) every > > pipe_heartbeat_interval_seconds_for_collecting_pipe_meta (default 3 > s). > > - Under continuous writes progress changes every round, so > > PipeHeartbeatParser submits a PipeHandleMetaChangeProcedure each time > > (~20/min measured). > > - Each procedure state transition goes through > > ConfigProcedureStore.update() -> consensus write, plus one full > > PipeHandleMetaChangePlan with every pipe's attributes. Measured ~8.4 > > consensus writes per procedure, ~5.5 ratis LogEntryProto/s. > Independent > > of data volume -- 1x100 and 1x1000 sensors grow identically. > > - Ratis keeps LogSegment.records and entryCache on heap and only frees > > them on purge after a snapshot. Defaults: > > config_node_ratis_snapshot_trigger_threshold = 400,000 entries, > > config_node_ratis_periodic_snapshot_interval = 24 h. At 5.5/s the > count > > trigger is ~20 h away; ~330 B/entry resident => ~7 MB/h, ~130 MB > before > > the first purge. > > > > Evidence: defaults, 1x100 -> OOM at 15 h 59 m; jmap -histo:live showed > > LogEntryProto growing linearly (176k -> 285k over 5.3 h). With heartbeat > > lowered to 1 s -> OOM at 8 h 34 m, exactly when the index reached the > 400k > > trigger. MAT on that dump: SegmentedRaftLogCache retains 95.6 MB (44 % > of > > heap), 439,659 TermIndex instances = the index span since last snapshot. > > The whole DataNode write path was 7.6 MB; no pipe event class in the top > > entries. Only data/confignode/consensus/*/current/log_* grows on disk. > > > > 2. PartitionInfo uses a fixed 32 MB buffer to take and load its snapshot > > > > private static final int PARTITION_TABLE_BUFFER_SIZE = 32 * 1024 * > 1024; > > processTakeSnapshot(): new BufferedOutputStream(fos, > > PARTITION_TABLE_BUFFER_SIZE) > > processLoadSnapshot(): new BufferedInputStream(..., > > PARTITION_TABLE_BUFFER_SIZE) > > > > The serialized partition table in the test was 1.7 KB. When the 400k > > trigger finally fired, StateMachineUpdater took the snapshot; > ClusterInfo, > > NodeInfo, ClusterSchemaInfo succeeded, then: > > > > java.lang.OutOfMemoryError: Java heap space > > at java.io.BufferedOutputStream.<init> > > at > > o.a.i.confignode.persistence.partition.PartitionInfo.processTakeSnapshot > > at > > o.a.i.confignode.persistence.executor.ConfigPlanExecutor.takeSnapshot > > at > > > o.a.i.confignode.consensus.statemachine.ConfigRegionStateMachine.takeSnapshot > > at org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot > > > > hs_err: tenured 152,960 K / 152,960 K. A zero-byte > > partition_info.bin-<uuid> > > was left in sm/.tmp.2_439299/. The snapshot is the only thing that can > > purge > > the log, it runs exactly when the heap is fullest, and it demands a 32 > MB > > contiguous block. Without CrashOnOutOfMemoryError it just fails, the log > > keeps growing, every retry fails the same way, and the node dies in a > > full-GC storm anyway. Any ConfigNode within ~32 MB of full at snapshot > > time hits this regardless of issue 1. > > > > Confirmation: same workload, clean data dir, with > > snapshot_trigger_threshold=5000, periodic_snapshot_interval=1800 and > > heartbeat=30 s has run 15 h so far without incident (29 snapshots on > > schedule, old gen oscillates 60-97 % on the flush cycle and recovers, > > 9 full GCs total). Two caveats: heartbeat=30 s raised steady-state pipe > > latency from ~220 ms to ~620-830 ms, so tuning is not a shippable answer; > > and even with those settings purge only frees closed segments -- the open > > segment (config_node_ratis_log_segment_size_max_in_byte = 24 MB, ~236k > > entries) stays fully resident, so at the default 3 s heartbeat a small > heap > > still fills that segment in ~12 h. > > > > There are already two prs to fix these two, [1] has been merged, [2] is > > still under review and will be merged soon. > > > > IMO, since 2.0.11 is the first time we release edge version, we need to > > make it more stable and wait for the above two prs being merged and > > cherry-pick them into rc/2.0.11. > > > > > > Best regards, > > --------------- > > Yuan Tian > > > > On Thu, Sep 3, 2026 at 9:23 AM Haonan Hou <[email protected]> wrote: > > > > > Hi all, > > > > > > Apache IoTDB 2.0.11 has been staged under [2] and it’s time to vote > > > on accepting it for release. All Maven artifacts are available under > [1]. > > > Voting will be open for 72hr. > > > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > > > are required to pass. > > > > > > Release tag: v2.0.11 > > > Hash for the release tag: 9883d933c8ab701b49bdcc7f93a2bd70608e9f96 > > > > > > Before voting +1, PMC members are required to download > > > the signed source code package, compile it as provided, and test > > > the resulting executable on their own platform, along with also > > > verifying that the package meets the requirements of the ASF policy > > > on releases. [3] > > > > > > [ ] +1 accept (indicate what you validated) > > > [ ] -1 reject (explanation required) > > > > > > > > > [1] > https://repository.apache.org/content/repositories/orgapacheiotdb-1201 > > > [2] https://dist.apache.org/repos/dist/dev/iotdb/2.0.11/rc1 > > > [3] https://www.apache.org/dev/release.html#approving-a-release > > > [4] > > > > https://dist.apache.org/repos/dist/dev/iotdb/2.0.11/rc1/RELEASE_NOTES.md > > > [5] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > > > > > Best regards, > > > Haonan Hou > > > > > >
