SEZ9 commented on issue #12361: URL: https://github.com/apache/seatunnel/issues/12361#issuecomment-6487188019
Thanks for the detailed report and the source references. The mechanism you describe matches what is in this thread: the final snapshot split is planned as `ChunkRange.of(chunkStart, null)`, and with `exactly_once = true` the records of a split are accumulated in a `LinkedHashMap` in `pollSplitRecordsIfExactlyOnce()` until snapshot and binlog backfill processing finish. So `snapshot.split.size` guides enumeration but does not cap the row count of that open-ended terminal split when inserts continue after planning, and recovery retries the same unfinished range. #13027 has been opened to reduce whole-split heap retention with temporary spill storage and lazy emission; review and suggestions there are welcome. @201811510411lw, if you are able to try that change against your workload, or share the sanitized `SHOW CREATE TABLE`, the full source split options, and the GC/heap evidence around the failure, that would help confirm the fix matches your data shape. <!-- streview-comment:2117 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
