dejii commented on PR #38149: URL: https://github.com/apache/beam/pull/38149#issuecomment-4243351828
@stankiewicz Thanks for the summary table. One comment on the leak characterization: There shouldn't be a memory leak. `FileIOTracker` uses Caffeine's `weakKeys()` with a removal listener that calls `fileIO.close()` when a `TableOperations` key is GC'd ([FileIOTracker.java#L37-L47](https://github.com/apache/iceberg/blob/87c743463b6311f2412e1addf19cf204c1b79e3d/core/src/main/java/org/apache/iceberg/io/FileIOTracker.java#L37-L47)). The strong reference chain is `tableCache -> Table -> TableOperations`. This should happen continuously throughout the DoFn's lifetime, not just at teardown, so even a DoFn running for long will naturally clean up per-table FileIOs for tables that go idle. The tracker is bounded by the number of distinct tables written to in the last 10 minutes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
