awkyu commented on PR #4070:
URL: https://github.com/apache/iceberg-python/pull/4070#issuecomment-6961034303
We hit #4069 in a long-running service: on Python 3.14 a web app whose
request handlers call `load_table(...)` + scan against an S3-compatible store
ran its workers out of file descriptors (`RLIMIT_NOFILE` 1024). Each surviving
FileIO pins its `S3FileSystem`, and each of those holds an eventfd. Once a
worker was full, even the S3 client's hostname lookup failed (`curlCode: 6,
Could not resolve hostname`).
We tested this PR against it, and it fixes half. With the cycle collector
off, 20 × (`load_table` + `scan().to_arrow()`) on a SqlCatalog over a local S3
endpoint (pyiceberg 0.12.0, pyarrow 24.0.0, Python 3.14.7):
| | eventfds left over | filesystems built |
|---|---|---|
| 0.12.0 as released | +40 | 40 |
| with this PR's change | +20 | 40 |
The other half is a second cycle with the same shape, in the scan planner.
`ManifestGroupPlanner.partition_filters` (`pyiceberg/table/__init__.py`, the
`@cached_property` returning
`KeyDefaultDict(self._build_partition_projection)`) stores a bound method of
the planner in the planner's own `__dict__`:
```
planner -> __dict__["partition_filters"] -> KeyDefaultDict -> bound method
-> planner
```
The planner also holds `io` (the table's FileIO), so every scan leaves the
table's FileIO, and its filesystem, alive until the collector runs. With this
PR applied, all 20 surviving FileIOs were pinned this way. Dropping only that
`partition_filters` entry after planning took the leftover eventfds from +20 to
+1.
Standalone reproduction (local warehouse, so it counts surviving
`PyArrowFileIO` objects rather than eventfds; needs only pyiceberg, pyarrow,
sqlalchemy):
```python
import gc, tempfile, weakref
from functools import lru_cache
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.io.pyarrow import PyArrowFileIO
from pyiceberg.table import DataScan, ManifestGroupPlanner
planners = []
def run(label):
wh = tempfile.mkdtemp()
cat = load_catalog("c", type="sql", uri=f"sqlite:///{wh}/c.db",
warehouse=f"file://{wh}", init_catalog_tables="true")
cat.create_namespace("ns")
cat.create_table("ns.t", pa.schema([("a",
pa.int64())])).append(pa.table({"a": [1]}))
cat.load_table("ns.t").scan().to_arrow()
gc.collect()
gc.disable()
live = lambda: sum(type(o) is PyArrowFileIO for o in gc.get_objects())
before = live()
for _ in range(20):
cat.load_table("ns.t").scan().to_arrow()
print(f"{label:15s} surviving FileIOs after 20 loads: {live() - before}")
gc.enable()
gc.collect()
run("released")
# This PR, in effect: the per-instance cache keeps only a weakref to the
FileIO.
init = PyArrowFileIO.__init__
def weak_init(self, properties=None):
init(self, properties or {})
ref = weakref.ref(self)
self.fs_by_scheme = lru_cache(lambda s, n=None: ref()._initialize_fs(s,
n))
PyArrowFileIO.__init__ = weak_init
run("this PR")
# The second cycle: drop the planner's cached partition_filters after
planning.
planner_init, plan_files = ManifestGroupPlanner.__init__, DataScan.plan_files
def tracked_init(self, *a, **k):
planner_init(self, *a, **k)
planners.append(weakref.ref(self))
def plan_and_unpin(self):
files = plan_files(self)
for ref in planners:
if (p := ref()) is not None:
p.__dict__.pop("partition_filters", None)
return files
ManifestGroupPlanner.__init__, DataScan.plan_files = tracked_init,
plan_and_unpin
run("this PR + planner")
```
```
released surviving FileIOs after 20 loads: 40
this PR surviving FileIOs after 20 loads: 20
this PR + planner surviving FileIOs after 20 loads: 0
```
So a full fix for #4069 probably needs this PR plus breaking the planner's
cycle, e.g. building `partition_filters` from a function that doesn't capture
the planner, or not caching it on the instance. The PR also conflicts with
`main` at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]