awkyu commented on PR #4070:
URL: https://github.com/apache/iceberg-python/pull/4070#issuecomment-6961034303

   We hit #4069 in a long-running service: on Python 3.14 a web app whose 
request handlers call `load_table(...)` + scan against an S3-compatible store 
ran its workers out of file descriptors (`RLIMIT_NOFILE` 1024). Each surviving 
FileIO pins its `S3FileSystem`, and each of those holds an eventfd. Once a 
worker was full, even the S3 client's hostname lookup failed (`curlCode: 6, 
Could not resolve hostname`).
   
   We tested this PR against it, and it fixes half. With the cycle collector 
off, 20 × (`load_table` + `scan().to_arrow()`) on a SqlCatalog over a local S3 
endpoint (pyiceberg 0.12.0, pyarrow 24.0.0, Python 3.14.7):
   
   | | eventfds left over | filesystems built |
   |---|---|---|
   | 0.12.0 as released | +40 | 40 |
   | with this PR's change | +20 | 40 |
   
   The other half is a second cycle with the same shape, in the scan planner. 
`ManifestGroupPlanner.partition_filters` (`pyiceberg/table/__init__.py`, the 
`@cached_property` returning 
`KeyDefaultDict(self._build_partition_projection)`) stores a bound method of 
the planner in the planner's own `__dict__`:
   
   ```
   planner -> __dict__["partition_filters"] -> KeyDefaultDict -> bound method 
-> planner
   ```
   
   The planner also holds `io` (the table's FileIO), so every scan leaves the 
table's FileIO, and its filesystem, alive until the collector runs. With this 
PR applied, all 20 surviving FileIOs were pinned this way. Dropping only that 
`partition_filters` entry after planning took the leftover eventfds from +20 to 
+1.
   
   Standalone reproduction (local warehouse, so it counts surviving 
`PyArrowFileIO` objects rather than eventfds; needs only pyiceberg, pyarrow, 
sqlalchemy):
   
   ```python
   import gc, tempfile, weakref
   from functools import lru_cache
   
   import pyarrow as pa
   from pyiceberg.catalog import load_catalog
   from pyiceberg.io.pyarrow import PyArrowFileIO
   from pyiceberg.table import DataScan, ManifestGroupPlanner
   
   planners = []
   
   
   def run(label):
       wh = tempfile.mkdtemp()
       cat = load_catalog("c", type="sql", uri=f"sqlite:///{wh}/c.db",
                          warehouse=f"file://{wh}", init_catalog_tables="true")
       cat.create_namespace("ns")
       cat.create_table("ns.t", pa.schema([("a", 
pa.int64())])).append(pa.table({"a": [1]}))
       cat.load_table("ns.t").scan().to_arrow()
       gc.collect()
       gc.disable()
       live = lambda: sum(type(o) is PyArrowFileIO for o in gc.get_objects())
       before = live()
       for _ in range(20):
           cat.load_table("ns.t").scan().to_arrow()
       print(f"{label:15s} surviving FileIOs after 20 loads: {live() - before}")
       gc.enable()
       gc.collect()
   
   
   run("released")
   
   # This PR, in effect: the per-instance cache keeps only a weakref to the 
FileIO.
   init = PyArrowFileIO.__init__
   
   
   def weak_init(self, properties=None):
       init(self, properties or {})
       ref = weakref.ref(self)
       self.fs_by_scheme = lru_cache(lambda s, n=None: ref()._initialize_fs(s, 
n))
   
   
   PyArrowFileIO.__init__ = weak_init
   run("this PR")
   
   # The second cycle: drop the planner's cached partition_filters after 
planning.
   planner_init, plan_files = ManifestGroupPlanner.__init__, DataScan.plan_files
   
   
   def tracked_init(self, *a, **k):
       planner_init(self, *a, **k)
       planners.append(weakref.ref(self))
   
   
   def plan_and_unpin(self):
       files = plan_files(self)
       for ref in planners:
           if (p := ref()) is not None:
               p.__dict__.pop("partition_filters", None)
       return files
   
   
   ManifestGroupPlanner.__init__, DataScan.plan_files = tracked_init, 
plan_and_unpin
   run("this PR + planner")
   ```
   
   ```
   released        surviving FileIOs after 20 loads: 40
   this PR         surviving FileIOs after 20 loads: 20
   this PR + planner surviving FileIOs after 20 loads: 0
   ```
   
   So a full fix for #4069 probably needs this PR plus breaking the planner's 
cycle, e.g. building `partition_filters` from a function that doesn't capture 
the planner, or not caching it on the instance. The PR also conflicts with 
`main` at the moment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to