workbench13 opened a new issue, #4414:
URL: https://github.com/apache/iceberg-python/issues/4414

   ### Apache Iceberg version
   
   0.12.0 (latest release). The same code is on `main`.
   
   ### Please describe the bug 🐞
   
   With a request signer configured (`s3.signer`, e.g. REST remote signing), 
`FsspecFileIO` intermittently sends **unsigned** S3 requests. The store answers 
403, which s3fs turns into `PermissionError: Forbidden`. It is usually a 
`HeadObject`, because that is the most frequent call.
   
   We saw this in long-running write jobs against a REST catalog (Lakekeeper) 
with remote signing on an S3-compatible store. The 403s were rare and looked 
random, and a retry always succeeded.
   
   #### Cause
   
   `pyiceberg.io.fsspec._s3` does the following:
   
   ```python
   fs = S3FileSystem(**s3_fs_kwargs)
   
   for event_name, event_function in register_events.items():
       fs.s3.meta.events.unregister(event_name, unique_id=1925)
       fs.s3.meta.events.register_last(event_name, event_function, 
unique_id=1925)
   ```
   
   1. **The S3FileSystem is shared.** `S3FileSystem(**s3_fs_kwargs)` comes from 
fsspec's instance cache. The signer is not part of the cache key, so every 
`FsspecFileIO` in the process with the same endpoint/region gets the **same** 
`S3FileSystem` and the same aiobotocore client.
   2. **The signer is swapped while requests are in flight.** Every call to 
`_s3` unregisters the signer on that shared client and registers its own. 
`FsspecFileIO.get_fs` caches per thread, so `_s3` runs once per FileIO per 
thread, i.e. every time a newly loaded table is read or written from a worker 
thread.
   3. **The gap sends unsigned requests.** fsspec's IO thread keeps signing 
other requests during the swap. A `before-sign.s3` emitted between `unregister` 
and `register_last` finds no signer, and since `signature_version` is 
`UNSIGNED`, the request goes out unsigned. Occasionally the two steps also race 
each other and raise `KeyError: 1925` from botocore's `unregister`.
   
   A second consequence, without any race: all FileIOs sign with the signer of 
whichever FileIO called `_s3` last. That signer carries that FileIO's 
properties: the signer URI/endpoint of that table and the `AuthManager` of that 
catalog. A process with two catalogs (different principals or warehouses) can 
therefore sign one catalog's requests with the other's credentials.
   
   A related, smaller issue: the signer is registered on the client object. If 
s3fs ever recreates its client (`set_session` does so when the aiohttp sessions 
are closed), the new client has no signer.
   
   #### Reproduction
   
   The script below needs only MinIO, or any S3 that rejects unsigned requests:
   
   ```
   docker run -p 9000:9000 -e MINIO_ROOT_USER=minio -e 
MINIO_ROOT_PASSWORD=minio123 minio/minio server /data
   ```
   
   It registers an in-process signer in `SIGNERS` that signs with SigV4, as a 
REST signer would. Four threads `HEAD` one object while two threads create new 
`FsspecFileIO`s and call `get_fs("s3")`, which is what loading a table and 
reading it from a worker does.
   
   <details><summary>repro.py</summary>
   
   ```python
   """Shared S3FileSystem + per-FileIO signer swap in pyiceberg.io.fsspec._s3.
   
   Needs a MinIO server (or any S3 that rejects unsigned requests), e.g.:
       docker run -p 9000:9000 -e MINIO_ROOT_USER=minio -e 
MINIO_ROOT_PASSWORD=minio123 minio/minio server /data
   """
   import collections
   import sys
   import threading
   import time
   
   import s3fs
   from botocore.auth import S3SigV4Auth
   from botocore.credentials import Credentials
   from pyiceberg.io import fsspec as pf
   
   ENDPOINT = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:9000";
   KEY, SECRET = (sys.argv[2], sys.argv[3]) if len(sys.argv) > 3 else ("minio", 
"minio123")
   
   
   class InProcessSigner(pf.S3RequestSigner):
       """Stands in for S3V4RestSigner: signs the request the way a REST signer 
would."""
   
       def __call__(self, request, **_):
           S3SigV4Auth(Credentials(KEY, SECRET), "s3", 
"us-east-1").add_auth(request)
   
   
   pf.SIGNERS["InProcessSigner"] = InProcessSigner
   PROPS = {"s3.endpoint": ENDPOINT, "s3.region": "us-east-1", "s3.signer": 
"InProcessSigner"}
   
   admin = s3fs.S3FileSystem(key=KEY, secret=SECRET, skip_instance_cache=True,
                             client_kwargs={"endpoint_url": ENDPOINT, 
"region_name": "us-east-1"})
   if not admin.exists("repro"):
       admin.mkdir("repro")
   admin.pipe("repro/data.parquet", b"x" * 100)
   
   # 1. Every FsspecFileIO with the same endpoint/region gets the same 
S3FileSystem.
   a, b = pf.FsspecFileIO({**PROPS, "table": "a"}), pf.FsspecFileIO({**PROPS, 
"table": "b"})
   print("same S3FileSystem for two FileIOs:", a.get_fs("s3") is b.get_fs("s3"))
   
   
   def run(seconds, new_fileios):
       fs = pf.FsspecFileIO(dict(PROPS)).get_fs("s3")
       ok, errors, stop = [0], collections.Counter(), threading.Event()
   
       def reader():
           while not stop.is_set():
               try:
                   fs.info("repro/data.parquet", refresh=True)
                   ok[0] += 1
               except Exception as e:  # noqa: BLE001
                   errors[f"{type(e).__name__}: {e}"[:60]] += 1
   
       def loader():
           # What happens whenever a table is loaded and then read/written from 
a worker thread.
           while not stop.is_set():
               try:
                   pf.FsspecFileIO(dict(PROPS)).get_fs("s3")
               except Exception as e:  # noqa: BLE001
                   errors[f"loader {type(e).__name__}: {e}"[:60]] += 1
   
       threads = [threading.Thread(target=reader) for _ in range(4)]
       threads += [threading.Thread(target=loader) for _ in range(new_fileios)]
       for t in threads:
           t.start()
       time.sleep(seconds)
       stop.set()
       for t in threads:
           t.join()
       return ok[0], dict(errors)
   
   
   # 2. Unsigned requests while other threads create FileIOs.
   print("readers only:            ", run(20, 0))
   print("readers + 2 FileIO loops:", run(20, 2))
   ```
   
   </details>
   
   Output (pyiceberg 0.12.0, s3fs/fsspec 2026.9.0, botocore 1.43.117, 
aiobotocore 3.10.0, Python 3.13, macOS), two runs:
   
   ```
   same S3FileSystem for two FileIOs: True
   readers only:             (44788, {})
   readers + 2 FileIO loops: (486, {'PermissionError: Forbidden': 27, 'loader 
KeyError: 1925': 4})
   
   readers only:             (43351, {})
   readers + 2 FileIO loops: (493, {'PermissionError: Forbidden': 8, 'loader 
KeyError: 1925': 7})
   ```
   
   The loaders run in a tight loop to make the race frequent. In real workloads 
the rate is much lower, which is why it showed up as rare 403s in long jobs.
   
   #### Possible fix
   
   When a signer is configured, don't put per-FileIO state on a process-wide 
cached client.
   
   - Create the filesystem outside the instance cache: 
`S3FileSystem(**s3_fs_kwargs, skip_instance_cache=True)`.
   - Cache it on the FileIO, shared by all its threads, instead of once per 
thread.
   - Register the signer once, before the filesystem is handed out, and never 
unregister it.
   
   Alternatively, include the signer identity in the cache key. That fixes the 
race but not the "last signer wins" problem across catalogs.
   
   To also cover client recreation, register the signer in an 
`S3FileSystem.set_session` override, or check it there.
   
   We currently work around it in our own `FsspecFileIO` subclass by doing 
exactly that: an own `S3FileSystem` subclass per FileIO, with 
`skip_instance_cache=True` and the signer registered in `set_session`. With 
that, the repro above shows 0 errors.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to