workbench13 opened a new issue, #4414:
URL: https://github.com/apache/iceberg-python/issues/4414
### Apache Iceberg version
0.12.0 (latest release). The same code is on `main`.
### Please describe the bug 🐞
With a request signer configured (`s3.signer`, e.g. REST remote signing),
`FsspecFileIO` intermittently sends **unsigned** S3 requests. The store answers
403, which s3fs turns into `PermissionError: Forbidden`. It is usually a
`HeadObject`, because that is the most frequent call.
We saw this in long-running write jobs against a REST catalog (Lakekeeper)
with remote signing on an S3-compatible store. The 403s were rare and looked
random, and a retry always succeeded.
#### Cause
`pyiceberg.io.fsspec._s3` does the following:
```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function,
unique_id=1925)
```
1. **The S3FileSystem is shared.** `S3FileSystem(**s3_fs_kwargs)` comes from
fsspec's instance cache. The signer is not part of the cache key, so every
`FsspecFileIO` in the process with the same endpoint/region gets the **same**
`S3FileSystem` and the same aiobotocore client.
2. **The signer is swapped while requests are in flight.** Every call to
`_s3` unregisters the signer on that shared client and registers its own.
`FsspecFileIO.get_fs` caches per thread, so `_s3` runs once per FileIO per
thread, i.e. every time a newly loaded table is read or written from a worker
thread.
3. **The gap sends unsigned requests.** fsspec's IO thread keeps signing
other requests during the swap. A `before-sign.s3` emitted between `unregister`
and `register_last` finds no signer, and since `signature_version` is
`UNSIGNED`, the request goes out unsigned. Occasionally the two steps also race
each other and raise `KeyError: 1925` from botocore's `unregister`.
A second consequence, without any race: all FileIOs sign with the signer of
whichever FileIO called `_s3` last. That signer carries that FileIO's
properties: the signer URI/endpoint of that table and the `AuthManager` of that
catalog. A process with two catalogs (different principals or warehouses) can
therefore sign one catalog's requests with the other's credentials.
A related, smaller issue: the signer is registered on the client object. If
s3fs ever recreates its client (`set_session` does so when the aiohttp sessions
are closed), the new client has no signer.
#### Reproduction
The script below needs only MinIO, or any S3 that rejects unsigned requests:
```
docker run -p 9000:9000 -e MINIO_ROOT_USER=minio -e
MINIO_ROOT_PASSWORD=minio123 minio/minio server /data
```
It registers an in-process signer in `SIGNERS` that signs with SigV4, as a
REST signer would. Four threads `HEAD` one object while two threads create new
`FsspecFileIO`s and call `get_fs("s3")`, which is what loading a table and
reading it from a worker does.
<details><summary>repro.py</summary>
```python
"""Shared S3FileSystem + per-FileIO signer swap in pyiceberg.io.fsspec._s3.
Needs a MinIO server (or any S3 that rejects unsigned requests), e.g.:
docker run -p 9000:9000 -e MINIO_ROOT_USER=minio -e
MINIO_ROOT_PASSWORD=minio123 minio/minio server /data
"""
import collections
import sys
import threading
import time
import s3fs
from botocore.auth import S3SigV4Auth
from botocore.credentials import Credentials
from pyiceberg.io import fsspec as pf
ENDPOINT = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:9000"
KEY, SECRET = (sys.argv[2], sys.argv[3]) if len(sys.argv) > 3 else ("minio",
"minio123")
class InProcessSigner(pf.S3RequestSigner):
"""Stands in for S3V4RestSigner: signs the request the way a REST signer
would."""
def __call__(self, request, **_):
S3SigV4Auth(Credentials(KEY, SECRET), "s3",
"us-east-1").add_auth(request)
pf.SIGNERS["InProcessSigner"] = InProcessSigner
PROPS = {"s3.endpoint": ENDPOINT, "s3.region": "us-east-1", "s3.signer":
"InProcessSigner"}
admin = s3fs.S3FileSystem(key=KEY, secret=SECRET, skip_instance_cache=True,
client_kwargs={"endpoint_url": ENDPOINT,
"region_name": "us-east-1"})
if not admin.exists("repro"):
admin.mkdir("repro")
admin.pipe("repro/data.parquet", b"x" * 100)
# 1. Every FsspecFileIO with the same endpoint/region gets the same
S3FileSystem.
a, b = pf.FsspecFileIO({**PROPS, "table": "a"}), pf.FsspecFileIO({**PROPS,
"table": "b"})
print("same S3FileSystem for two FileIOs:", a.get_fs("s3") is b.get_fs("s3"))
def run(seconds, new_fileios):
fs = pf.FsspecFileIO(dict(PROPS)).get_fs("s3")
ok, errors, stop = [0], collections.Counter(), threading.Event()
def reader():
while not stop.is_set():
try:
fs.info("repro/data.parquet", refresh=True)
ok[0] += 1
except Exception as e: # noqa: BLE001
errors[f"{type(e).__name__}: {e}"[:60]] += 1
def loader():
# What happens whenever a table is loaded and then read/written from
a worker thread.
while not stop.is_set():
try:
pf.FsspecFileIO(dict(PROPS)).get_fs("s3")
except Exception as e: # noqa: BLE001
errors[f"loader {type(e).__name__}: {e}"[:60]] += 1
threads = [threading.Thread(target=reader) for _ in range(4)]
threads += [threading.Thread(target=loader) for _ in range(new_fileios)]
for t in threads:
t.start()
time.sleep(seconds)
stop.set()
for t in threads:
t.join()
return ok[0], dict(errors)
# 2. Unsigned requests while other threads create FileIOs.
print("readers only: ", run(20, 0))
print("readers + 2 FileIO loops:", run(20, 2))
```
</details>
Output (pyiceberg 0.12.0, s3fs/fsspec 2026.9.0, botocore 1.43.117,
aiobotocore 3.10.0, Python 3.13, macOS), two runs:
```
same S3FileSystem for two FileIOs: True
readers only: (44788, {})
readers + 2 FileIO loops: (486, {'PermissionError: Forbidden': 27, 'loader
KeyError: 1925': 4})
readers only: (43351, {})
readers + 2 FileIO loops: (493, {'PermissionError: Forbidden': 8, 'loader
KeyError: 1925': 7})
```
The loaders run in a tight loop to make the race frequent. In real workloads
the rate is much lower, which is why it showed up as rare 403s in long jobs.
#### Possible fix
When a signer is configured, don't put per-FileIO state on a process-wide
cached client.
- Create the filesystem outside the instance cache:
`S3FileSystem(**s3_fs_kwargs, skip_instance_cache=True)`.
- Cache it on the FileIO, shared by all its threads, instead of once per
thread.
- Register the signer once, before the filesystem is handed out, and never
unregister it.
Alternatively, include the signer identity in the cache key. That fixes the
race but not the "last signer wins" problem across catalogs.
To also cover client recreation, register the signer in an
`S3FileSystem.set_session` override, or check it there.
We currently work around it in our own `FsspecFileIO` subclass by doing
exactly that: an own `S3FileSystem` subclass per FileIO, with
`skip_instance_cache=True` and the signer registered in `set_session`. With
that, the repro above shows 0 errors.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]