idanhazan commented on issue #4031:
URL:
https://github.com/apache/iceberg-python/issues/4031#issuecomment-6863669746
We hit this in production on 0.12.0 and can confirm the cause.
**Shape:** one `overwrite()` replacing tens of thousands of data files
across many partitions of a table partitioned by `day(timestamp)` and an
identity column, so the delete predicate is a large `Or` of partition matches.
The manifest list is long, and the default thread pool evaluates the manifests
concurrently. The commit failed with `ValidationException: Missing required
files to delete: …` for files that were live in the parent snapshot. A large
predicate makes each `visit` long, which widens the window between
`self.partition_fields = partitions` and the leaf reads.
**Workaround** (until a fix lands): serialize only the evaluation, keeping
the manifest reads parallel, by wrapping `_build_manifest_evaluator` on the
`_OverwriteFiles` instance:
```python
from threading import Lock
def one_at_a_time(build):
lock = Lock()
def serialized(spec_id):
evaluate = build(spec_id)
def run(manifest):
with lock:
return evaluate(manifest)
return run
return serialized
overwrite = transaction.update_snapshot().overwrite()
vars(overwrite)["_build_manifest_evaluator"] =
one_at_a_time(overwrite._build_manifest_evaluator)
```
With this in place, the full run (~26,600 partitions in 11 overwrites)
committed with no false refusals.
**A fix upstream** could make `_ManifestEvalVisitor.eval` stateless, passing
the partition summaries through the visit rather than storing them on `self`,
or build one evaluator per call or thread in `_deleted_entries`. Either would
also cover any future caller that shares an evaluator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]