idanhazan commented on issue #4031:
URL: 
https://github.com/apache/iceberg-python/issues/4031#issuecomment-6863669746

   We hit this in production on 0.12.0 and can confirm the cause.
   
   **Shape:** one `overwrite()` replacing tens of thousands of data files 
across many partitions of a table partitioned by `day(timestamp)` and an 
identity column, so the delete predicate is a large `Or` of partition matches. 
The manifest list is long, and the default thread pool evaluates the manifests 
concurrently. The commit failed with `ValidationException: Missing required 
files to delete: …` for files that were live in the parent snapshot. A large 
predicate makes each `visit` long, which widens the window between 
`self.partition_fields = partitions` and the leaf reads.
   
   **Workaround** (until a fix lands): serialize only the evaluation, keeping 
the manifest reads parallel, by wrapping `_build_manifest_evaluator` on the 
`_OverwriteFiles` instance:
   
   ```python
   from threading import Lock
   
   def one_at_a_time(build):
       lock = Lock()
       def serialized(spec_id):
           evaluate = build(spec_id)
           def run(manifest):
               with lock:
                   return evaluate(manifest)
           return run
       return serialized
   
   overwrite = transaction.update_snapshot().overwrite()
   vars(overwrite)["_build_manifest_evaluator"] = 
one_at_a_time(overwrite._build_manifest_evaluator)
   ```
   
   With this in place, the full run (~26,600 partitions in 11 overwrites) 
committed with no false refusals.
   
   **A fix upstream** could make `_ManifestEvalVisitor.eval` stateless, passing 
the partition summaries through the visit rather than storing them on `self`, 
or build one evaluator per call or thread in `_deleted_entries`. Either would 
also cover any future caller that shares an evaluator.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to