Neuw84 opened a new pull request, #17281:
URL: https://github.com/apache/iceberg/pull/17281
…ites
Position delta writes (MERGE/UPDATE/DELETE in merge-on-read mode) broadcast
a map of every scanned data file with position deletes to its full DeleteFile
metadata. Executors only need enough of it to locate and read the previous
delete content (location, format, sizes, key metadata, DV offsets) and to
identify files back to the driver; partition data, column bounds and counts
ride along only to be echoed back for RowDelta.removeDeletes at commit time.
Under streaming upserts against v3 DV tables this map converges to one entry
per data file in the table and is rebuilt and re-broadcast every micro-batch,
adding driver and executor memory pressure that grows with table age.
Keep the full map driver-local, broadcast slim copies built via
FileMetadata.deleteFileBuilder (with referencedDataFile normalized so
file-scoped checks keep working without the file_path column bounds), and
resolve reported rewritten files back to their full metadata at commit time
before calling removeDeletes. The resolution step is required for correctness
of the slimming: manifest filtering uses the removed file's spec ID and
partition, which slim copies do not carry, so reported rewritten files must
never reach the RowDelta directly.
Measured serialized size of the broadcast map (1000 entries, Java
serialization - Spark's default broadcast serializer):
map contents full slim factor
DV entries (v3) 661 B/entry 466 B/entry 1.4x
v2 file-scoped entries with stats 1069 B/entry 581 B/entry 1.8x
The serialized win is bounded by what slimming cannot remove (the data file
location key and the referenced data file location dominate each entry); the
deserialized heap saving per executor is larger because the per-entry object
graph (partition data, stats maps, split offsets) is gone. Commit messages
shrink correspondingly, since executors echo the slim copies back as rewritten
files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
