comphead commented on code in PR #7491: URL: https://github.com/apache/datafusion-comet/pull/7491#discussion_r4485576747
########## docs/source/user-guide/latest/iceberg.md: ########## @@ -194,6 +194,16 @@ For a custom S3-compatible endpoint, configure the catalog with the endpoint, pa These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION` or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials. +### Plan data per task + +The native reader sends each task only the delete files, partition values and residual filters +that its own files need, the way Spark's Iceberg reader does. A coalesce without a shuffle +(`coalesce(n)`, or a write that coalesces its input) combines several input partitions into one +task, and that task then carries a copy of a shared delete file for every input partition that +uses it. Scans where many partitions share delete files, such as tables with unpartitioned +equality deletes, can make coalesced tasks large. Compacting delete files, or repartitioning +instead of coalescing, avoids this. Review Comment: Thanks for documenting this limit. As far as I can tell Spark's reader avoids it. A coalesced task writes all of its input partitions into one `ObjectOutputStream` (`CoalescedRDDPartition.writeObject` writes `parents` through `defaultWriteObject`), and Iceberg's `DeleteFileIndex.forDataFile` hands out references to the same `DeleteFile` objects, so a shared delete file is written once per task and later occurrences are back-references. The latest release ships it once too, in the common block. With the threshold gone, this now also applies to unpartitioned equality deletes, which every partition uses, so a coalesced task over such a table carries one copy per input partition where the release carries one. Could each partition hold its entries as references to one shared object per entry, rather than as copied bytes, so that Java serialization shares them within a task the same way? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
