parthchandra commented on code in PR #7491: URL: https://github.com/apache/datafusion-comet/pull/7491#discussion_r4487491468
########## docs/source/user-guide/latest/iceberg.md: ########## @@ -194,6 +194,16 @@ For a custom S3-compatible endpoint, configure the catalog with the endpoint, pa These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION` or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials. +### Plan data per task + +The native reader sends each task only the delete files, partition values and residual filters +that its own files need, the way Spark's Iceberg reader does. A coalesce without a shuffle +(`coalesce(n)`, or a write that coalesces its input) combines several input partitions into one +task, and that task then carries a copy of a shared delete file for every input partition that +uses it. Scans where many partitions share delete files, such as tables with unpartitioned +equality deletes, can make coalesced tasks large. Compacting delete files, or repartitioning +instead of coalescing, avoids this. Review Comment: Agreed, this was a regression against the release for coalesced reads. Each delete file is now serialized once per scan, and every partition that uses it holds the same array and path string (IcebergPartitionPlanData). Java serialization writes it once per task stream, so a coalesced task carries one copy, and the driver keeps one copy too. The executor sets file_path_idx when it builds the partition's message, because the path pool order differs per partition. PlanDataInjectorSuite has a serialization test for the coalesced case, and the doc paragraph is updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
