mixermt commented on PR #5898: URL: https://github.com/apache/datafusion-comet/pull/5898#issuecomment-6696709799
Answers below; each change is its own commit. **1. Probe at scale** — the lookup was quadratic; fixed, and both bounds now count entries. - `missing` is a map lookup, not a scan of the listing per directory (which also rebuilt `Path.toString` per comparison). - Every child of a parent is still kept, by name now rather than full path. Keeping only the directories a scan reads would list the parent again for every scan that touches a partition not seen before, a paged round trip per 1000 children each time; keeping all of them costs one listing per table per hour. Spark's own `FileStatusCache` keeps a status per *file* of a partitioned file-source table, 250 MB by default; this holds a flag per directory. - What is kept is bounded to a million children in all, on the order of 100 MB at worst, least recently used parents evicted first; the 10000-parent cap is gone. - The per-scan budget counts entries: a million, which at HDFS's default page of 1000 (`dfs.ls.limit`) is the thousand round trips the old budget meant for small parents. The listing is paged (`listStatusIterator`), so a parent beyond the budget costs the budget and is not kept. - On a key-provider cluster the planning cost is one paged listing per parent per table per hour; `spark.comet.scan.icebergNative.hdfs.enabled=false` (3) removes it. - Test in `CometIcebergNativeScanSuite`: 50k partitions under one parent, one `listStatus`, no `getFileStatus`, asserted under ten seconds and running in well under one (the old lookup took minutes), 50k entries kept; the budget declines without keeping; the kept bound evicts the older table, whose next scan lists again. **2. Scaladoc** — `getTableUuid` moved above the block. **3. HDFS switch** — added `spark.comet.scan.icebergNative.hdfs.enabled`, default true. When false, a scan that reads HDFS files falls back with the key in the reason, the other HDFS checks and the probe are skipped, and other storage is unaffected. Planner test in `CometScanSchemeFallbackSuite`; documented in the HDFS section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
