andygrove commented on issue #7451: URL: https://github.com/apache/datafusion-comet/issues/7451#issuecomment-6607146128
I read the source to check the AAL table above (AAL `v1.3.1`, Hadoop `rel/release-3.5.0`, parquet-java `apache-parquet-1.17.0`, Spark `v4.2.0`). Several rows need correcting. - **Footer prefetch.** For files up to 1 GiB, AAL reads the last 32 KiB and the 1 MiB before it, which holds the page index. Above 1 GiB it reads 1 MiB plus 8 MiB ([LogicalIOConfiguration.java#L36-L40](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/logical/LogicalIOConfiguration.java#L36-L40)). A file no bigger than the two ranges is read whole. - **Parsed footer store (45 entries).** It only feeds AAL's own column prefetch ([ParquetColumnPrefetchStore.java#L130-L137](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/logical/impl/ParquetColumnPrefetchStore.java#L130-L137)). parquet-java parses the footer itself, from the tail bytes that AAL holds in memory. - **Parquet-aware row-group prefetch.** It is inactive in Spark 4.2: - parquet-java 1.17 reads row groups with `readVectored` by default ([ParquetInputFormat.java#L168-L173](https://github.com/apache/parquet-java/blob/apache-parquet-1.17.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L168-L173)). - AAL's Parquet logic hooks only `read(byte[], ...)` ([ParquetLogicalIOImpl.java#L75-L82](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/logical/impl/ParquetLogicalIOImpl.java#L75-L82)). `readVectored` goes straight to physical I/O ([DefaultLogicalIOImpl.java#L120-L136](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/logical/impl/DefaultLogicalIOImpl.java#L120-L136)). - The prefetch also never reaches the next row group ([ParquetUtils.java#L133-L139](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/logical/parquet/ParquetUtils.java#L133-L139)). - **Where the faster data reads come from.** AAL's physical I/O does the work: - Vectored ranges merge across gaps of up to 1 MiB. - Contiguous reads over about 11 MiB are split into 8 MiB GETs ([PhysicalIOConfiguration.java#L47-L52](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/physical/PhysicalIOConfiguration.java#L47-L52)). - Reads run on a 96-thread pool with no per-stream cap. - The classic stream merges only gaps under 128 KiB, caps a merged read at 2 MiB, and keeps 4 ranges in flight ([Constants.java#L1610-L1633](https://github.com/apache/hadoop/blob/rel/release-3.5.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L1610-L1633)). - **Small-object prefetch.** Confirmed: files up to 8 MiB are read with one GET when they are opened. - **Block cache.** There is one per S3A filesystem instance, keyed by path and etag, holding up to 2 GiB. Entries expire 1 s after their last access, and cleanup runs every 5 s ([PhysicalIOConfiguration.java#L34-L46](https://github.com/awslabs/analytics-accelerator-s3/blob/v1.3.1/input-stream/src/main/java/software/amazon/s3/analyticsaccelerator/io/physical/PhysicalIOConfiguration.java#L34-L46)). - **Metadata (HEAD) cache.** It saves nothing here. Spark 4.2 calls `HadoopInputFile.fromPath` for every split ([ParquetFooterReader.java#L100](https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetFooterReader.java#L100)), which does a `getFileStatus` HEAD ([HadoopInputFile.java#L59-L62](https://github.com/apache/parquet-java/blob/apache-parquet-1.17.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/HadoopInputFile.java#L59-L62)). S3A then passes the etag on to AAL. - **S3 client.** AAL uses S3A's synchronous client (Apache HTTP), not the CRT client ([AnalyticsStreamFactory.java#L117-L120](https://github.com/apache/hadoop/blob/rel/release-3.5.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/streams/AnalyticsStreamFactory.java#L117-L120)). These are the serial S3 round trips per file that follow from this, traced from source rather than measured: | Read | Classic | AAL | Comet native scan | |---|---|---|---| | 5 KB single-file table | HEAD + 4-5 GETs | HEAD + 1 GET | 2 GETs (footer, then data) | | 128 MB split, 3 of 20 columns, footer and page index within 512 KiB | 5 or more | 3 (HEAD, tail, data) | 2 (tail, data) | AAL brings Spark close to the request pattern Comet's native scan already had. That is why the Spark baseline improved and Comet's time did not. On question 1 above, Comet's scan still lacks three things that AAL does: - A single GET for a small file. - Parallel 8 MiB parts for large reads. - A larger tail read for big files. #7459 tracks those changes and the measurements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
