andygrove opened a new pull request, #4981: URL: https://github.com/apache/datafusion-comet/pull/4981
## Which issue does this PR close? Partially addresses #4842 (Item #9 in the remediation section: "Delete or implement the dead IO configs (`parallel.io.*`, `mergeRanges`) so benchmarking isn't misled"). ## Rationale for this change The following configs are defined in `CometConf.scala` but referenced nowhere in current native or JVM code. They were consumed by the JVM-side Parquet reader that has since been removed: - `spark.comet.parquet.read.parallel.io.enabled` (default `true`) - `spark.comet.parquet.read.parallel.io.thread-pool.size` (default `16`) - `spark.comet.parquet.read.io.mergeRanges` (default `true`) - `spark.comet.parquet.read.io.mergeRanges.delta` (default `8 MB`) - `spark.comet.parquet.read.io.adjust.readRange.skew` (default `false`) These are a trap: they appear tunable but change nothing. Anyone benchmarking "Comet with IO tuning" is tuning a no-op. ## What changes are included in this PR? - Delete the five dead config entries from `CometConf.scala`. - Delete the stale `ParquetReadSuite` "test merge scan range" test, which only asserted that the (dead) config propagated to the Hadoop conf. The auto-generated `configs.md` will be regenerated on the next docs build and will no longer list these entries. ## How are these changes tested? Compile-checked against `spark-3.5`. No behavior change — these configs were no-ops in current code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
