ump45nose opened a new pull request, #7098: URL: https://github.com/apache/datafusion-comet/pull/7098
## Which issue does this PR close? Closes #7091. ## Rationale for this change When a native plan's spill files reach `spark.comet.maxTempDirectorySize`, the task fails with DataFusion's message, which tells the user to raise `datafusion.runtime.max_temp_directory_size`. That setting does nothing in Comet (and with `spark.comet.exec.respectDataFusionConfigs` on it can make plan creation panic, as @viirya found), so the error points users at the wrong knob. ## What changes are included in this PR? Following @viirya's suggestion on the issue, the hint is added where the JVM exception message is built, in `natural_exception` in `native/jni-bridge/src/errors.rs`, rather than in `From<DataFusionError> for CometError`. Both the plain `DataFusion` path and the `DataFusionError::External` fallback end up there, and the hint is not lost when the error round-trips through `DataFusionError`. DataFusion raises this error as a plain `std::io::Error`, so the match is on the `` `datafusion.runtime.max_temp_directory_size` `` key in the message. The DataFusion sentence is kept, and the hint after it says that `datafusion.runtime.*` settings don't apply in Comet, names `spark.comet.maxTempDirectorySize`, and says the limit is per native plan, so an executor can use several times that much disk. ## How are these changes tested? `spill_limit_error_names_the_comet_config` gets the error from a real `DiskManager` with a 10-byte limit instead of copying DataFusion's text, so a rewording in a future DataFusion upgrade will fail the test rather than silently drop the hint. It wraps the error as `IoError`, `ArrowError` and `External`, checks the full JVM message for each, and checks that an unrelated IO error message is unchanged. The test fails without the change. I ran the `datafusion-comet-jni-bridge` lib tests (the four tests that start a JVM need the Maven-built `comet-common` classes, which I didn't build) and `cargo clippy --all-targets -- -D warnings` on that crate. AI disclosure: the change, the test and this description were written with an AI coding assistant. The hint wording is the part most worth a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
