sunchao opened a new pull request, #7719: URL: https://github.com/apache/datafusion-comet/pull/7719
## Which issue does this PR close? Part of #7128. This covers two additional diagnostics alongside #7556, #7561, #7562 and #7565. ## Rationale for this change The native Parquet planner saves the original Spark file URL alongside the object key so a timestamp overflow can identify the failing file in a multi-file task. The reader copies that URL into the structured error sent back to Spark. If the URL carries credentials, they reach the task failure. Spark file listings usually supply credential-free paths, so this is defense in depth. [File attribution in #5891](https://github.com/apache/datafusion-comet/pull/5891) S3 store creation also logs the resolved configuration at debug level. Even a scan of a plain `s3://bucket/file.parquet` can print a credential from a custom `fs.s3a.endpoint` or its per-bucket override. The endpoint comes from Hadoop configuration independently of the file URL handled in #7556. ## What changes are included in this PR? - Remove userinfo, query and fragment from the diagnostic path when the planner constructs `SparkFilePath`. Keep the Azure container, Spark's original scheme and the encoded filename. The reader uses the normalized object key for I/O. - Log the bucket and the names of the resolved S3 settings. The builder receives the full settings for requests. ## How are these changes tested? A native scan test writes a synthetic Parquet file with a millisecond timestamp that overflows on conversion to microseconds. It plans and executes a scan with a synthetic query and fragment on the file URL, then checks the exact percent-encoded path in `ParquetTimestampOverflow`. A table test covers S3A aliases, a hostless alias, HDFS userinfo, ABFS containers and escaped path characters. Checked `rustfmt --check` and `git diff --check` locally. Test execution is delegated to hosted CI. The S3 log change was checked by inspecting the arguments to `debug!`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
