sunchao opened a new pull request, #7719:
URL: https://github.com/apache/datafusion-comet/pull/7719

   ## Which issue does this PR close?
   
   Part of #7128. This covers two additional diagnostics alongside #7556, 
#7561, #7562 and #7565.
   
   ## Rationale for this change
   
   The native Parquet planner saves the original Spark file URL alongside the 
object key so a timestamp overflow can identify the failing file in a 
multi-file task. The reader copies that URL into the structured error sent back 
to Spark. If the URL carries credentials, they reach the task failure. Spark 
file listings usually supply credential-free paths, so this is defense in 
depth. [File attribution in 
#5891](https://github.com/apache/datafusion-comet/pull/5891)
   
   S3 store creation also logs the resolved configuration at debug level. Even 
a scan of a plain `s3://bucket/file.parquet` can print a credential from a 
custom `fs.s3a.endpoint` or its per-bucket override. The endpoint comes from 
Hadoop configuration independently of the file URL handled in #7556.
   
   ## What changes are included in this PR?
   
   - Remove userinfo, query and fragment from the diagnostic path when the 
planner constructs `SparkFilePath`. Keep the Azure container, Spark's original 
scheme and the encoded filename. The reader uses the normalized object key for 
I/O.
   - Log the bucket and the names of the resolved S3 settings. The builder 
receives the full settings for requests.
   
   ## How are these changes tested?
   
   A native scan test writes a synthetic Parquet file with a millisecond 
timestamp that overflows on conversion to microseconds. It plans and executes a 
scan with a synthetic query and fragment on the file URL, then checks the exact 
percent-encoded path in `ParquetTimestampOverflow`. A table test covers S3A 
aliases, a hostless alias, HDFS userinfo, ABFS containers and escaped path 
characters.
   
   Checked `rustfmt --check` and `git diff --check` locally. Test execution is 
delegated to hosted CI. The S3 log change was checked by inspecting the 
arguments to `debug!`.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to