sunchao commented on code in PR #5763:
URL: https://github.com/apache/datafusion-comet/pull/5763#discussion_r3960702998
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala:
##########
@@ -129,8 +128,11 @@ object CometDataWritingCommand extends
CometOperatorSerde[DataWritingCommandExec
// Collect S3/cloud storage configurations
val session = op.session
val hadoopConf =
session.sessionState.newHadoopConfWithOptions(cmd.options)
+ // `outputPath` is `Path.toString`, which is not a valid URI string: it
leaves spaces and
+ // literal `%` unescaped, so `URI.create` would throw (and the catch
below would silently
+ // give the write back to Spark). Going through `Path` escapes them
again.
val objectStoreOptions =
- NativeConfig.extractObjectStoreOptions(hadoopConf,
URI.create(outputPath))
+ NativeConfig.extractObjectStoreOptions(hadoopConf,
cmd.outputPath.toUri)
Review Comment:
### Correctness
[P2] Keep escaped HDFS destinations on Spark until native path decoding is
fixed
This removes the Spark 3.x fallback for HDFS destinations containing spaces,
but the native HDFS path is still different from the committer's path. With
local native input and a cold HDFS object-store cache,
`create_hdfs_object_store` passes `url.path()` to
`object_store::path::Path::parse`, retaining `dir%20with%20space`.
`ParquetWriter` then sends that path through OpenDAL/libhdfs, while Spark
commits the original `dir with space` staging directory. Data can therefore
remain outside the committed output even though job commit succeeds. The
local-path regression test does not exercise this branch. Please decline such
HDFS writes during planning until native path handling preserves Hadoop
filenames on both cold and warm cache paths. This is a source-confirmed
destination mismatch. A Hadoop committer component probe confirmed the commit
consequence, but I did not run an end-to-end HDFS write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
