LuciferYang commented on code in PR #40945:
URL: https://github.com/apache/spark/pull/40945#discussion_r1191844146
##########
core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:
##########
@@ -623,15 +623,13 @@ private[spark] object SparkHadoopUtil extends Logging {
fs.create(path)
} else {
try {
- // Use reflection as this uses APIs only available in Hadoop 3
- val builderMethod = fs.getClass().getMethod("createFile",
classOf[Path])
// the builder api does not resolve relative paths, nor does it create
parent dirs, while
// the old api does.
if (!fs.mkdirs(path.getParent())) {
throw new IOException(s"Failed to create parents of $path")
}
val qualifiedPath = fs.makeQualified(path)
- val builder = builderMethod.invoke(fs, qualifiedPath)
+ val builder = fs.createFile(qualifiedPath)
val builderCls = builder.getClass()
// this may throw a NoSuchMethodException if the path is not on hdfs
val replicateMethod = builderCls.getMethod("replicate")
Review Comment:
> I think this method is very private and only available for
`HdfsDataOutputStreamBuilder`. I'm OK if it stay as it is.
Can we confirm this?
My concern is if there is a `XDataOutputStreamBuilder`(Maybe user specific
implementation), if `XDataOutputStreamBuilder` is subclass of
`FSDataOutputStreamBuilder ` instead of `HdfsDataOutputStreamBuilder`, but
`XDataOutputStreamBuilder` also happens to have `replicate ` method.
In this scenario, the current way can be executed, but changing to similar
to
```
fs.createFile(qualifiedPath) match {
case hb: HdfsDataOutputStreamBuilder => hb.replicate().build()
case _ => fs.create(path)
}
```
will inadvertently disrupt this behavior, so I let it stay as it is in this
pr ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
