This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bb56226c10df [SPARK-45432][CORE] Remove deprecated Hadoop-2
`LocatedFileStatus` constructor
bb56226c10df is described below
commit bb56226c10df9d07540c97f62e31ee09981464cf
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 6 00:22:15 2023 -0700
[SPARK-45432][CORE] Remove deprecated Hadoop-2 `LocatedFileStatus`
constructor
### What changes were proposed in this pull request?
This PR aims to remove `Hadoop 2`'s `LocatedFileStatus` constructor which
is deprecated at Apache Hadoop 3.0.0.
### Why are the changes needed?
Since SPARK-42452 removes Hadoop 2 at Apache Spark 3.5.0, we can use Hadoop
3 API completely.
[HDFS-6984](https://issues.apache.org/jira/browse/HDFS-6984) deprecated
this constructor in favor of new constructor at Apache Hadoop 3.0.0-beta1
-
https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/LocatedFileStatus.html#LocatedFileStatus-long-boolean-int-long-long-long-org.apache.hadoop.fs.permission.FsPermission-java.lang.String-java.lang.String-org.apache.hadoop.fs.Path-org.apache.hadoop.fs.Path-org.apache.hadoop.fs.BlockLocation:A-
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43239 from dongjoon-hyun/SPARK-45432.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala
b/core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala
index bb03cac8f1f6..db1eeeef4712 100644
--- a/core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala
+++ b/core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala
@@ -272,7 +272,8 @@ private[spark] object HadoopFSUtils extends Logging {
}
}
val lfs = new LocatedFileStatus(f.getLen, f.isDirectory,
f.getReplication, f.getBlockSize,
- f.getModificationTime, 0, null, null, null, null, f.getPath,
locations)
+ f.getModificationTime, 0, null, null, null, null, f.getPath,
+ f.hasAcl, f.isEncrypted, f.isErasureCoded, locations)
if (f.isSymlink) {
lfs.setSymlink(f.getSymlink)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
