This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 01600ae [SPARK-38652][K8S] `uploadFileUri` should preserve file scheme
01600ae is described below
commit 01600aeded50f4e9751ef48de5e7ebf2493bfb2c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Mar 30 08:26:41 2022 -0700
[SPARK-38652][K8S] `uploadFileUri` should preserve file scheme
### What changes were proposed in this pull request?
This PR replaces `new Path(fileUri.getPath)` with `new Path(fileUri)`.
By using `Path` class constructor with URI parameter, we can preserve file
scheme.
### Why are the changes needed?
If we use, `Path` class constructor with `String` parameter, it loses file
scheme information.
Although the original code works so far, it fails at Apache Hadoop 3.3.2
and breaks dependency upload feature which is covered by K8s Minikube
integration tests.
```scala
test("uploadFileUri") {
val fileUri = org.apache.spark.util.Utils.resolveURI("/tmp/1.txt")
assert(new Path(fileUri).toString == "file:/private/tmp/1.txt")
assert(new Path(fileUri.getPath).toString == "/private/tmp/1.txt")
}
```
### Does this PR introduce _any_ user-facing change?
No, this will prevent a regression at Apache Spark 3.3.0 instead.
### How was this patch tested?
Pass the CIs.
In addition, this PR and #36009 will recover K8s IT `DepsTestsSuite`.
Closes #36010 from dongjoon-hyun/SPARK-38652.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit cab8aa1c4fe66c4cb1b69112094a203a04758f76)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
index 0c8d964..9ab6d30 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
@@ -320,7 +320,7 @@ object KubernetesUtils extends Logging {
fs.mkdirs(new Path(s"${uploadPath}/${randomDirName}"))
val targetUri =
s"${uploadPath}/${randomDirName}/${fileUri.getPath.split("/").last}"
log.info(s"Uploading file: ${fileUri.getPath} to dest:
$targetUri...")
- uploadFileToHadoopCompatibleFS(new Path(fileUri.getPath), new
Path(targetUri), fs)
+ uploadFileToHadoopCompatibleFS(new Path(fileUri), new
Path(targetUri), fs)
targetUri
} catch {
case e: Exception =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
