This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 50eaaf723 [CELEBORN-1450] MRAppMasterWithCeleborn should get
FileSystem via mapreduce.job.dir for HDFS federation
50eaaf723 is described below
commit 50eaaf723f3cda528b9c0252a8d5cc193ce71397
Author: SteNicholas <[email protected]>
AuthorDate: Wed Jun 5 10:04:56 2024 +0800
[CELEBORN-1450] MRAppMasterWithCeleborn should get FileSystem via
mapreduce.job.dir for HDFS federation
### What changes were proposed in this pull request?
`MRAppMasterWithCeleborn` gets `FileSystem` via `mapreduce.job.dir` for
HDFS federation.
### Why are the changes needed?
`MRAppMasterWithCeleborn` should get `FileSystem` via `mapreduce.job.dir`
for HDFS federation. Otherwise, the Celeborn conf path of `celeborn.xml` is
wrong for the following example configuration:
```
<property>
<name>fs.defaultFS</name>
<value>viewfs://bigdata-cluster</value>
</property>
<property>
<name>fs.viewfs.mounttable.bigdata-cluster.link./application</name>
<value>hdfs://bigdata-proxy-ns/application</value>
</property>
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Cluster test.
Closes #2545 from SteNicholas/CELEBORN-1450.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: mingji <[email protected]>
---
.../org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java
b/client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java
index fc5e0b3a4..130f4e8fb 100644
---
a/client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java
+++
b/client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java
@@ -81,9 +81,9 @@ public class MRAppMasterWithCeleborn extends MRAppMaster {
private void writeLifecycleManagerConfToTask(JobConf conf, JobConf lmConf)
throws CelebornIOException {
try {
- FileSystem fs = FileSystem.get(conf);
String jobDirStr = conf.get(MRJobConfig.MAPREDUCE_JOB_DIR);
Path celebornConfPath = new Path(jobDirStr,
HadoopUtils.MR_CELEBORN_CONF);
+ FileSystem fs = celebornConfPath.getFileSystem(conf);
try (FSDataOutputStream out =
FileSystem.create(
