[
https://issues.apache.org/jira/browse/HDFS-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782567#comment-17782567
]
Ayush Saxena commented on HDFS-17246:
-------------------------------------
Thanx [~gaurava] for the pointers. I think that code which is creating trouble
in windows was added as part of YARN-9568, which seems to have chosen a path
which is not HDFS compatible on Windows.
I think there was a similar problem solved as part of HDFS-13296.
Can you use the same util and see if the test works?
{noformat}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
index 6472a21f961..49370d801bb 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
@@ -103,6 +103,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.hadoop.test.GenericTestUtils.getTempPath;
import static
org.apache.hadoop.yarn.server.resourcemanager.resource.TestResourceProfiles.TEST_CONF_RESET_RESOURCE_TYPES;
/**
@@ -336,7 +337,7 @@ public void serviceInit(Configuration conf) throws
Exception {
// to ensure that any FileSystemNodeAttributeStore started by the RM always
// uses a unique path, if unset, force it under the test dir.
if (conf.get(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR) == null) {
- File nodeAttrDir = new File(getTestWorkDir(), "nodeattributes");
+ File nodeAttrDir = new File(getTempPath("nodeattributes"));
conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR,
nodeAttrDir.getCanonicalPath());
}
{noformat}
I think in the current fix, You are making HDFS allow {{:}} at first index in
case of Windows, I think we can change the path itself, rather than changing
the HDFS logic.
{{NodeAttributeTestUtils}} also lands up to that new method ultimately which
handles windows path for that conf here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeAttributeTestUtils.java#L45-L50]
> Fix shaded client for building Hadoop on Windows
> ------------------------------------------------
>
> Key: HDFS-17246
> URL: https://issues.apache.org/jira/browse/HDFS-17246
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 3.4.0
> Environment: Windows 10
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2023-11-03-17-31-14-990.png
>
>
> Currently, the *shaded client* Yetus personality in Hadoop fails to build on
> Windows -
> https://github.com/apache/hadoop/blob/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/dev-support/bin/hadoop.sh#L541-L615.
> This happens due to the integration test failures in Hadoop client modules -
> https://github.com/apache/hadoop/tree/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/hadoop-client-modules/hadoop-client-integration-tests.
> There are several issues that need to be addressed in order to get the
> integration tests working -
> # Set the HADOOP_HOME, needed by the Mini DFS and YARN clusters spawned by
> the integration tests.
> # Add Hadoop binaries to PATH, so that winutils.exe can be located.
> # Create a new user with Symlink privilege in the Docker image. This is
> needed for the proper working of Mini YARN cluster, spawned by the
> integration tests.
> # Fix a bug in DFSUtilClient.java that prevents colon ( *:* ) in the path.
> The colon is used a delimiter for the PATH variable while specifying multiple
> paths. However, this isn't a delimiter in the case of Windows and must be
> handled appropriately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
