[
https://issues.apache.org/jira/browse/HDFS-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783690#comment-17783690
]
Gautham Banasandra edited comment on HDFS-17246 at 11/7/23 4:40 PM:
--------------------------------------------------------------------
[~ayushtkn] I tried re-running the test by setting -
{code:java}
conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR,
GenericTestUtils.getRandomizedTestDir().getAbsolutePath());
{code}
Here are the changes that I made on top of today's trunk branch -
{code}
PS D:\projects\github\apache\hadoop> git branch --show-current
trunk
PS D:\projects\github\apache\hadoop> git diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
index 71cff2e3915..84122509787 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
@@ -663,7 +663,8 @@ public static boolean isValidName(String src) {
String element = components[i];
if (element.equals(".") ||
// For Windows, we must allow the : in the drive letter.
- (!Shell.WINDOWS && i == 1 && element.contains(":")) ||
+// (!Shell.WINDOWS && i == 1 && element.contains(":")) ||
+ (element.contains(":")) ||
(element.contains("/"))) {
return false;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
index 6472a21f961..37892504a4a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
@@ -338,7 +338,7 @@ public void serviceInit(Configuration conf) throws
Exception {
if (conf.get(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR) == null) {
File nodeAttrDir = new File(getTestWorkDir(), "nodeattributes");
conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR,
- nodeAttrDir.getCanonicalPath());
+ GenericTestUtils.getRandomizedTestDir().getAbsolutePath());
}
super.serviceInit(
conf instanceof YarnConfiguration ? conf : new
YarnConfiguration(conf));
PS D:\projects\github\apache\hadoop>
{code}
Here's the test that I ran -
{code}
@REM shadedclient - Windows
mvn verify -fae^
-pl hadoop-client-modules/hadoop-client-integration-tests^
-Pdist^
-Dtar^
-Dtest=NoUnitTests^
-Dmaven.javadoc.skip=true^
-Dcheckstyle.skip=true^
-Dspotbugs.skip=true^
-Pnative-win^
-Dhttps.protocols=TLSv1.2^
-Drequire.openssl^
-Drequire.test.libhadoop^
-Dshell-executable=C:\PROGRA~1\Git\bin\bash.exe^
-Dopenssl.prefix=D:\projects\github\vcpkg\installed\x64-windows^
-Dcmake.prefix.path=D:\projects\github\vcpkg\installed\x64-windows^
-Dwindows.cmake.toolchain.file=D:\projects\github\vcpkg\scripts\buildsystems\vcpkg.cmake^
-Dwindows.cmake.build.type=RelWithDebInfo^
-Dwindows.build.hdfspp.dll=off^
-Dwindows.no.sasl=on^
-Duse.platformToolsetVersion=v142
{code}
It failed with the following error, indicating that the *:* wasn't allowed in
the path. Please refer to
[^failsafe-reports-for-trying-to-use-GenericTestUtils.zip] for full details.
{code}
Caused by: java.lang.IllegalArgumentException: Pathname
/H:/hadoop-client-modules/hadoop-client-integration-tests/target/test/data/9eyKX4U7Ew
from
H:/hadoop-client-modules/hadoop-client-integration-tests/target/test/data/9eyKX4U7Ew
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:258)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1497)
... 37 more
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITUseMiniCluster.clusterUp:84 » YarnRuntime
java.lang.IllegalArgumentException...
[ERROR] ITUseMiniCluster.clusterUp:84 » YarnRuntime
java.lang.IllegalArgumentException...
[INFO]
[ERROR] Tests run: 5, Failures: 0, Errors: 2, Skipped: 0
{code}
The following returns a relative path -
{code:java}
GenericTestUtils.getRandomizedTestDir()
{code}
The problem has got to do with the subsequent call to -
{code:java}
getAbsolutePath()
{code}
I don't think there's any way around this, unless we get MiniYARNCluster to
work with *hdfs://* paths.
was (Author: gautham):
[~ayushtkn] I tried re-running the test by setting -
{code:java}
conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR,
GenericTestUtils.getRandomizedTestDir().getAbsolutePath());
{code}
Here are the changes that I made on top of today's trunk branch -
{code}
PS D:\projects\github\apache\hadoop> git branch --show-current
trunk
PS D:\projects\github\apache\hadoop> git diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
index 71cff2e3915..84122509787 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
@@ -663,7 +663,8 @@ public static boolean isValidName(String src) {
String element = components[i];
if (element.equals(".") ||
// For Windows, we must allow the : in the drive letter.
- (!Shell.WINDOWS && i == 1 && element.contains(":")) ||
+// (!Shell.WINDOWS && i == 1 && element.contains(":")) ||
+ (element.contains(":")) ||
(element.contains("/"))) {
return false;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
index 6472a21f961..37892504a4a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
@@ -338,7 +338,7 @@ public void serviceInit(Configuration conf) throws
Exception {
if (conf.get(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR) == null) {
File nodeAttrDir = new File(getTestWorkDir(), "nodeattributes");
conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR,
- nodeAttrDir.getCanonicalPath());
+ GenericTestUtils.getRandomizedTestDir().getAbsolutePath());
}
super.serviceInit(
conf instanceof YarnConfiguration ? conf : new
YarnConfiguration(conf));
PS D:\projects\github\apache\hadoop>
{code}
Here's the test that I ran -
{code}
@REM shadedclient - Windows
mvn verify -fae^
-pl hadoop-client-modules/hadoop-client-integration-tests^
-Pdist^
-Dtar^
-Dtest=NoUnitTests^
-Dmaven.javadoc.skip=true^
-Dcheckstyle.skip=true^
-Dspotbugs.skip=true^
-Pnative-win^
-Dhttps.protocols=TLSv1.2^
-Drequire.openssl^
-Drequire.test.libhadoop^
-Dshell-executable=C:\PROGRA~1\Git\bin\bash.exe^
-Dopenssl.prefix=D:\projects\github\vcpkg\installed\x64-windows^
-Dcmake.prefix.path=D:\projects\github\vcpkg\installed\x64-windows^
-Dwindows.cmake.toolchain.file=D:\projects\github\vcpkg\scripts\buildsystems\vcpkg.cmake^
-Dwindows.cmake.build.type=RelWithDebInfo^
-Dwindows.build.hdfspp.dll=off^
-Dwindows.no.sasl=on^
-Duse.platformToolsetVersion=v142
{code}
It failed with the following error, indicating that the *:* wasn't allowed in
the path.
{code}
Caused by: java.lang.IllegalArgumentException: Pathname
/H:/hadoop-client-modules/hadoop-client-integration-tests/target/test/data/9eyKX4U7Ew
from
H:/hadoop-client-modules/hadoop-client-integration-tests/target/test/data/9eyKX4U7Ew
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:258)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1497)
... 37 more
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITUseMiniCluster.clusterUp:84 » YarnRuntime
java.lang.IllegalArgumentException...
[ERROR] ITUseMiniCluster.clusterUp:84 » YarnRuntime
java.lang.IllegalArgumentException...
[INFO]
[ERROR] Tests run: 5, Failures: 0, Errors: 2, Skipped: 0
{code}
The following returns a relative path -
{code:java}
GenericTestUtils.getRandomizedTestDir()
{code}
The problem has got to do with the subsequent call to -
{code:java}
getAbsolutePath()
{code}
I don't think there's any way around this, unless we get MiniYARNCluster to
work with *hdfs://* paths.
> Fix shaded client for building Hadoop on Windows
> ------------------------------------------------
>
> Key: HDFS-17246
> URL: https://issues.apache.org/jira/browse/HDFS-17246
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 3.4.0
> Environment: Windows 10
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: failsafe-reports-for-trying-to-use-GenericTestUtils.zip,
> image-2023-11-03-17-31-14-990.png
>
>
> Currently, the *shaded client* Yetus personality in Hadoop fails to build on
> Windows -
> https://github.com/apache/hadoop/blob/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/dev-support/bin/hadoop.sh#L541-L615.
> This happens due to the integration test failures in Hadoop client modules -
> https://github.com/apache/hadoop/tree/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/hadoop-client-modules/hadoop-client-integration-tests.
> There are several issues that need to be addressed in order to get the
> integration tests working -
> # Set the HADOOP_HOME, needed by the Mini DFS and YARN clusters spawned by
> the integration tests.
> # Add Hadoop binaries to PATH, so that winutils.exe can be located.
> # Create a new user with Symlink privilege in the Docker image. This is
> needed for the proper working of Mini YARN cluster, spawned by the
> integration tests.
> # Fix a bug in DFSUtilClient.java that prevents colon ( *:* ) in the path.
> The colon is used a delimiter for the PATH variable while specifying multiple
> paths. However, this isn't a delimiter in the case of Windows and must be
> handled appropriately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
