[
https://issues.apache.org/jira/browse/HADOOP-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126201#comment-18126201
]
ASF GitHub Bot commented on HADOOP-19973:
-----------------------------------------
vt921102 commented on PR #8693:
URL: https://github.com/apache/hadoop/pull/8693#issuecomment-6118911789
Thanks @anujmodi2021 — integration validation for `branch-3.4` is complete.
As with #8692 I ran this **on Windows**, and for this branch also **on JDK
8**:
`branch-3.4` is the last Hadoop line supporting Java 8, and the defect only
reproduces where `File.separator` is a backslash. So this exercises both the
target
platform and the target JDK for this backport.
### Environment
- Windows 11 Pro 24H2, 4 vCPU / 8 GiB
- **Eclipse Temurin JDK 1.8.0_504**, Apache Maven 3.9.16
- Visual Studio 2022 Build Tools, MSVC v143, Windows SDK 10.0.22621
- ADLS Gen2 (StorageV2, Standard_LRS), **HNS enabled**, **North Europe**,
auth = **SharedKey**
The storage account belongs to my employer, so I have not named it here;
happy to
share it with a committer privately if the project needs it on record.
### Commits tested
| Role | Commit |
|---|---|
| Patched | `28e36e8de29ecc7f2656f2589863915cbc029ddd` (this PR) |
| Baseline | `536b278f3eae32707dae7f9d73b55c254d58b535` (this PR's parent) |
### Command
```
mvn.cmd -B -ntp -Duse.platformToolsetVersion=v143 \
-pl hadoop-tools/hadoop-azure -Dparallel-tests=abfs -Dscale verify
```
### Results — patched vs baseline, same VM and account
| | Patched | Baseline |
|---|---|---|
| Tests run | **1531** | 1530 |
| Failures | **8** | 7 |
| Errors | **45** | 46 |
| Skipped | 260 | 260 |
| Duration | 14:23 | 16:01 |
| Unique failing entries | **39** | **39** |
Both exited 0, with 139 ABFS test classes started in each.
**No regression.** 38 of the 39 failing entries are identical. The two
differences
are retry-flaky and cancel each other out:
| Test | Patched | Baseline |
|---|---|---|
| `TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting` |
fails | passes |
| `ITestAzureBlobFileSystemRenameUnicode.testRenameFileUsingUnicode` |
passes | fails |
The first is a timing-based fuzzy-validation unit test; the second also
flaked in the
`branch-3.5` baseline. Neither relates to list-status path construction.
### The JUnit 4 adaptation works
```
TestAbfsClient: patched 3 tests, 0 failures
baseline 2 tests, 0 failures
```
The additional test is
`testGetVersionedFileStatusFromEntryUsesForwardSlash`. This is
the one place this backport differs from trunk — `branch-3.4` is still on
JUnit 4,
its `AbfsDfsClient` constructor takes six arguments and its
`AbfsClientContextBuilder`
has no `withFileSystemId` — so confirming it compiles and passes under JDK 8
is the
point of this run. The directly relevant suite is also clean in both runs:
```
ITestAzureBlobFileSystemListStatus: Tests run: 20, Failures: 0, Errors: 0,
Skipped: 7
```
### Failures present in both runs
| Cause | Tests |
|---|---|
| Account is SharedKey-only; these tests need a SAS token provider
configured | `ITestAzureBlobFileSystemAuthorization` (~26),
`ITestAzureBlobFileSystemChooseSAS` (3) |
| Ran in a service account context with no primary group |
`ITestAbfsIdentityTransformer`,
`ITestAzureBlobFileSystemBasics.testWorkingDirectory` |
| YARN mini-cluster does not start on this Windows host |
`ITestAbfsTerasort` |
| Pre-existing, unchanged between runs |
`ITestAzureBlobFileSystemDelete.testDeleteIdempotencyTriggerHttp404` |
### Notes for anyone reproducing this on `branch-3.4`
1. `branch-3.4` has no `mvnw.cmd`, so Maven must be installed separately.
2. On Build Tools rather than the full VS IDE,
`-Duse.platformToolsetVersion=v143`
is needed because `convert-ms-winutils` requires `devenv`.
3. Windows SDK must be pinned to 10.0.22621; winutils does not compile
against
10.0.26100 (`error C2733: 'GetFileInformationByName'`). I mentioned this
on #8692
and am happy to file it as its own JIRA.
### Caveats
1. SharedKey-only account, so SAS and OAuth code paths were not covered by
this run;
that accounts for most of the errors seen identically in both runs.
2. The suite ran in a service account context, causing the two failures
noted above.
3. The SDK pin affects the build harness only, not the code under test.
> Hadoop file system can not browse the folder that has children folder with
> colon in Windows
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-19973
> URL: https://issues.apache.org/jira/browse/HADOOP-19973
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.4.3, 3.5.1
> Reporter: Toan Le
> Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.4, 3.5.1
>
> Attachments: Screenshot 2026-02-11 at 11.40.36.png,
> test_hadoop_azure.zip
>
>
> Hi guys,
> I am facing an issue when browsing the children's folder name with a colon in
> ADLS.
> Here is my case. My root folder in ADLS contains a folder with a name that
> includes a colon.
> When I used the Hadoop file system. I got this error on Windows:
> {code:java}
> java -cp ".;./*" AbfsListStatusTest.java [main] WARN
> org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> [main] ERROR org.apache.hadoop.fs.azurebfs.services.AbfsClient - Unable to
> deserialize list results for Uri
> abfss://<containerName>@<storeAccountName>.dfs.core.windows.net
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0: \FestSpecial: at
> org.apache.hadoop.fs.Path.initialize(Path.java:264) at
> org.apache.hadoop.fs.Path.<init>(Path.java:222) at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getVersionedFileStatusFromEntry(AbfsClient.java:1846)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.parseListPathResults(AbfsDfsClient.java:1373)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.listPath(AbfsDfsClient.java:347)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1278)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1242)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1224)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:713)
> at AbfsListStatusOAuthTest.main(AbfsListStatusTest.java:45)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:484)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:208)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:135)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: \FestSpecial:
> at java.base/java.net.URI$Parser.fail(URI.java:2995)
> at java.base/java.net.URI$Parser.checkChars(URI.java:3166)
> at java.base/java.net.URI$Parser.checkChar(URI.java:3176)
> at java.base/java.net.URI$Parser.parse(URI.java:3191)
> at java.base/java.net.URI.<init>(URI.java:807)
> at org.apache.hadoop.fs.Path.initialize(Path.java:261)
> ... 14 more Exception in thread "main" Status code: -1 error code: null error
> message: Parsing of Json List Response Failed in
> DfsClient.java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Illegal character in scheme name at index 0: \FestSpecial: at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.parseListPathResults(AbfsDfsClient.java:1390)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.listPath(AbfsDfsClient.java:347)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1278)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1242)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1224)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:713)
> at AbfsListStatusOAuthTest.main(AbfsListStatusTest.java:45) Caused
> by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0: \FestSpecial: at
> org.apache.hadoop.fs.Path.initialize(Path.java:264) at
> org.apache.hadoop.fs.Path.<init>(Path.java:222) at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getVersionedFileStatusFromEntry(AbfsClient.java:1846)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.parseListPathResults(AbfsDfsClient.java:1373)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.listPath(AbfsDfsClient.java:347)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1278)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1242)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1224)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:713)
> at AbfsListStatusOAuthTest.main(AbfsListStatusTest.java:45)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:484)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:208)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:135)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: \FestSpecial: at
> java.base/java.net.URI$Parser.fail(URI.java:2995) at
> java.base/java.net.URI$Parser.checkChars(URI.java:3166) at
> java.base/java.net.URI$Parser.checkChar(URI.java:3176) at
> java.base/java.net.URI$Parser.parse(URI.java:3191) at
> java.base/java.net.URI.<init>(URI.java:807) at
> org.apache.hadoop.fs.Path.initialize(Path.java:261) ... 14 more
> {code}
> I tested my script, and it worked on Linux.
> Could you please help me check it?
> I compressed all my scripts ({_}AbfsListStatusTest.java{_}) and library into
> the ticket ({_}test_hadoop_azure.zip{_}).
> Here is the _AbfsListStatusTest.java:_
> {code:java}
> {code}
> _import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import java.net.URI;
> public class AbfsListStatusOAuthTest {
> public static void main(String[] args) throws Exception {
> // Inputs (use env vars or hardcode)
> String accountName = System.getenv("ADLS_STORAGE_ACCOUNT_NAME");
> String containerName = System.getenv("ADLS_CONTAINER_NAME");
> String tenantId = System.getenv("ADLS_TENANT_ID");
> String clientId = System.getenv("ADLS_CLIENT_ID");
> String clientSecret = System.getenv("ADLS_CLIENT_SECRET");
> String dirPath = (args.length > 0) ? args[0] : "/";
> // Use abfss for OAuth
> String fsUri = String.format("abfss://%s@%s.dfs.core.windows.net",
> containerName, accountName);
> String accountFqdn = accountName + ".dfs.core.windows.net";
> Configuration conf = new Configuration(false);
> // Ensure ABFS impls are known
> conf.set("fs.abfss.impl",
> "org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem");
> conf.set("fs.abfs.impl", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem");
> // — OAuth configs (scoped to the account host) —
> conf.set("fs.azure.account.auth.type." + accountFqdn, "OAuth");
> conf.set("fs.azure.account.oauth.provider.type." + accountFqdn,
> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider");
> conf.set("fs.azure.account.oauth2.client.id." + accountFqdn, clientId);
> conf.set("fs.azure.account.oauth2.client.secret." + accountFqdn,
> clientSecret);
> // Token endpoint (try v1 first — Hadoop ABFS commonly uses this)
> conf.set("fs.azure.account.oauth2.client.endpoint." + accountFqdn,
> "https://login.microsoftonline.com/"; + tenantId + "/oauth2/token");
> // Create FS and call listStatus
> FileSystem fs = FileSystem.get(new URI(fsUri), conf);
> FileStatus[] statuses = fs.listStatus(new Path(dirPath));
> System.out.println("Found " + statuses.length + " entries under " + dirPath);
> for (FileStatus st : statuses)
> { System.out.printf("%s isDir=%s len=%d%n", st.getPath(), st.isDirectory(),
> st.getLen()); }
> fs.close();
> }
> }_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
