[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685838#comment-16685838 ]
Steve Loughran commented on HADOOP-15872: ----------------------------------------- And, and this is where it gets really interesting, hadoop 3.2 & trunk without this patch get through all the tests, but getFileStatus() fails on the cli (hadoop fs -ls; cloudstore store diag) As far as I can tell, this is a change in the behaviour in the ADLS endpoint {code:java} 2018-11-13 22:16:15,835 [main] INFO diag.StoreDiag (DurationInfo.java:<init>(53)) - Starting: Creating filesystem 2018-11-13 22:16:15,936 [main] DEBUG azurebfs.AzureBlobFileSystem (AzureBlobFileSystem.java:initialize(103)) - Initializing AzureBlobFileSystem for abfs://stevel-test...@account.blob.core.windows.net/ 2018-11-13 22:16:16,161 [main] DEBUG services.AbfsClientThrottlingIntercept (AbfsClientThrottlingIntercept.java:initializeSingleton(62)) - Client-side throttling is enabled for the ABFS file system. 2018-11-13 22:16:16,174 [main] INFO diag.StoreDiag (DurationInfo.java:close(100)) - Creating filesystem: duration 0:00:341 AzureBlobFileSystem{uri=abfs://stevel-test...@account.blob.core.windows.net, user='stevel', primaryUserGroup='staff'} 2018-11-13 22:16:16,174 [main] INFO diag.StoreDiag (DurationInfo.java:<init>(53)) - Starting: GetFileStatus abfs://stevel-test...@account.blob.core.windows.net/ 2018-11-13 22:16:16,174 [main] DEBUG azurebfs.AzureBlobFileSystem (AzureBlobFileSystem.java:getFileStatus(389)) - AzureBlobFileSystem.getFileStatus path: abfs://stevel-test...@account.blob.core.windows.net/ 2018-11-13 22:16:16,175 [main] DEBUG azurebfs.AzureBlobFileSystem (AzureBlobFileSystem.java:performAbfsAuthCheck(1025)) - ABFS authorizer is not initialized. No authorization check will be performed. 2018-11-13 22:16:16,175 [main] DEBUG azurebfs.AzureBlobFileSystemStore (AzureBlobFileSystemStore.java:getIsNamespaceEnabled(178)) - getFilesystemProperties for filesystem: stevel-testing 2018-11-13 22:16:16,964 [main] DEBUG services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(192)) - HttpRequest: 400,,cid=5b89e9e3-b87e-4329-9451-8326331e26b0,rid=3a5caef2-401e-005c-529e-7bb87e000000,sent=0,recv=0,HEAD,https://ACCOUNT.blob.core.windows.net/stevel-testing?resource=filesystem&timeout=90 2018-11-13 22:16:16,969 [main] INFO diag.StoreDiag (DurationInfo.java:close(100)) - GetFileStatus abfs://stevel-test...@account.blob.core.windows.net/: duration 0:00:794 Operation failed: "The value for one of the HTTP headers is not in the correct format.", 400, HEAD, https://ACCOUNT.blob.core.windows.net/stevel-testing?resource=filesystem&timeout=90 at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:134) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getFilesystemProperties(AbfsClient.java:197) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:181) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:454) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:395) at org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:692) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:386) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:331) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:989) at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:998) at storediag.main(storediag.java:24) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) 2018-11-13 22:16:16,971 [main] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status -1: Operation failed: "The value for one of the HTTP headers is not in the correct format.", 400, HEAD, https://ACCOUNT.blob.core.windows.net/stevel-testing?resource=filesystem&timeout=90 2018-11-13 22:16:16,973 [shutdown-hook-0] DEBUG azurebfs.AzureBlobFileSystem (AzureBlobFileSystem.java:close(383)) - AzureBlobFileSystem.close {code} > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > ------------------------------------------------------------- > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure > Affects Versions: 3.2.0 > Reporter: Thomas Marquardt > Assignee: junhua gu > Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org