[
https://issues.apache.org/jira/browse/HADOOP-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344201#comment-16344201
]
Abraham Fine commented on HADOOP-15140:
---------------------------------------
[[email protected]] I went ahead and added two tests to
{{AbstractContractGetFileStatusTest}}:
{code:java}
@Test
public void testGetFileStatusRootURI() throws Throwable {
String fileSystemURI = getFileSystem().getUri().toString();
assertTrue("uri should not end with '/': " + fileSystemURI,
fileSystemURI.endsWith("//") || !fileSystemURI.endsWith("/"));
ContractTestUtils.assertIsDirectory(
getFileSystem().getFileStatus(new Path(fileSystemURI)));
}
@Test
public void testGetFileStatusRootFromChild() throws Throwable {
ContractTestUtils.assertIsDirectory(
getFileSystem().getFileStatus(new Path("/dir").getParent()));
}
{code}
These tests ran against S3, Azure, Azure Data Lake, and HDFS using: {{mvn test
-Dtest="**/*ContractGetFileStatus*" -DS3guard -fae
-Dmaven.test.failure.ignore=true}}
{{testGetFileStatusRootFromChild}} never appears to fail.
{{testGetFileStatusRootURI}} does on HDFS, S3, and Azure Data Lake (Azure
native passes). Local file systems also pass.
Here are the failures and their corresponding stack traces:
{code:java}
[INFO] Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.344
s <<< FAILURE! - in
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
[ERROR]
testGetFileStatusRootURI(org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus)
Time elapsed: 0.02 s <<< ERROR!
java.lang.IllegalArgumentException: Pathname from hdfs://localhost:63826 is
not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:242)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1568)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1565)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1580)
at
org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
113.029 s <<< FAILURE! - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[ERROR]
testGetFileStatusRootURI(org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus)
Time elapsed: 1.27 s <<< ERROR!
java.lang.IllegalArgumentException: path must be absolute
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
at
org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2130)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2070)
at
org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
110.904 s <<< FAILURE! - in
org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[ERROR]
testGetFileStatusRootURI(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List)
Time elapsed: 1.278 s <<< ERROR!
java.lang.IllegalArgumentException: path must be absolute
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
at
org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2130)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2070)
at
org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 43.848
s <<< FAILURE! - in
org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive
[ERROR]
testGetFileStatusRootURI(org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive)
Time elapsed: 1.052 s <<< ERROR!
com.microsoft.azure.datalake.store.ADLException:
Error getting info for file
path is null
Last encountered exception thrown after 1 tries. [HTTP0(null)]
[ServerRequestId:null]
at
com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1147)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getDirectoryEntry(ADLStoreClient.java:725)
at
org.apache.hadoop.fs.adl.AdlFileSystem.getFileStatus(AdlFileSystem.java:476)
at
org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
So it seems to me the current behavior in S3 is correct (perhaps the error
message could be improved) and we should work to make all the implementations
consistent. What do you think?
> S3guard mistakes root URI without / as non-absolute path
> --------------------------------------------------------
>
> Key: HADOOP-15140
> URL: https://issues.apache.org/jira/browse/HADOOP-15140
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Abraham Fine
> Priority: Major
>
> If you call {{getFileStatus("s3a://bucket")}} then S3Guard will throw an
> exception in putMetadata, as it mistakes the empty path for "non-absolute
> path"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
