[
https://issues.apache.org/jira/browse/HADOOP-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745554#comment-15745554
]
Tony Wu commented on HADOOP-13897:
----------------------------------
Hi [~steve_l], thanks a lot for your response.
{{TestAdlFileContextMainOperationsLive}} does setup the FC to use ADL as
default FS:
{code}
public class TestAdlFileContextMainOperationsLive
extends FileContextMainOperationsBaseTest {
...
@Override
public void setUp() throws Exception {
Configuration conf = AdlStorageConfiguration.getConfiguration();
String fileSystem = conf.get(KEY_FILE_SYSTEM);
if (fileSystem == null || fileSystem.trim().length() == 0) {
throw new Exception("Default file system not configured.");
}
URI uri = new URI(fileSystem);
FileSystem fs = AdlStorageConfiguration.createStorageConnector();
fc = FileContext.getFileContext(
new DelegateToFileSystem(uri, fs, conf, fs.getScheme(), false) {
}, conf);
super.setUp();
}
{code}
However the test that's failing is creating a second FC, with default config:
{code}
@Test
/*
* Test method
* org.apache.hadoop.fs.FileContext.getFileContext(AbstractFileSystem)
*/
public void testGetFileContext1() throws IOException {
final Path rootPath = getTestRootPath(fc, "test");
AbstractFileSystem asf = fc.getDefaultFileSystem();
// create FileContext using the protected #getFileContext(1) method:
////<<<<<<<<<<<<<<
FileContext fc2 = FileContext.getFileContext(asf); ////<<<<<<<<<<<<<< 2nd
FC created
////>>>>>>>>>>>>>>
// Now just check that this context can do something reasonable:
final Path path = new Path(rootPath, "zoo");
FSDataOutputStream out = fc2.create(path, EnumSet.of(CREATE),
Options.CreateOpts.createParent());
out.close();
Path pathResolved = fc2.resolvePath(path);
assertEquals(pathResolved.toUri().getPath(), path.toUri().getPath());
}
{code}
{{FileContext.getFileContext()}} uses the default configuration:
{code}
/**
* Create a FileContext for specified file system using the default config.
*
* @param defaultFS
* @return a FileContext with the specified AbstractFileSystem
* as the default FS.
*/
protected static FileContext getFileContext(
final AbstractFileSystem defaultFS) {
return getFileContext(defaultFS, new Configuration());
}
{code}
It looks like {{TestAdlFileContextMainOperationsLive#testGetFileContext1}}
should be using {{AdlStorageConfiguration.getConfiguration()}} to create
{{fc2}}. Or maybe {{testGetFileContext1}} should be omitted as the protected
API {{FileContext#getFileContext(final AbstractFileSystem defaultFS)}} does not
appear to be used anywhere else but this test case.
The rest of {{TestAdlFileContextMainOperationsLive}} and all other
{{hadoop-azure-datalake}} live tests passes successfully during my test.
> TestAdlFileContextMainOperationsLive#testGetFileContext1 fails consistently
> ---------------------------------------------------------------------------
>
> Key: HADOOP-13897
> URL: https://issues.apache.org/jira/browse/HADOOP-13897
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.0.0-alpha2
> Reporter: Tony Wu
>
> {{TestAdlFileContextMainOperationsLive#testGetFileContext1}} (this is a live
> test against Azure Data Lake Store) fails consistently with the following
> error:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.55 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
> testGetFileContext1(org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive)
> Time elapsed: 11.229 sec <<< ERROR!
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:136)
> at
> org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
> at
> org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at
> org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
> at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
> at
> org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134)
> at
> org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
> at
> org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at
> org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
> at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
> at
> org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.IllegalArgumentException: No value for
> dfs.adls.oauth2.access.token.provider found in conf file.
> at
> org.apache.hadoop.fs.adl.AdlFileSystem.getNonEmptyVal(AdlFileSystem.java:935)
> at
> org.apache.hadoop.fs.adl.AdlFileSystem.getCustomAccessTokenProvider(AdlFileSystem.java:207)
> at
> org.apache.hadoop.fs.adl.AdlFileSystem.getAccessTokenProvider(AdlFileSystem.java:241)
> at
> org.apache.hadoop.fs.adl.AdlFileSystem.initialize(AdlFileSystem.java:151)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.<init>(DelegateToFileSystem.java:52)
> at org.apache.hadoop.fs.adl.Adl.<init>(Adl.java:39)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134)
> at
> org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
> at
> org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at
> org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
> at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
> at
> org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.testGetFileContext1:1350
> ยป Runtime
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
