[
https://issues.apache.org/jira/browse/YARN-535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620974#comment-13620974
]
Steve Loughran commented on YARN-535:
-------------------------------------
{code}
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Time elapsed: 4137 sec <<< ERROR!
java.lang.RuntimeException: Error parsing 'yarn-site.xml' :
org.xml.sax.SAXParseException: Premature end of file.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2050)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1899)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1816)
at
org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:465)
at
org.apache.hadoop.conf.Configuration.asXmlDocument(Configuration.java:2127)
at
org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:2096)
at
org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:2086)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.setup(TestUnmanagedAMLauncher.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:246)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:153)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1887)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1875)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1946)
... 29 more
{code}
This stack trace is a failure to read the file yarn-site.xml, which is actually
being written on line 63 of TestUnmanagedAMLauncher -a file that
is already open for writing.
It is possible that some filesystems (here, HFS+) make that write visible while
it is still
going on, triggering a failure which then corrupts later builds at init time
{code}
$ ls -l target/test-classes/yarn-site.xml
-rw-r--r-- 1 stevel staff 0 3 Apr 15:37 target/test-classes/yarn-site.xml
{code}
This is newer than the one in test/properties, so Maven doesn't fix it next
test run
{code}
$ ls -l src/test/resources/yarn-site.xml
-rw-r--r--@ 1 stevel staff 830 28 Nov 16:29 src/test/resources/yarn-site.xml
{code}
as a result, follow on tests fail when MiniYARNCluster tries to read it.
{code}
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Time elapsed: 515 sec <<< ERROR!
java.lang.RuntimeException: Error parsing 'yarn-site.xml' :
org.xml.sax.SAXParseException: Premature end of file.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2050)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1899)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1816)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:719)
at
org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:738)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:948)
at org.apache.hadoop.security.Groups.<init>(Groups.java:62)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:244)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:213)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:674)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:582)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:458)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:439)
at
org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:425)
at
org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:93)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.setup(TestUnmanagedAMLauncher.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
{code}
# This test should copy over the new yarn-site.xml rather than overwrite the
live one
# there should be a yarn-site-template.xml that is copied over
# the {{Configuration.loadResource()}} could skip size 0 files instead of
trying to parse them.
> TestUnmanagedAMLauncher can corrupt target/test-classes/yarn-site.xml during
> write phase, breaks later test runs
> ----------------------------------------------------------------------------------------------------------------
>
> Key: YARN-535
> URL: https://issues.apache.org/jira/browse/YARN-535
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications
> Affects Versions: 3.0.0
> Environment: OS/X laptop, HFS+ filesystem
> Reporter: Steve Loughran
> Priority: Minor
>
> the setup phase of {{TestUnmanagedAMLauncher}} overwrites {{yarn-site.xml}}.
> As {{Configuration.writeXml()}} does a reread of all resources, this will
> break if the (open-for-writing) resource is already visible as an empty file.
> This leaves a corrupted {{target/test-classes/yarn-site.xml}}, which breaks
> later test runs -because it is not overwritten by later incremental builds,
> due to timestamps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
