[
https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620969#comment-13620969
]
Steve Loughran commented on MAPREDUCE-3502:
-------------------------------------------
{code}
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Time elapsed: 4137 sec <<< ERROR!
java.lang.RuntimeException: Error parsing 'yarn-site.xml' :
org.xml.sax.SAXParseException: Premature end of file.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2050)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1899)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1816)
at
org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:465)
at
org.apache.hadoop.conf.Configuration.asXmlDocument(Configuration.java:2127)
at
org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:2096)
at
org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:2086)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.setup(TestUnmanagedAMLauncher.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:246)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:153)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1887)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1875)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1946)
... 29 more
{code}
This stack trace is a failure to read the file yarn-site.xml, which is actually
being written on line 63 of TestUnmanagedAMLauncher -a file that
is already open for writing.
It is possible that some filesystems (here, HFS+) make that write visible while
it is still
going on, triggering a failure which then corrupts later builds at init time
{code}
$ ls -l target/test-classes/yarn-site.xml
-rw-r--r-- 1 stevel staff 0 3 Apr 15:37 target/test-classes/yarn-site.xml
{code}
This is newer than the one in test/properties, so Maven doesn't fix it next
test run
{code}
$ ls -l src/test/resources/yarn-site.xml
-rw-r--r--@ 1 stevel staff 830 28 Nov 16:29 src/test/resources/yarn-site.xml
{code}
as a result, follow on tests fail when MiniYARNCluster tries to read it.
{code}
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Time elapsed: 515 sec <<< ERROR!
java.lang.RuntimeException: Error parsing 'yarn-site.xml' :
org.xml.sax.SAXParseException: Premature end of file.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2050)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1899)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1816)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:719)
at
org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:738)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:948)
at org.apache.hadoop.security.Groups.<init>(Groups.java:62)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:244)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:213)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:674)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:582)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:458)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:439)
at
org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:425)
at
org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:93)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.setup(TestUnmanagedAMLauncher.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
{code}
# This test should copy over the new yarn-site.xml rather than overwrite the
live one
# there should be a yarn-site-template.xml that is copied over
# the {{Configuration.loadResource()}} could skip size 0 files instead of
trying to parse them.
> Review all Service.stop() operations and make sure that they work before a
> service is started
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3502
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch
>
> Original Estimate: 24h
> Time Spent: 2.5h
> Remaining Estimate: 21.5h
>
> MAPREDUCE-3431 has shown that some of the key services's shutdown operations
> are not robust against being invoked before the service is started. They need
> to be by
> # not calling other things if the other things are null
> # not being re-entrant (i.e. make synchronized if possible),
> Maybe
> # have a StopService operation that only stops a service if it is live
> # factor out the is-running test from the base service class and make it a
> pre-check for all the child services, so they bail out sooner rather than
> later. This would be the best as it would be the one guaranteed to work
> consistently across all instances, so only one or two would need testing
> my first iteration will skip the sync though it's something to consider.
> Testing: try to create each instance; call stop() straight after
> construction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
