[
https://issues.apache.org/jira/browse/YARN-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-11204:
------------------------------
Component/s: log-aggregation
(was: test)
> Various MapReduce tests fail with NPE in
> AggregatedLogDeletionService.stopRMClient
> ----------------------------------------------------------------------------------
>
> Key: YARN-11204
> URL: https://issues.apache.org/jira/browse/YARN-11204
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 3.4.0
> Reporter: Szilard Nemeth
> Assignee: Szilard Nemeth
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments:
> hadoop-mapreduce-project_hadoop-mapreduce-client_testlogs.txt,
> testAllOpportunisticMaps_logs.txt
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> During testing of HADOOP-15327, I noticed that lots of unit test are failing
> in the module called 'hadoop-mapreduce-client-jobclient'.
> See this link for details:
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3259/9/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
> In case of the above Jenkins link expires later, I attached the same text
> file to this jira.
> Let's see one example:
> org.apache.hadoop.mapred.TestMROpportunisticMaps#testAllOpportunisticMaps
> Logs are also attached.
> An example stacktrace, for reference:
> {code}
> 2022-06-29 11:24:13,510 INFO [Listener at 0.0.0.0/8049]
> service.AbstractService (AbstractService.java:noteFailure(268)) - Service
> TestMROpportunisticMaps failed in state STOPPED
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.stopRMClient(AggregatedLogDeletionService.java:322)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.serviceStop(AggregatedLogDeletionService.java:229)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:160)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:134)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStop(JobHistoryServer.java:203)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster$JobHistoryServerWrapper.serviceStop(MiniMRYarnCluster.java:293)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:160)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:134)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapred.MiniMRYarnClusterAdapter.stop(MiniMRYarnClusterAdapter.java:56)
> at
> org.apache.hadoop.mapred.TestMROpportunisticMaps.doTest(TestMROpportunisticMaps.java:108)
> at
> org.apache.hadoop.mapred.TestMROpportunisticMaps.doTest(TestMROpportunisticMaps.java:74)
> at
> org.apache.hadoop.mapred.TestMROpportunisticMaps.testAllOpportunisticMaps(TestMROpportunisticMaps.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
> at
> com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
> {code}
> The cause of this NPE is YARN-11188 (commit link:
> https://github.com/apache/hadoop/commit/e6ecc4f3e4433ae23fd745f6e0c641a019664253).
> The issue is that in
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#stopRMClient,
> there's an iteration over
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#tasks,
> that is only initialized in
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#scheduleLogDeletionTasks.
> This method won't run if log aggregation is not enabled, but the
> AggregatedLogDeletionService is active.
> An easy fix for this is to initialize
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService#tasks to
> an empty list on the field level.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
