eubnara opened a new pull request, #8691:
URL: https://github.com/apache/hadoop/pull/8691
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
When a container is killed while it is localizing, the `LocalizerRunner`
thread is interrupted. In the `finally` block of `LocalizerRunner.run()`,
dispatching the `ContainerResourceFailedEvent` then throws
`YarnRuntimeException(InterruptedException)`, which skips everything after
it: resources left in DOWNLOADING state are never unlocked, and the
deletion tasks for the localization dirs, the `_tmp` download dirs and the
nmPrivate token file are never scheduled.
This PR revives the fix from #7893 by @zheng-weihao (stale-closed after
100 days of inactivity), unchanged: the event dispatch is wrapped in
try/catch so the cleanup below it always runs. The remaining review
comment on #7893 (dropping `FSError` from the catch list) had already been
addressed in its final revision. Credit to the original author is kept in
the commit message.
What this PR adds on top of #7893 is a unit test, which was the other
blocker ("no new or modified tests").
We hit this in production on a ~600 NodeManager cluster running
DefaultContainerExecutor: because DCE runs the localizer inside the NM
JVM, every kill-during-localization also leaked DFS block reader sockets
into the long-lived NM process, leaving DataNodes with FIN_WAIT1
connections whose send queues never drain. The HDFS side of that leak is
tracked separately in HDFS-17965 (#8690); this issue is the YARN-side
trigger and also leaves resources stuck in DOWNLOADING state regardless of
the container executor in use.
### How was this patch tested?
New unit test
`TestResourceLocalizationService#testDownloadingResourcesCleanedUpWhenDispatchFails`:
a mocked dispatcher throws `YarnRuntimeException(InterruptedException)`
when the `ContainerResourceFailedEvent` is dispatched, simulating the
kill-during-localization interrupt. Without the fix, `run()` propagates
the exception and the DOWNLOADING resource is never unlocked (test fails).
With the fix, the resource is unlocked and the deletion tasks are
scheduled (test passes).
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
Contains content generated by Claude Code (Anthropic Claude): the
new unit test. The fix itself is unchanged from #7893. All content
was human-reviewed and complies with the ASF Generative Tooling
Guidance (https://www.apache.org/legal/generative-tooling.html).
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
