[
https://issues.apache.org/jira/browse/MAPREDUCE-7539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091136#comment-18091136
]
ASF GitHub Bot commented on MAPREDUCE-7539:
-------------------------------------------
ferdelyi commented on PR #8556:
URL: https://github.com/apache/hadoop/pull/8556#issuecomment-4788162391
Change in LogAggregationIndexedFileController.java explained by Claude Code:
1. Core bug fix — UUID validation in loadIndexedLogsMeta (line ~937)
Before:
if (this.uuid == null) {
this.uuid = createUUID(appId);
}
if (uuidReadLen != UUID_LENGTH || !Arrays.equals(this.uuid, uuidRead)) {
After:
byte[] expectedUuid = createUUID(appId);
if (uuidReadLen != UUID_LENGTH || !Arrays.equals(expectedUuid, uuidRead)) {
This is the root fix. The IFile format embeds a UUID derived from the
application ID to tie log files to a specific app. When reading logs, the
controller was checking the file's UUID against this.uuid — an instance field
set once and never reset.
After HADOOP-15984 made HsWebServices a singleton, the same controller
instance handles requests for multiple applications. The instance field
this.uuid would hold the UUID of the first application ever read. Every
subsequent app's log files would fail validation with "UUID mismatch"
and return no results — until JHS restarted.
The fix: derive the expected UUID directly from the appId argument on
every call, never relying on the cached instance field.
---
2. CachedIndexedLogsMeta class removed
The inner class CachedIndexedLogsMeta and its use in getApplicationOwner /
getApplicationAcls are deleted entirely. Both methods now call
loadIndexedLogsMeta() directly:
// Before:
if (this.cachedIndexedLogsMeta == null ||
!this.cachedIndexedLogsMeta.getRemoteLogPath().equals(aggregatedLogPath)) {
this.cachedIndexedLogsMeta = new
CachedIndexedLogsMeta(loadIndexedLogsMeta(...), aggregatedLogPath);
}
return this.cachedIndexedLogsMeta.getCachedIndexedLogsMeta().getUser();
// After:
return loadIndexedLogsMeta(aggregatedLogPath, appId).getUser();
The cache was stale for the same reason — it was an instance field that
persisted across application requests. Removing it also simplifies the code
with no meaningful performance impact (these paths are not hot).
---
3. Write-side state reset in initializeWriter (line ~162)
The cachedIndexedLogsMeta field is replaced by currentWriteAppId. When
initializeWriter is called, it now detects an app change and clears per-app
write state:
if (currentWriteAppId == null || !currentWriteAppId.equals(appId)) {
currentWriteAppId = appId;
indexedLogsMeta = null;
uuid = null;
}
This ensures the write path also starts clean when the controller is
reused for a different application, preventing stale uuid and indexedLogsMeta
from a previous write leaking into a new one.
---
4. Bug fix in loadUUIDFromLogFile (line ~1305)
A pre-existing logic inversion bug was fixed:
// Before (buggy — deletes matching files, keeps mismatches):
if (actual != uuid.length || Arrays.equals(b, uuid)) {
deleteFileWithRetries(fc, checkPath);
} else if (id == null) {
id = uuid; // also wrong: stores the input parameter, not what was
read
}
// After (correct):
if (actual != uuid.length || !Arrays.equals(b, uuid)) {
deleteFileWithRetries(fc, checkPath);
} else if (id == null) {
id = b; // stores the bytes actually read from the file
}
Two bugs fixed here: the missing ! meant valid UUID files were being
deleted while corrupt ones were kept, and id = uuid was storing the method's
input parameter rather than the bytes read from disk (b).
---
Summary: The commit fixes a singleton-controller bug where instance-level
UUID and metadata cache fields caused all applications except the first to fail
log reads. The fix makes UUID derivation stateless (computed from appId on
every call) and removes the per-instance cache entirely.
A logic inversion bug in the UUID file scan is also corrected as a side
fix.
> JobHistoryServer API fails to serve aggregated logs due to uuid mismatch
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-7539
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7539
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 3.5.0
> Reporter: Brian Goerlitz
> Assignee: Ferenc Erdelyi
> Priority: Major
> Labels: pull-request-available
>
> Due to Singleton annotations added in HADOOP-15984 for HSWebServices, the
> first time an ifile log is read via the {{/ws/v1/history/aggregatedlogs}}
> API, the UUID of the log is stored in the HSWebServices instance of the
> {{LogAggregationIndexedFileController}} and used for verification of all
> future log files. This results in failure to read any aggregated log files
> belonging to an app that is not the first one accessed after JHS restart.
> {noformat}
> 2026-06-08 20:16:40,368 WARN
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController:
> Can not get log meta from the log
> file:hdfs://nn:8020/tmp/logs/systest/bucket-logs-ifile/0002/application_1780935195539_0002/nm_8041
> The UUID from
> hdfs://nn:8020/tmp/logs/systest/bucket-logs-ifile/0002/application_1780935195539_0002/nm_8041
> is not correct. The offset of loaded UUID is 296605
> 2026-06-08 20:16:40,368 WARN
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler: SERVICE_UNAVAILABLE
> javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogMeta(LogServlet.java:134)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogsInfo(LogServlet.java:325)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getLogsInfo(LogServlet.java:263)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getAggregatedLogsMeta(HsWebServices.java:521)
> ...
> Caused by: org.apache.hadoop.yarn.webapp.NotFoundException: HTTP 404 Not Found
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogMeta(LogServlet.java:122)
> ... 86 more
> Caused by: java.lang.Exception: Can not get log meta for request.
> at
> org.apache.hadoop.yarn.webapp.NotFoundException.<init>(NotFoundException.java:45)
> ... 87 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
