[
https://issues.apache.org/jira/browse/MAPREDUCE-7539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098830#comment-18098830
]
ASF GitHub Bot commented on MAPREDUCE-7539:
-------------------------------------------
K0K0V0K commented on code in PR #8556:
URL: https://github.com/apache/hadoop/pull/8556#discussion_r3644456248
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/LogAggregationIndexedFileController.java:
##########
@@ -121,22 +121,41 @@ public class LogAggregationIndexedFileController
private int fsNumRetries = 3;
private long fsRetryInterval = 1000L;
private static final int VERSION = 1;
- private IndexedLogsMeta indexedLogsMeta = null;
- private IndexedPerAggregationLogMeta logsMetaInThisCycle;
- private long logAggregationTimeInThisCycle;
- private FSDataOutputStream fsDataOStream;
private Algorithm compressAlgo;
- private CachedIndexedLogsMeta cachedIndexedLogsMeta = null;
- private boolean logAggregationSuccessfullyInThisCyCle = false;
- private long currentOffSet = 0;
- private Path remoteLogCheckSumFile;
- private FileContext fc;
- private UserGroupInformation ugi;
- private byte[] uuid = null;
private final int UUID_LENGTH = 32;
private long logRollOverMaxFileSize;
private Clock sysClock;
+ /**
+ * All mutable state that belongs to a single write session
+ * (one {@link #initializeWriter} / {@link #write} / {@link #postWrite} /
+ * {@link #closeWriter} lifecycle). Bundling it here means the read path
+ * cannot accidentally touch write-path state, regardless of whether the
+ * controller instance is shared across applications.
+ */
+ private static final class WriteSession {
+ /** UUID derived from the application being written. */
+ private final byte[] uuid;
+ /** Accumulated log metadata for the current aggregated file. */
+ private IndexedLogsMeta indexedLogsMeta;
+ /** Log metadata accumulated within this single write cycle. */
+ private IndexedPerAggregationLogMeta logsMetaInThisCycle;
+ private long logAggregationTimeInThisCycle;
+ private boolean logAggregationSuccessfullyInThisCyCle = false;
+ private long currentOffSet = 0;
+ private Path remoteLogCheckSumFile;
+ private FileContext fc;
+ private UserGroupInformation ugi;
+ private FSDataOutputStream fsDataOStream;
+
+ WriteSession(byte[] uuid) {
+ this.uuid = uuid;
+ }
+ }
+
+ /** Non-null while a write session is in progress; null otherwise. */
+ private WriteSession writeSession = null;
Review Comment:
Thanks @ferdelyi for the quick fix!
Now i am wondering this state does not have some concurrency problem.
The `#initializeWriter` says:
```
// Allocate a fresh WriteSession for every initializeWriter call so that
// all write-path state is completely isolated from other applications.
// This is safe whether the controller is used by a single NM aggregation
// thread or reused across requests by a singleton web service.
```
But after that the newly created object will be set like
```
this.writeSession = session;
```
So if we call
```
controller.initializeWriter(app1Context);
controller.initializeWriter(app2Context);
```
than writer of app1 will be overwritten with app2 context, and when
controller try to write app1 in fact will write to app2 location?
> JobHistoryServer API fails to serve aggregated logs due to uuid mismatch
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-7539
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7539
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 3.5.0
> Reporter: Brian Goerlitz
> Assignee: Ferenc Erdelyi
> Priority: Major
> Labels: pull-request-available
>
> Due to Singleton annotations added in HADOOP-15984 for HSWebServices, the
> first time an ifile log is read via the {{/ws/v1/history/aggregatedlogs}}
> API, the UUID of the log is stored in the HSWebServices instance of the
> {{LogAggregationIndexedFileController}} and used for verification of all
> future log files. This results in failure to read any aggregated log files
> belonging to an app that is not the first one accessed after JHS restart.
> {noformat}
> 2026-06-08 20:16:40,368 WARN
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController:
> Can not get log meta from the log
> file:hdfs://nn:8020/tmp/logs/systest/bucket-logs-ifile/0002/application_1780935195539_0002/nm_8041
> The UUID from
> hdfs://nn:8020/tmp/logs/systest/bucket-logs-ifile/0002/application_1780935195539_0002/nm_8041
> is not correct. The offset of loaded UUID is 296605
> 2026-06-08 20:16:40,368 WARN
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler: SERVICE_UNAVAILABLE
> javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogMeta(LogServlet.java:134)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogsInfo(LogServlet.java:325)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getLogsInfo(LogServlet.java:263)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getAggregatedLogsMeta(HsWebServices.java:521)
> ...
> Caused by: org.apache.hadoop.yarn.webapp.NotFoundException: HTTP 404 Not Found
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogMeta(LogServlet.java:122)
> ... 86 more
> Caused by: java.lang.Exception: Can not get log meta for request.
> at
> org.apache.hadoop.yarn.webapp.NotFoundException.<init>(NotFoundException.java:45)
> ... 87 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
