[
https://issues.apache.org/jira/browse/YARN-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317251#comment-16317251
]
Jason Lowe commented on YARN-6929:
----------------------------------
Sorry for the delay. Looks like the patch needs to be rebased.
How is this going to work with existing log aggregation setups? In other
words, as soon as this is deployed, will users be able to access logs that were
aggregated under the old scheme? Or during a rolling upgrade where some
nodemanagers are aggregating to the old location and some are aggregating to
the new location, will a log reader be able to find all of the logs?
I'm not thrilled with seeing the hdfs and hdfs-client dependency added for what
is essentially a config key. This is especially true since YARN should not
require HDFS be the underlying filesystem, and therefore this key may not be
set. Even if HDFS is being used as the underlying filesystem,
dfs.namenode.fs-limits.max-directory-items is a property intended to be set and
read on the namenode server and not intended for client consumption. There's
no guarantee it has been set properly on the client side. And if it isn't set
properly on the client side, the log reader will not be able to find the logs
properly. As such I think it makes more sense to have a separate config
property for this. If users want to tie the two values together they can set
the YARN property value to "$\{dfs.namenode.fs-limits.max-directory-items\}"
These logs looks like they were added for debug purposes and either should be
removed or marked as a debug log:
{noformat}
LOG.info("UserDir="+userDir.getPath());
[...]
LOG.info("USERDIR="+userDirPath);
[...]
LOG.info("CLUSTERTIMETSMA="+clusterTimestampDir.getPath());
[...]
LOG.info("BUCKET="+bucketDir.getPath());
{noformat}
CLUSTERTIMETSMA is a typo if the log is preserved.
To make this more efficient and reduce the load on the remote filesystem, we
should avoid a double liststat call. We know how many entries are in the
original liststat, and we know that deleting any child in the directory is
going to update the modtime of the parent directory. If we delete a directory
then we know the modtime is updated, so it won't be worth immediately checking
again to see if it should be deleted. It will only be deleted if there were
zero entries in the directory at the start of the loop, so we don't need to
stat it again after the loop.
Speaking of deleting bucket/cluster directories, won't it be a problem if one
thread is trying to delete a bucket just as another thread tries to aggregate
an app there? In other words, just because the directory looks empty now, how
to we know it will remain empty? The writing code path goes out of its way to
check if directories are there and creates them if missing, but the deletion
code could be removing a directory just after the write code checks to make
sure it is there.
Nit: To get a string value the code should use String.valueOf or Long.toString,
Integer.toString, etc. rather than adding an empty string to something.
Nit: This change seems unnecessary:
{noformat}
- LOG.info("aggregated log deletion started.");
+ LOG.info("aggregated log deletion started here.");
{noformat}
Nit: It looks like there are a number of reformatting/whitespace-only changes
which are unrelated to this patch that makes it harder to read and backport,
e.g.:
{noformat}
logIOException("Error reading root log dir this deletion " +
- "attempt is being aborted", e);
+ "attempt is being aborted", e);
}
{noformat}
> yarn.nodemanager.remote-app-log-dir structure is not scalable
> -------------------------------------------------------------
>
> Key: YARN-6929
> URL: https://issues.apache.org/jira/browse/YARN-6929
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 2.7.3
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Attachments: YARN-6929.1.patch, YARN-6929.2.patch, YARN-6929.2.patch,
> YARN-6929.3.patch, YARN-6929.patch
>
>
> The current directory structure for yarn.nodemanager.remote-app-log-dir is
> not scalable. Maximum Subdirectory limit by default is 1048576 (HDFS-6102).
> With retention yarn.log-aggregation.retain-seconds of 7days, there are more
> chances LogAggregationService fails to create a new directory with
> FSLimitException$MaxDirectoryItemsExceededException.
> The current structure is
> <yarn.nodemanager.remote-app-log-dir>/<user>/logs/<job_name>. This can be
> improved with adding date as a subdirectory like
> <yarn.nodemanager.remote-app-log-dir>/<user>/logs/<date>/<job_name>
> {code}
> WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
> Application failed to init aggregation
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /app-logs/yarn/logs is exceeded: limit=1048576
> items=1048576
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2021)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2072)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:1841)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsRecursively(FSNamesystem.java:4351)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4262)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4221)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4194)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
>
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600)
>
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:308)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:366)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:320)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /app-logs/yarn/logs is exceeded: limit=1048576
> items=1048576
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2021)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2072)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:1841)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsRecursively(FSNamesystem.java:4351)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4262)
>
> {code}
> Thanks to Robert Mancuso for finding this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
