[
https://issues.apache.org/jira/browse/YARN-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750288#comment-13750288
]
Chris Riccomini commented on YARN-896:
--------------------------------------
bq. Because many of these systems roll their logs to avoid filling up disks we
will probably need a protocol of some sort for the container to communicate
with the Node Manager when logs are ready to be processed.
Somewhat related, and a random thought: Samza is currently piping stdout and
stderr to files when it sets its AM and container commands (... 1>logs/stdout
2>logs/stderr). These files never get rolled, and they can get very big for
long-lived processes (10s-100s of gigs). We encourage using a logging system
(log4j) to handle this stuff, but there are cases where the recommended use is
not followed (sigh :)).
What do you guys think about tweaking the NM to consume the STDOUT/STDERR
streams from the ProcessBuilder when it executes AM/container commands, writing
them to stdout/stderr files, and rolling them periodically (e.g. daily)?
> Roll up for long lived YARN
> ---------------------------
>
> Key: YARN-896
> URL: https://issues.apache.org/jira/browse/YARN-896
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Robert Joseph Evans
>
> YARN is intended to be general purpose, but it is missing some features to be
> able to truly support long lived applications and long lived containers.
> This ticket is intended to
> # discuss what is needed to support long lived processes
> # track the resulting JIRA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
