[
https://issues.apache.org/jira/browse/STORM-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976539#comment-14976539
]
ASF GitHub Bot commented on STORM-901:
--------------------------------------
GitHub user zhuoliu reopened a pull request:
https://github.com/apache/storm/pull/822
[STORM-901] Worker Artifacts Directory for Hierarchical Log Management and
Debugging
Originally all logs (both for workers and daemons) are under
storm-home/logs, this causes name confusion
("wc22-1-1445957152-worker-6700.log") and difficulty for debugging (hard to
store and manage heapdump and gc log files for all topologies' workers in a
single directory).
For better management, we present the Worker Artifacts Directory for
Hierarchical Log Management and Debugging:
1. workers' logs and metadata are moved to the new directory
"storm-local/workers-artifacts", while daemon logs are still placed in
storm-home/logs.
2. Cleanup, view and list implementations are modified accordingly in
logviewer and UI. Specifically, we add per-worker quota and total quota for
files' size, and the cleanup thread will periodically cleanup the oldest files
in each worker directory and the whole artifacts if the quota is exceeded. Note
that, the active workers' active logs will be skipped during cleaning.
3. In addition, a symlink called artifacts is created from each worker's
directory to its log directory "workers-artifacts/topoId/port/". gc.log and
heapdump files can be configured by users to place under this symlink, thus
able to be downloaded from UI conveniently. Other useful features can be easily
support by worker artifacts directory, where their resource files can be dumped.
4. Unit tests are added and the event log are supported under each worker's
artifacts directory.
Current log structures under storm-local/workers-artifacts:
--topoId0
--topoId1
------port1
------port2
---------worker.log
---------worker.log.err
---------worker.log.out
---------worker.yaml
---------gc.log
----------heapdump
----------events.log
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/storm 901
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/822.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #822
----
commit 4ab6a0f2d2af36102157b632ee55929534165b36
Author: zhuol <[email protected]>
Date: 2015-10-26T16:37:43Z
STORM-901 worker-artifacts for logviewer
commit 28899baf3a24ab435cbb3db13decafb652ba3afe
Author: zhuol <[email protected]>
Date: 2015-10-26T20:50:10Z
Solve event-log issue, ui, dir, routes, listLogs
commit 8d9bb1fb8cddef900d53d42b06fd8952cb5b4cb7
Author: zhuol <[email protected]>
Date: 2015-10-27T02:59:27Z
Fix supervisor test
commit c7bbd091b06b68bd8d22a6f66a4210bcf4a475fa
Author: zhuol <[email protected]>
Date: 2015-10-27T14:34:03Z
Add unit tests for logviewer cleanup, filter, sorting, etc.
----
> Worker Artifacts Directory
> --------------------------
>
> Key: STORM-901
> URL: https://issues.apache.org/jira/browse/STORM-901
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Zhuo Liu
>
> At Yahoo we have added in a separate directory that stores a workers log
> files, but also provides a place where the worker can place other files that
> it might help with debugging. It is a new directory in the current working
> directory of the worker process. The files in there are cleaned up if they
> get too large, but provides a place for heap dumps GC logs, etc.
> This work is already done, but needs to be put back into open source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
