GitHub user zhuoliu opened a pull request:
https://github.com/apache/storm/pull/822
[YSTORM-901] Worker Artifacts Directory for Hierarchical Log Management and
Debugging
Originally all logs (both for workers and daemons) are under
storm-home/logs, this causes name confusion
("wc22-1-1445957152-worker-6700.log") and difficulty for debugging (hard to
store and manage heapdump and gc log files).
For better management, we present the Worker Artifacts Directory for
Hierarchical Log Management and Debugging:
1. workers' logs and metadata are moved to the new
directory "storm-local/workers-artifacts", while daemon logs are still
placed in storm-home/logs.
2. Cleanup, view and list implementations are modified accordingly in
logviewer and UI. Specifically, we add per-worker quota and total quota for
files' size, and the cleanup thread will periodically cleanup the oldest files
in each worker directory and the whole artifacts if the quota is exceeded. Note
that, the active workers' active logs will be skipped during cleaning.
3. In addition, a symlink called artifacts is created
from each worker's directory to its log directory
"workers-artifacts/topoId/port/".
gc.log and coredump files can be configured by users to place under this
symlink, thus able to be downloaded from UI conveniently.
4. Unit tests are added and the event log are supported under each worker's
artifacts directory.
Current log structures:
--topoId0
--topoId1
------port1
------port2
---------worker.log
---------worker.log.err
---------worker.log.out
---------worker.yaml
---------gc.log
----------heapdump
----------events.log
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/storm 901
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/822.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #822
----
commit 4ab6a0f2d2af36102157b632ee55929534165b36
Author: zhuol <[email protected]>
Date: 2015-10-26T16:37:43Z
STORM-901 worker-artifacts for logviewer
commit 28899baf3a24ab435cbb3db13decafb652ba3afe
Author: zhuol <[email protected]>
Date: 2015-10-26T20:50:10Z
Solve event-log issue, ui, dir, routes, listLogs
commit 8d9bb1fb8cddef900d53d42b06fd8952cb5b4cb7
Author: zhuol <[email protected]>
Date: 2015-10-27T02:59:27Z
Fix supervisor test
commit c7bbd091b06b68bd8d22a6f66a4210bcf4a475fa
Author: zhuol <[email protected]>
Date: 2015-10-27T14:34:03Z
Add unit tests for logviewer cleanup, filter, sorting, etc.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
