[
https://issues.apache.org/jira/browse/STORM-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008022#comment-15008022
]
Zhuo Liu commented on STORM-1206:
---------------------------------
In STORM-40 (https://github.com/apache/storm/pull/835/files), we set for apache
storm as "-Xloggc:artifacts/gc.log", so I would say for now, this issue is not
that emergent unless the worker dumps so other files into artifacts directory
crazily.
> Reduce logviewer memory usage
> -----------------------------
>
> Key: STORM-1206
> URL: https://issues.apache.org/jira/browse/STORM-1206
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Zhuo Liu
> Assignee: Zhuo Liu
>
> In production, we ran into an issue with logviewers bouncing with out of
> memory errors. Note that this happens very rarely, we met this in some
> extreme case when super frequently restarting of workers generates a huge
> number of gc files (~1M files).
> What was happening is that if there are lots of log files (~1 M files) for a
> particular headless user, we would have so many strings resident in memory
> that logviewer would run out of heap space.
> We were able to work around this by increasing the heap space, but we should
> consider putting some sort of an upper bound on the number of files so that
> we don't run in to this issue, even with the bigger heap.
> Using the java DirectoryStream can avoid holding all file names in memory
> during file listing. Also, a multi-round directory cleaner can be introduced
> to delete files while disk quota is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
