Zhuo Liu, thanks for the explanation, I see. On Sun, Nov 15, 2015 at 1:18 AM, Zhuo Liu (JIRA) <[email protected]> wrote:
> > [ > https://issues.apache.org/jira/browse/STORM-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005491#comment-15005491 > ] > > Zhuo Liu commented on STORM-1206: > --------------------------------- > > Hello Cody, we already have a hierarchical log file structure as > topology/worker/all files, please check storm-901. So in default case, each > worker directory will only have very limited number of log files and gc > files(10). We met the above problem in a very extreme case where we set the > gc log file name with pid, causing the automatic file number limit not > working and crazy restarting creates a huge number of gc files for a single > worker. > > > Reduce logviewer memory usage > > ----------------------------- > > > > Key: STORM-1206 > > URL: https://issues.apache.org/jira/browse/STORM-1206 > > Project: Apache Storm > > Issue Type: Improvement > > Components: storm-core > > Reporter: Zhuo Liu > > Assignee: Zhuo Liu > > > > In production, we ran into an issue with logviewers bouncing with out of > memory errors. Note that this happens very rarely, we met this in some > extreme case when super frequently restarting of workers generates a huge > number of gc files (~1M files). > > What was happening is that if there are lots of log files (~1 M files) > for a particular headless user, we would have so many strings resident in > memory that logviewer would run out of heap space. > > We were able to work around this by increasing the heap space, but we > should consider putting some sort of an upper bound on the number of files > so that we don't run in to this issue, even with the bigger heap. > > Using the java DirectoryStream can avoid holding all file names in > memory during file listing. Also, a multi-round directory cleaner can be > introduced to delete files while disk quota is exceeded. > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
