[
https://issues.apache.org/jira/browse/TRAFODION-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Subbiah closed TRAFODION-1051.
-------------------------------------
> LP Bug: 1430876 - infra - hadoop/hbase fill up root filesystem
> --------------------------------------------------------------
>
> Key: TRAFODION-1051
> URL: https://issues.apache.org/jira/browse/TRAFODION-1051
> Project: Apache Trafodion
> Issue Type: Bug
> Components: Build Infrastructure
> Reporter: Steve Varnau
> Assignee: Steve Varnau
> Priority: Blocker
>
> In some circumstances, hadoop and hbase spew gigabytes of messages into log
> files, filling up root filesystem. Because jenkins can’t write a temp file,
> the next job fails completely. Jenkins eventually detects machine out of
> disk space and disables it, but it may cause several jobs to fail.
> The problem seems to happen pretty rapidly. Since jenkins uses /tmp, which
> shares filesystem with /var/log, a job cannot run to clean things up.
> Running "hbase-clean.sh initialize" gets hbase/hadoop out of bad mode (once
> some disk space is reclaimed).
> Ideas:
> - can we get jenkins to use a different temp directory (/mnt/tmp)?
> Then we could check for and clean up at beginning of a job.
> - Run some sort of cron job to clean up /var/log/h*
> - Run postbuild step to stop hbase and clean up /var/log?
> That should take care of jobs that finish normally or timeout. But if disk
> is already full, that might prevent step from running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
