[
https://issues.apache.org/jira/browse/TRAFODION-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717284#comment-14717284
]
Suresh Subbiah commented on TRAFODION-1051:
-------------------------------------------
Steve Varnau (steve-varnau) wrote on 2015-03-11: #1
Or re-direct hadoop and hbase logs to /mnt/log?
Steve Varnau (steve-varnau) on 2015-03-11
Changed in trafodion:
importance: Undecided → Critical
assignee: nobody → Steve Varnau (steve-varnau)
Trafodion-Gerrit (neo-devtools) on 2015-03-16
Changed in trafodion:
status: New → In Progress
Trafodion-Gerrit (neo-devtools) wrote on 2015-03-17: Fix merged to infra
(master) #2
Reviewed: https://review.trafodion.org/1312
Committed:
https://github.com/trafodion/infra/commit/08c752d340ffbf6d029d731b84012b13903ab7e6
Submitter: Trafodion Jenkins
Branch: master
commit 08c752d340ffbf6d029d731b84012b13903ab7e6
Author: Steve Varnau <email address hidden>
Date: Fri Mar 13 08:59:55 2015 -0700
Check-node script to record disk space
This will be used to track disk usage in case we have repeat
of failure mode where all root filesystem space is consumed.
Disk space usage will be output to the stdout (job console)
and also to logs in jenkins home directory.
So far this script is not used. Need to let it propagate
out to all test slave nodes before updating jenkins to
use it in various jobs.
Partial-Bug: #1430876
Change-Id: Ie4bf356d13930456a94f36ad4a4983d0e446ea3a
Trafodion-Gerrit (neo-devtools) wrote on 2015-03-18: #3
Reviewed: https://review.trafodion.org/1319
Committed:
https://github.com/trafodion/infra/commit/b455a5ca366bde976ced371cf8074e387a88adff
Submitter: Trafodion Jenkins
Branch: master
commit b455a5ca366bde976ced371cf8074e387a88adff
Author: Steve Varnau <email address hidden>
Date: Tue Mar 17 11:12:51 2015 -0700
Include check-node step in test jobs
Add check-node at beginning and end of each test job, but not
build or other quick jobs.
Implement end of job check as a post-build publisher. This will
run the script even at the end of jobs that time-out.
Also add path of HDFS data storage for HW configs.
Partial-Bug: 1430876
Change-Id: I12d64b61ead6b3a7bb97642b4ed14fdba788d05e
Steve Varnau (steve-varnau) wrote on 2015-03-18: #4
This issue has not re-occurred, indicating it is either gone or happens in rare
circumstances. Possibly it only occurs in test environment because of the odd
ways that jobs can get interrupted and re-started.
To mitigate this issue, diskspace logging has been introduced in test jobs so
that the status is logged at beginning and end of each job, both in the job
console, and in the jenkins home directory.
This will enable better analysis of a recurrence; having a log will pinpoint
when things went bad, hopefully identifying the smoking gun.
Closing this bug.
Changed in trafodion:
status: In Progress → Fix Released
> LP Bug: 1430876 - infra - hadoop/hbase fill up root filesystem
> --------------------------------------------------------------
>
> Key: TRAFODION-1051
> URL: https://issues.apache.org/jira/browse/TRAFODION-1051
> Project: Apache Trafodion
> Issue Type: Bug
> Components: Build Infrastructure
> Reporter: Steve Varnau
> Assignee: Steve Varnau
> Priority: Blocker
>
> In some circumstances, hadoop and hbase spew gigabytes of messages into log
> files, filling up root filesystem. Because jenkins can’t write a temp file,
> the next job fails completely. Jenkins eventually detects machine out of
> disk space and disables it, but it may cause several jobs to fail.
> The problem seems to happen pretty rapidly. Since jenkins uses /tmp, which
> shares filesystem with /var/log, a job cannot run to clean things up.
> Running "hbase-clean.sh initialize" gets hbase/hadoop out of bad mode (once
> some disk space is reclaimed).
> Ideas:
> - can we get jenkins to use a different temp directory (/mnt/tmp)?
> Then we could check for and clean up at beginning of a job.
> - Run some sort of cron job to clean up /var/log/h*
> - Run postbuild step to stop hbase and clean up /var/log?
> That should take care of jobs that finish normally or timeout. But if disk
> is already full, that might prevent step from running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
