Chukwa log4j appender logs corrupted data if the system is under high stress
----------------------------------------------------------------------------
Key: HADOOP-5046
URL: https://issues.apache.org/jira/browse/HADOOP-5046
Project: Hadoop Core
Issue Type: Bug
Components: contrib/chukwa
Environment: Redhat EL 5.1, Java 6
Reporter: Eric Yang
Data from Iostat indicates that log files did not write properly when system is
under high stress.
2008-12-29 03:03:48,510 INFO
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec: Linux 2.6.9-55.ELsmp
(example1002) 12/29/08^D
^D
avg-cpu: %user %nice %sys %iowait %idle^D
1.19 0.35 0.85 2.63 94.99^D
^D
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util^D
sda 0.13 33.31 3.02 3.53 281.94 311.02 140.97 155.51 90.52
0.56 86.19 2.30 1.51^D
sdb 4.52 3.45 5.93 1.51 107.12 39.67 53.56 19.83 19.74
0.07 9.98 3.53 2.63^D
sdc 4.57 22.76 7.12 1.71 395.58 195.76 197.79 97.88 66.93
0.24 27.13 3.90 3.44^D
sdd 4.52 18.17 6.13 1.65 151.17 158.59 75.58 79.30 39.81
0.45 57.96 3.84 2.98^D
^D
avg-cpu: %user %nice %sys %iowait %idle^D
24.83 0.00 0.29 0.07 74.81^D
^D
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util^D
sda 1.04 1.78 41.99 3.64 8706.75 45.10 4353.37 22.55 191.82
0.21 4.50 3.70 16.89^D
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2008-12-29
03:08:48,513 INFO org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec:
Linux 2.6.9-55.ELsmp (example1002) 12/29/08^D
^D
The most probable reason is that disk buffer got paged out before it is written
to disk. Exec plugin can be configured to always flush on every output. For
hadoop logs, this need to be fine tuned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
