First we usually wait if it repeats again in situation like this :)
On Thu, Mar 1, 2012 at 3:01 PM, tom pierce <[email protected]> wrote: > What's the best way to even begin tracking this down? To my eye, this looks > like something has gone wrong on the server (full disk, maybe?). > > -tom > > > On 03/01/2012 03:06 PM, Apache Jenkins Server wrote: >> >> >> See<https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters/58/changes> >> >> Changes: >> >> [tcp] cleanups to make Jenkins a little happier >> >> [tcp] MAHOUT-980: Fix DistributedCache usage to allow EMR deployment >> >> ------------------------------------------ >> [...truncated 8490 lines...] >> 12/03/01 20:00:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:16 INFO mapred.JobClient: map 39% reduce 0% >> 12/03/01 20:00:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:22 INFO mapred.JobClient: map 40% reduce 0% >> 12/03/01 20:00:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:28 INFO mapred.JobClient: map 41% reduce 0% >> 12/03/01 20:00:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:34 INFO mapred.JobClient: map 42% reduce 0% >> 12/03/01 20:00:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:40 INFO mapred.JobClient: map 43% reduce 0% >> 12/03/01 20:00:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:46 INFO mapred.JobClient: map 44% reduce 0% >> 12/03/01 20:00:49 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:52 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:55 INFO mapred.LocalJobRunner: >> 12/03/01 20:00:55 INFO mapred.JobClient: map 45% reduce 0% >> 12/03/01 20:00:58 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:01 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:01 INFO mapred.JobClient: map 46% reduce 0% >> 12/03/01 20:01:04 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:07 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:07 INFO mapred.JobClient: map 47% reduce 0% >> 12/03/01 20:01:10 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:12 INFO mapred.MapTask: Spilling map output: buffer full= >> true >> 12/03/01 20:01:12 INFO mapred.MapTask: bufstart = 0; bufend = 79691690; >> bufvoid = 99614720 >> 12/03/01 20:01:12 INFO mapred.MapTask: kvstart = 0; kvend = 203102; length >> = 327680 >> 12/03/01 20:01:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:13 INFO mapred.JobClient: map 48% reduce 0% >> 12/03/01 20:01:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:18 INFO mapred.MapTask: Finished spill 0 >> 12/03/01 20:01:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:19 INFO mapred.JobClient: map 49% reduce 0% >> 12/03/01 20:01:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:28 INFO mapred.JobClient: map 50% reduce 0% >> 12/03/01 20:01:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:31 INFO mapred.JobClient: map 51% reduce 0% >> 12/03/01 20:01:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:40 INFO mapred.JobClient: map 52% reduce 0% >> 12/03/01 20:01:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:46 INFO mapred.JobClient: map 53% reduce 0% >> 12/03/01 20:01:49 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:52 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:52 INFO mapred.JobClient: map 54% reduce 0% >> 12/03/01 20:01:55 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:58 INFO mapred.LocalJobRunner: >> 12/03/01 20:01:58 INFO mapred.JobClient: map 55% reduce 0% >> 12/03/01 20:02:01 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:04 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:07 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:07 INFO mapred.JobClient: map 56% reduce 0% >> 12/03/01 20:02:10 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:13 INFO mapred.JobClient: map 57% reduce 0% >> 12/03/01 20:02:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:19 INFO mapred.JobClient: map 58% reduce 0% >> 12/03/01 20:02:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:25 INFO mapred.JobClient: map 59% reduce 0% >> 12/03/01 20:02:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:34 INFO mapred.JobClient: map 60% reduce 0% >> 12/03/01 20:02:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:40 INFO mapred.JobClient: map 61% reduce 0% >> 12/03/01 20:02:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:46 INFO mapred.JobClient: map 62% reduce 0% >> 12/03/01 20:02:49 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:52 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:52 INFO mapred.JobClient: map 63% reduce 0% >> 12/03/01 20:02:55 INFO mapred.LocalJobRunner: >> 12/03/01 20:02:58 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:01 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:01 INFO mapred.JobClient: map 64% reduce 0% >> 12/03/01 20:03:04 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:07 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:10 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:10 INFO mapred.JobClient: map 65% reduce 0% >> 12/03/01 20:03:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:19 INFO mapred.JobClient: map 66% reduce 0% >> 12/03/01 20:03:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:28 INFO mapred.JobClient: map 67% reduce 0% >> 12/03/01 20:03:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:37 INFO mapred.JobClient: map 68% reduce 0% >> 12/03/01 20:03:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:43 INFO mapred.JobClient: map 69% reduce 0% >> 12/03/01 20:03:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:49 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:49 INFO mapred.JobClient: map 70% reduce 0% >> 12/03/01 20:03:52 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:55 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:58 INFO mapred.LocalJobRunner: >> 12/03/01 20:03:58 INFO mapred.JobClient: map 71% reduce 0% >> 12/03/01 20:04:01 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:04 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:04 INFO mapred.JobClient: map 72% reduce 0% >> 12/03/01 20:04:07 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:10 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:10 INFO mapred.JobClient: map 73% reduce 0% >> 12/03/01 20:04:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:16 INFO mapred.JobClient: map 74% reduce 0% >> 12/03/01 20:04:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:22 INFO mapred.JobClient: map 75% reduce 0% >> 12/03/01 20:04:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:31 INFO mapred.JobClient: map 76% reduce 0% >> 12/03/01 20:04:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:37 INFO mapred.JobClient: map 77% reduce 0% >> 12/03/01 20:04:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:43 INFO mapred.JobClient: map 78% reduce 0% >> 12/03/01 20:04:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:49 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:49 INFO mapred.JobClient: map 79% reduce 0% >> 12/03/01 20:04:52 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:55 INFO mapred.LocalJobRunner: >> 12/03/01 20:04:55 INFO mapred.JobClient: map 80% reduce 0% >> 12/03/01 20:04:58 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:01 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:01 INFO mapred.JobClient: map 81% reduce 0% >> 12/03/01 20:05:04 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:07 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:07 INFO mapred.JobClient: map 82% reduce 0% >> 12/03/01 20:05:10 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:13 INFO mapred.JobClient: map 83% reduce 0% >> 12/03/01 20:05:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:19 INFO mapred.JobClient: map 84% reduce 0% >> 12/03/01 20:05:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:25 INFO mapred.JobClient: map 85% reduce 0% >> 12/03/01 20:05:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:31 INFO mapred.JobClient: map 86% reduce 0% >> 12/03/01 20:05:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:37 INFO mapred.JobClient: map 87% reduce 0% >> 12/03/01 20:05:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:46 INFO mapred.JobClient: map 88% reduce 0% >> 12/03/01 20:05:49 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:52 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:52 INFO mapred.JobClient: map 89% reduce 0% >> 12/03/01 20:05:55 INFO mapred.LocalJobRunner: >> 12/03/01 20:05:58 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:01 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:01 INFO mapred.JobClient: map 90% reduce 0% >> 12/03/01 20:06:04 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:07 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:07 INFO mapred.JobClient: map 91% reduce 0% >> 12/03/01 20:06:10 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:13 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:13 INFO mapred.JobClient: map 92% reduce 0% >> 12/03/01 20:06:16 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:19 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:19 INFO mapred.JobClient: map 93% reduce 0% >> 12/03/01 20:06:22 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:25 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:28 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:28 INFO mapred.JobClient: map 94% reduce 0% >> 12/03/01 20:06:31 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:34 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:37 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:37 INFO mapred.JobClient: map 95% reduce 0% >> 12/03/01 20:06:40 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:43 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:46 INFO mapred.MapTask: Spilling map output: buffer full= >> true >> 12/03/01 20:06:46 INFO mapred.MapTask: bufstart = 79691690; bufend = >> 59768438; bufvoid = 99614720 >> 12/03/01 20:06:46 INFO mapred.MapTask: kvstart = 203102; kvend = 85998; >> length = 327680 >> 12/03/01 20:06:46 WARN mapred.LocalJobRunner: job_local_0005 >> java.io.IOException: Spill failed >> at >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029) >> at >> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691) >> at >> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansClusterer.emitPointProbToCluster(FuzzyKMeansClusterer.java:165) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansMapper.map(FuzzyKMeansMapper.java:45) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansMapper.map(FuzzyKMeansMapper.java:35) >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) >> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could >> not find any valid local directory for output/spill1.out >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127) >> at >> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:121) >> at >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1392) >> at >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853) >> at >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344) >> 12/03/01 20:06:46 INFO mapred.LocalJobRunner: >> 12/03/01 20:06:46 INFO mapred.JobClient: map 96% reduce 0% >> 12/03/01 20:06:46 INFO mapred.JobClient: Job complete: job_local_0005 >> 12/03/01 20:06:46 INFO mapred.JobClient: Counters: 11 >> 12/03/01 20:06:46 INFO mapred.JobClient: File Input Format Counters >> 12/03/01 20:06:46 INFO mapred.JobClient: Bytes Read=4734976 >> 12/03/01 20:06:46 INFO mapred.JobClient: FileSystemCounters >> 12/03/01 20:06:46 INFO mapred.JobClient: FILE_BYTES_READ=398738970 >> 12/03/01 20:06:46 INFO mapred.JobClient: FILE_BYTES_WRITTEN=357205174 >> 12/03/01 20:06:46 INFO mapred.JobClient: Map-Reduce Framework >> 12/03/01 20:06:46 INFO mapred.JobClient: Map output materialized >> bytes=0 >> 12/03/01 20:06:46 INFO mapred.JobClient: Combine output records=20 >> 12/03/01 20:06:46 INFO mapred.JobClient: Map input records=20685 >> 12/03/01 20:06:46 INFO mapred.JobClient: Spilled Records=20 >> 12/03/01 20:06:46 INFO mapred.JobClient: Map output bytes=159384560 >> 12/03/01 20:06:46 INFO mapred.JobClient: SPLIT_RAW_BYTES=155 >> 12/03/01 20:06:46 INFO mapred.JobClient: Map output records=413680 >> 12/03/01 20:06:46 INFO mapred.JobClient: Combine input records=203102 >> Exception in thread "main" java.lang.InterruptedException: Fuzzy K-Means >> Iteration failed processing >> /tmp/mahout-work-jenkins/reuters-fkmeans/clusters-4 >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.runIteration(FuzzyKMeansDriver.java:252) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.buildClustersMR(FuzzyKMeansDriver.java:426) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.buildClusters(FuzzyKMeansDriver.java:345) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.run(FuzzyKMeansDriver.java:295) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.run(FuzzyKMeansDriver.java:125) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at >> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.main(FuzzyKMeansDriver.java:67) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) >> at >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) >> at >> org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188) >> Build step 'Execute shell' marked build as failure >> >
