See
<https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters-II/507/changes>
Changes:
[gsingers] add some helpers to AbstractJob, add a main to DictionaryVectorizer
to try and isolate some issues in testing DicVec on Hadoop for MAHOUT-1247
[gsingers] MAHOUT-1211: clean up use of deprecated closeQuietly api
[gsingers] add changelog for M-1103 and M-1126
[gsingers] MAHOUT-1103: properly partition the data for MapReduce
[gsingers] MAHOUT-1126: add in unpack options to control exclusion of LICENSE
file
[ssc] MAHOUT-1164 Make ARFF integration generate meta-data in JSON format
[ssc] MAHOUT-1163: Make random forest classifier meta-data file human readable
------------------------------------------
[...truncated 5725 lines...]
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0018_m_000000_0 is done. And is in the process of
commiting
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0018_m_000000_0' done.
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 1 sorted segments
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 1 segments left of total size: 162 bytes
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0018_r_000000_0 is done. And is in the process of
commiting
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0018_r_000000_0 is allowed to commit now
Jun 9, 2013 6:28:38 PM
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
INFO: Saved output of task 'attempt_local_0018_r_000000_0' to
/tmp/mahout-work-hudson/reuters-lda-model/model-18
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO: reduce > reduce
Jun 9, 2013 6:28:38 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0018_r_000000_0' done.
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 100% reduce 100%
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0018
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Counters: 17
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: File Output Format Counters
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Written=389
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=1528463202
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=1542274427
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: File Input Format Counters
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Read=152
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Map output materialized bytes=166
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Map input records=0
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce shuffle bytes=0
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=40
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Map output bytes=120
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Total committed heap usage (bytes)=3856662528
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: SPLIT_RAW_BYTES=119
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Combine input records=20
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input records=20
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input groups=20
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Combine output records=20
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce output records=20
Jun 9, 2013 6:28:39 PM org.apache.hadoop.mapred.Counters log
INFO: Map output records=20
Jun 9, 2013 6:28:39 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run iteration 19 of 20
Jun 9, 2013 6:28:39 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run: Iteration 19 of 20, input path:
/tmp/mahout-work-hudson/reuters-lda-model/model-18
Jun 9, 2013 6:28:40 PM org.apache.hadoop.mapreduce.lib.input.FileInputFormat
listStatus
INFO: Total input paths to process : 1
Jun 9, 2013 6:28:42 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0019
Jun 9, 2013 6:28:42 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 9, 2013 6:28:42 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Jun 9, 2013 6:28:43 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 0% reduce 0%
Jun 9, 2013 6:28:44 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: data buffer = 79691776/99614720
Jun 9, 2013 6:28:44 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: record buffer = 262144/327680
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Retrieving configuration
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initializing read model
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initializing write model
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initializing model trainer
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Starting training threadpool with 4 threads
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Stopping model trainer
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initiating stopping of training threadpool
Jun 9, 2013 6:28:44 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: threadpool took: 0.654236ms
Jun 9, 2013 6:28:45 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: readModel.stop() took 1008.969526ms
Jun 9, 2013 6:28:46 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: writeModel.stop() took 1009.547291ms
Jun 9, 2013 6:28:46 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Writing model
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
INFO: Starting flush of map output
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0019_m_000000_0 is done. And is in the process of
commiting
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0019_m_000000_0' done.
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 1 sorted segments
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 1 segments left of total size: 162 bytes
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0019_r_000000_0 is done. And is in the process of
commiting
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0019_r_000000_0 is allowed to commit now
Jun 9, 2013 6:28:46 PM
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
INFO: Saved output of task 'attempt_local_0019_r_000000_0' to
/tmp/mahout-work-hudson/reuters-lda-model/model-19
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO: reduce > reduce
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0019_r_000000_0' done.
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 100% reduce 100%
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0019
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Counters: 17
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: File Output Format Counters
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Written=389
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=1613377866
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=1627956401
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: File Input Format Counters
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Read=152
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Map output materialized bytes=166
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Map input records=0
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce shuffle bytes=0
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=40
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Map output bytes=120
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Total committed heap usage (bytes)=4057989120
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: SPLIT_RAW_BYTES=119
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Combine input records=20
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input records=20
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input groups=20
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Combine output records=20
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce output records=20
Jun 9, 2013 6:28:46 PM org.apache.hadoop.mapred.Counters log
INFO: Map output records=20
Jun 9, 2013 6:28:46 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run iteration 20 of 20
Jun 9, 2013 6:28:46 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run: Iteration 20 of 20, input path:
/tmp/mahout-work-hudson/reuters-lda-model/model-19
Jun 9, 2013 6:28:49 PM org.apache.hadoop.mapred.JobClient$2 run
INFO: Cleaning up the staging area
file:/tmp/hadoop-hudson/mapred/staging/hudson1761131945/.staging/job_local_0020
Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException:
No space left on device
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:200)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:347)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:50)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
at
org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:67)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1143)
at
org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:841)
at
org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:717)
at org.apache.hadoop.mapred.JobClient.access$400(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:927)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.runIteration(CVB0Driver.java:515)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:305)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:187)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.main(CVB0Driver.java:548)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:282)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:198)
... 37 more
Build step 'Execute shell' marked build as failure
