Mahout uses a few file formats quite a bit in its various job implementations.
![]() Table of Contents |
File formats
Raw formats for import
- Text files
- can be parsed into SequenceFiles of:
- (line number, text of line)
- (file name, full contents of file)
- (line number, parts of line extracted with regex patterns)
- can also be parsed into Lucene indexes:
- ARFF files
- Weka project text data format
- Parsed into SequenceFile of <Int,Vector>
- Mailbox files
- can be parsed into SequenceFiles of:
- (mail message id, text body of mail message)
- no html or attachment support
- CSV files
- generally without column or row headers
- no "multiple values per column" options
- Hadoop SequenceFile
- canonical, no variations. Currently no use of metadata.
- Lucene indexes
- translated into SequenceFiles
Raw formats for export
- SequenceFiles
- Text lines, mostly of the toString() variety
- MatrixWritable for ConfusionMatrix
- CSV for MatrixWritable
- A special CSV format for Clusters
- GraphML XML for Clusters
Who Stores What in a SequenceFile?
"Simple" Text Vectors
Simple text vectors represent documents. The dimensions are the set of terms in the entire document set. Each document vector stores a number in the position of each term it contains. This number may be derived from the count of the term inside the document.
Encoded Text Vectors
Each vector represents a document. However, term dimensions are "collapsed" stochastically, meaning each term in the full term set is mapped randomly to several smaller indexes.
Directories
<Integer,Text> pairs which match matrix rows to input text keys like movie names, document file names etc. These are made by RowIdJob.
Matrices
Matrices are almost universally stored as LongWritable/VectorWritable pairs, where VectorWritable can be sparse or dense.
Clusters
Clusters are stored in complex data structures.
FPGrowth Clusters
These are stored in a custom data structure.
Life cycle
All Mahout jobs generally assume that files generated have no lifespan. All Writable formats may change, and some may disappear. There are no file compatibility requirements.
Data Formats 