[
https://issues.apache.org/jira/browse/MAHOUT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866278#action_12866278
]
Robin Anil commented on MAHOUT-391:
-----------------------------------
Its failing for me .
ava.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at java.io.DataInputStream.readDouble(DataInputStream.java:451)
at
org.apache.mahout.math.VectorWritable.readFields(VectorWritable.java:163)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
at
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceFileRecordReader.java:103)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:78)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
May 12, 2010 12:25:14 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
There is some problem in symmetry between serialization and deserialization. I
would suggest that you use the build reuters script to test it out.
> Make vector more space efficient with variable-length encoding, et al
> ---------------------------------------------------------------------
>
> Key: MAHOUT-391
> URL: https://issues.apache.org/jira/browse/MAHOUT-391
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Sean Owen
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 0.4
>
> Attachments: MAHOUT-391.patch, MAHOUT-391.patch
>
>
> There are a few things we can do to make Vector representations smaller on
> disk:
> - Use variable-length encoding for integer values like size and element
> indices in sparse representations
> - Further, delta-encode indices in sequential representations
> - Let caller specify that precision isn't crucial in values, allowing it to
> store values as floats
> Since indices are usually small-ish, I'd guess this saves 2 bytes or so on
> average, out of 12 bytes per element now.
> Using floats where applicable saves another 4. Not bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
