[
https://issues.apache.org/jira/browse/MAHOUT-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14592641#comment-14592641
]
ASF GitHub Bot commented on MAHOUT-1653:
----------------------------------------
Github user andrewpalumbo commented on a diff in the pull request:
https://github.com/apache/mahout/pull/136#discussion_r32784888
--- Diff:
spark/src/main/scala/org/apache/mahout/sparkbindings/drm/CheckpointedDrmSpark.scala
---
@@ -165,7 +168,14 @@ class CheckpointedDrmSpark[K: ClassTag](
else if (classOf[Writable].isAssignableFrom(ktag.runtimeClass)) (x:
K) => x.asInstanceOf[Writable]
else throw new IllegalArgumentException("Do not know how to convert
class tag %s to Writable.".format(ktag))
- rdd.saveAsSequenceFile(path)
+ // rdd.saveAsSequenceFile(path)
+ // this is a (working) deprecated method used as a stop-gap while we
investigate the shell issues
+ SparkContext.rddToSequenceFileRDDFunctions(rdd.asInstanceOf[RDD[(K,
Vector)]]).saveAsSequenceFile(path)
+ // discussion here: http://search-lucene.com/m/rcu7o1J1Z7mvnGmP1
+ // The correct function to use is here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1671
+ // The commit message for this change is here:
https://github.com/apache/spark/tree/master/core/src/test/scala/org/apache/sparktest
+ // The key and value factories will be determined using reflection on
the RDD if the right classes are set to null
+ // see this example:
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/SparkContext.scala#L2236
}
--- End diff --
I think that it should be something like:
```
implicit val keyWritableFactory =
spark.WritableFactory.writableWritableFactory[IntWritable]
implicit val valueWritableFactory =
spark.WritableFactory.writableWritableFactory[VectorWritable]
```
For an Int-keyed Drm. Similarly with String-keyed, and Long-keyed.
> Spark 1.3
> ---------
>
> Key: MAHOUT-1653
> URL: https://issues.apache.org/jira/browse/MAHOUT-1653
> Project: Mahout
> Issue Type: Dependency upgrade
> Reporter: Andrew Musselman
> Assignee: Andrew Palumbo
> Fix For: 0.11.0
>
>
> Support Spark 1.3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
