[
https://issues.apache.org/jira/browse/SPARK-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-1074.
------------------------------
Resolution: Not a Problem
Am I right in thinking that if you want to save a JavaPairRDD to HDFS, you have
key-value pairs, and so you want to use JavaPairRDD.saveAsNewAPIHadoopFile, and
SparkContext.sequenceFile to read it? This works. objectFile doesn't seem like
the right approach anyway.
> JavaPairRDD as Object File
> --------------------------
>
> Key: SPARK-1074
> URL: https://issues.apache.org/jira/browse/SPARK-1074
> Project: Spark
> Issue Type: Bug
> Components: Input/Output, Java API
> Affects Versions: 0.9.0
> Reporter: Kevin Mader
> Priority: Minor
>
> So I can perform a save command on a JavaPairRDD
> {code:java}
> static public void HSave(JavaPairRDD<D3int, int[]> baseImg,String path) {
> final String outpath=(new File(path)).getAbsolutePath();
> baseImg.saveAsObjectFile(outpath);
> }
> {code}
> When I use the objectFile command from the JavaSparkContext
> {code:java}
> static public ReadObjectFile(JavaSparkContext jsc, final String path) {
> JavaPairRDD<D3int, int[]> newImage=(JavaPairRDD<D3int,int[]>)
> jsc.objectFile(path);
> }
> {code}
> I get an error cannot cast from JavaRDD to JavaPairRDD. Is there a way to get
> back to JavaPairRDD or will I need to map my data to a JavaRDD, save, load,
> then remap the JavaRDD back to the JavaPairRDD
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
