Re: types allowed for saveasobjectfile?
Yes, any java serializable object. Its important to note that since its saving serialized objects it is as brittle as java serialization when it comes to version changes, so if you can make your data fit in something like sequence files, parquet, avro, or similar it can be not only more space efficient but also more future proof. On Thu, Aug 27, 2015 at 2:24 PM, Arun Luthra arun.lut...@gmail.com wrote: Ah, yes, that did the trick. So more generally, can this handle any serializable object? On Thu, Aug 27, 2015 at 2:11 PM, Jonathan Coveney jcove...@gmail.com wrote: array[String] doesn't pretty print by default. Use .mkString(,) for example El jueves, 27 de agosto de 2015, Arun Luthra arun.lut...@gmail.com escribió: What types of RDD can saveAsObjectFile(path) handle? I tried a naive test with an RDD[Array[String]], but when I tried to read back the result with sc.objectFile(path).take(5).foreach(println), I got a non-promising output looking like: [Ljava.lang.String;@46123a [Ljava.lang.String;@76123b [Ljava.lang.String;@13144c [Ljava.lang.String;@75146d [Ljava.lang.String;@79118f Arun -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
Re: types allowed for saveasobjectfile?
Ah, yes, that did the trick. So more generally, can this handle any serializable object? On Thu, Aug 27, 2015 at 2:11 PM, Jonathan Coveney jcove...@gmail.com wrote: array[String] doesn't pretty print by default. Use .mkString(,) for example El jueves, 27 de agosto de 2015, Arun Luthra arun.lut...@gmail.com escribió: What types of RDD can saveAsObjectFile(path) handle? I tried a naive test with an RDD[Array[String]], but when I tried to read back the result with sc.objectFile(path).take(5).foreach(println), I got a non-promising output looking like: [Ljava.lang.String;@46123a [Ljava.lang.String;@76123b [Ljava.lang.String;@13144c [Ljava.lang.String;@75146d [Ljava.lang.String;@79118f Arun
Re: types allowed for saveasobjectfile?
array[String] doesn't pretty print by default. Use .mkString(,) for example El jueves, 27 de agosto de 2015, Arun Luthra arun.lut...@gmail.com escribió: What types of RDD can saveAsObjectFile(path) handle? I tried a naive test with an RDD[Array[String]], but when I tried to read back the result with sc.objectFile(path).take(5).foreach(println), I got a non-promising output looking like: [Ljava.lang.String;@46123a [Ljava.lang.String;@76123b [Ljava.lang.String;@13144c [Ljava.lang.String;@75146d [Ljava.lang.String;@79118f Arun
Re: types allowed for saveasobjectfile?
So println of any array of strings will look like that. The java.util.Arrays class has some options to print arrays nicely. On Thu, Aug 27, 2015 at 2:08 PM, Arun Luthra arun.lut...@gmail.com wrote: What types of RDD can saveAsObjectFile(path) handle? I tried a naive test with an RDD[Array[String]], but when I tried to read back the result with sc.objectFile(path).take(5).foreach(println), I got a non-promising output looking like: [Ljava.lang.String;@46123a [Ljava.lang.String;@76123b [Ljava.lang.String;@13144c [Ljava.lang.String;@75146d [Ljava.lang.String;@79118f Arun -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau