Re: types allowed for saveasobjectfile?

2015-08-27 Thread Holden Karau 
Yes, any java serializable object. Its important to note that since its
saving serialized objects it is as brittle as java serialization when it
comes to version changes, so if you can make your data fit in something
like sequence files, parquet, avro, or similar it can be not only more
space efficient but also more future proof.

On Thu, Aug 27, 2015 at 2:24 PM, Arun Luthra arun.lut...@gmail.com wrote:

 Ah, yes, that did the trick.

 So more generally, can this handle any serializable object?

 On Thu, Aug 27, 2015 at 2:11 PM, Jonathan Coveney jcove...@gmail.com
 wrote:

 array[String] doesn't pretty print by default. Use .mkString(,) for
 example


 El jueves, 27 de agosto de 2015, Arun Luthra arun.lut...@gmail.com
 escribió:

 What types of RDD can saveAsObjectFile(path) handle? I tried a naive
 test with an RDD[Array[String]], but when I tried to read back the result
 with sc.objectFile(path).take(5).foreach(println), I got a non-promising
 output looking like:

 [Ljava.lang.String;@46123a
 [Ljava.lang.String;@76123b
 [Ljava.lang.String;@13144c
 [Ljava.lang.String;@75146d
 [Ljava.lang.String;@79118f


 Arun





-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau

types allowed for saveasobjectfile?

2015-08-27 Thread Arun Luthra 
What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
with an RDD[Array[String]], but when I tried to read back the result with
sc.objectFile(path).take(5).foreach(println), I got a non-promising output
looking like:

[Ljava.lang.String;@46123a
[Ljava.lang.String;@76123b
[Ljava.lang.String;@13144c
[Ljava.lang.String;@75146d
[Ljava.lang.String;@79118f


Arun

Re: types allowed for saveasobjectfile?

2015-08-27 Thread Arun Luthra 
Ah, yes, that did the trick.

So more generally, can this handle any serializable object?

On Thu, Aug 27, 2015 at 2:11 PM, Jonathan Coveney jcove...@gmail.com
wrote:

 array[String] doesn't pretty print by default. Use .mkString(,) for
 example


 El jueves, 27 de agosto de 2015, Arun Luthra arun.lut...@gmail.com
 escribió:

 What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
 with an RDD[Array[String]], but when I tried to read back the result with
 sc.objectFile(path).take(5).foreach(println), I got a non-promising output
 looking like:

 [Ljava.lang.String;@46123a
 [Ljava.lang.String;@76123b
 [Ljava.lang.String;@13144c
 [Ljava.lang.String;@75146d
 [Ljava.lang.String;@79118f


 Arun

Re: types allowed for saveasobjectfile?

2015-08-27 Thread Jonathan Coveney 
array[String] doesn't pretty print by default. Use .mkString(,) for
example

El jueves, 27 de agosto de 2015, Arun Luthra arun.lut...@gmail.com
escribió:

 What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
 with an RDD[Array[String]], but when I tried to read back the result with
 sc.objectFile(path).take(5).foreach(println), I got a non-promising output
 looking like:

 [Ljava.lang.String;@46123a
 [Ljava.lang.String;@76123b
 [Ljava.lang.String;@13144c
 [Ljava.lang.String;@75146d
 [Ljava.lang.String;@79118f


 Arun

Re: types allowed for saveasobjectfile?

2015-08-27 Thread Holden Karau 
So println of any array of strings will look like that. The
java.util.Arrays class has some options to print arrays nicely.

On Thu, Aug 27, 2015 at 2:08 PM, Arun Luthra arun.lut...@gmail.com wrote:

 What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
 with an RDD[Array[String]], but when I tried to read back the result with
 sc.objectFile(path).take(5).foreach(println), I got a non-promising output
 looking like:

 [Ljava.lang.String;@46123a
 [Ljava.lang.String;@76123b
 [Ljava.lang.String;@13144c
 [Ljava.lang.String;@75146d
 [Ljava.lang.String;@79118f


 Arun




-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau