Re: Converting a json struct to map

2014-11-19 Thread Michael Armbrust 
You can override the schema inference by passing a schema as the second
argument to jsonRDD, however thats not a super elegant solution.  We are
considering one option to make this easier here:
https://issues.apache.org/jira/browse/SPARK-4476

On Tue, Nov 18, 2014 at 11:06 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:

 Something like this?

val map_rdd = json_rdd.map(json = {
   val mapper = new ObjectMapper() with ScalaObjectMapper
   mapper.registerModule(DefaultScalaModule)

   val myMap = mapper.readValue[Map[String,String]](json)

   myMap
 })

 Thanks
 Best Regards

 On Wed, Nov 19, 2014 at 11:01 AM, Daniel Haviv danielru...@gmail.com
 wrote:

 Hi,
 I'm loading a json file into a RDD and then save that RDD as parquet.
 One of the fields is a map of keys and values but it is being translated
 and stored as a struct.

 How can I convert the field into a map?


 Thanks,
 Daniel

Re: Converting a json struct to map

2014-11-19 Thread Daniel Haviv 
Thank you Michael
I will try it out tomorrow

Daniel

 On 19 בנוב׳ 2014, at 21:07, Michael Armbrust mich...@databricks.com wrote:
 
 You can override the schema inference by passing a schema as the second 
 argument to jsonRDD, however thats not a super elegant solution.  We are 
 considering one option to make this easier here: 
 https://issues.apache.org/jira/browse/SPARK-4476
 
 On Tue, Nov 18, 2014 at 11:06 PM, Akhil Das ak...@sigmoidanalytics.com 
 wrote:
 Something like this?
 
val map_rdd = json_rdd.map(json = {
   val mapper = new ObjectMapper() with ScalaObjectMapper
   mapper.registerModule(DefaultScalaModule)
 
   val myMap = mapper.readValue[Map[String,String]](json)
   
   myMap
 })
 
 Thanks
 Best Regards
 
 On Wed, Nov 19, 2014 at 11:01 AM, Daniel Haviv danielru...@gmail.com 
 wrote:
 Hi,
 I'm loading a json file into a RDD and then save that RDD as parquet.
 One of the fields is a map of keys and values but it is being translated 
 and stored as a struct.
 
 How can I convert the field into a map?
 
 
 Thanks,
 Daniel

Re: Converting a json struct to map

2014-11-19 Thread Yin Huai 
Oh, actually, we do not support MapType provided by the schema given to
jsonRDD at the moment (my bad..). Daniel, you need to wait for the patch of
4476 (I should have one soon).

Thanks,

Yin

On Wed, Nov 19, 2014 at 2:32 PM, Daniel Haviv danielru...@gmail.com wrote:

 Thank you Michael
 I will try it out tomorrow

 Daniel

 On 19 בנוב׳ 2014, at 21:07, Michael Armbrust mich...@databricks.com
 wrote:

 You can override the schema inference by passing a schema as the second
 argument to jsonRDD, however thats not a super elegant solution.  We are
 considering one option to make this easier here:
 https://issues.apache.org/jira/browse/SPARK-4476

 On Tue, Nov 18, 2014 at 11:06 PM, Akhil Das ak...@sigmoidanalytics.com
 wrote:

 Something like this?

val map_rdd = json_rdd.map(json = {
   val mapper = new ObjectMapper() with ScalaObjectMapper
   mapper.registerModule(DefaultScalaModule)

   val myMap = mapper.readValue[Map[String,String]](json)

   myMap
 })

 Thanks
 Best Regards

 On Wed, Nov 19, 2014 at 11:01 AM, Daniel Haviv danielru...@gmail.com
 wrote:

 Hi,
 I'm loading a json file into a RDD and then save that RDD as parquet.
 One of the fields is a map of keys and values but it is being translated
 and stored as a struct.

 How can I convert the field into a map?


 Thanks,
 Daniel

Converting a json struct to map

2014-11-18 Thread Daniel Haviv 
Hi,
I'm loading a json file into a RDD and then save that RDD as parquet.
One of the fields is a map of keys and values but it is being translated and 
stored as a struct.

How can I convert the field into a map?


Thanks,
Daniel

Re: Converting a json struct to map

2014-11-18 Thread Akhil Das 
Something like this?

   val map_rdd = json_rdd.map(json = {
  val mapper = new ObjectMapper() with ScalaObjectMapper
  mapper.registerModule(DefaultScalaModule)

  val myMap = mapper.readValue[Map[String,String]](json)

  myMap
})

Thanks
Best Regards

On Wed, Nov 19, 2014 at 11:01 AM, Daniel Haviv danielru...@gmail.com
wrote:

 Hi,
 I'm loading a json file into a RDD and then save that RDD as parquet.
 One of the fields is a map of keys and values but it is being translated
 and stored as a struct.

 How can I convert the field into a map?


 Thanks,
 Daniel