inferSchema() will work better than jsonRDD() in your case,
>>> from pyspark.sql import Row
>>> srdd = sqlContext.inferSchema(rdd.map(lambda x: Row(**x)))
>>> srdd.first()
Row( field1=5, field2='string', field3={'a'=1, 'c'=2})
On Wed, Dec 3, 2014 at 12:11 AM, sahanbull <[email protected]> wrote:
> Hi Guys,
>
> I am trying to use SparkSQL to convert an RDD to SchemaRDD so that I can
> save it in parquet format.
>
> A record in my RDD has the following format:
>
> RDD1
> {
> field1:5,
> field2: 'string',
> field3: {'a':1, 'c':2}
> }
>
> I am using field3 to represent a "sparse vector" and it can have keys:
> 'a','b' or 'c' and values any int value
>
> The current approach I am using is :
> schemaRDD1 = sqc.jsonRDD(RDD1.map(lambda x: simplejson.dumps(x)))
>
> But when I do this, the dictionary in field 3 also gets converted to a
> SparkSQL Row. This converts field3 to be a dense data structure where it
> holds value None for every key that is not present in the field 3 for each
> record.
>
> When I do
>
> test = RDD1.map(lambda x: simplejson.dumps(x))
> test.first()
>
> my output is: {"field1": 5, "field2":"string", "field3" :{"a":1,"c":2}}
>
> But then when I do
> schemaRDD = sqc.jsonRDD(test)
> schemaRDD.first()
>
> my output is : Row( field1=5, field2='string', field3 = Row(a=1,b=None,c=2))
>
> in realty, I have 1000s of probable keys in field 3 and only 2 to 3 of them
> occur per record. So When tic converts to a Row, it generates thousands of
> None fields per record.
> Is there anyways for me to store "field3" as a dictionary instead of
> converting it into a Row in the schemaRDD??
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Using-sparkSQL-to-convert-a-collection-of-python-dictionary-of-dictionaries-to-schma-RDD-tp20228.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
