Sandeep Pal created SPARK-9040:
----------------------------------
Summary: StructField datatype Conversion Error
Key: SPARK-9040
URL: https://issues.apache.org/jira/browse/SPARK-9040
Project: Spark
Issue Type: Bug
Affects Versions: 1.3.0
Environment: Cloudera 5.3 on CDH 6
Reporter: Sandeep Pal
The following issue occurs if I specify the StructFields in specific order in
StructType as follow:
fields = [StructField("d", IntegerType(), True),StructField("b", IntegerType(),
True),StructField("a", StringType(), True),StructField("c", IntegerType(),
True)]
But the following code words fine:
fields = [StructField("d", IntegerType(), True),StructField("b", IntegerType(),
True),StructField("c", IntegerType(), True),StructField("a", StringType(),
True)]
<ipython-input-27-9d675dd6a2c9> in <module>()
18
19 schema = StructType(fields)
---> 20 schemasimid_simple = sqlContext.createDataFrame(simid_simplereqfields,
schema)
21 schemasimid_simple.registerTempTable("simid_simple")
/usr/local/bin/spark-1.3.1-bin-hadoop2.6/python/pyspark/sql/context.py in
createDataFrame(self, data, schema, samplingRatio)
302
303 for row in rows:
--> 304 _verify_type(row, schema)
305
306 # convert python objects to sql data
/usr/local/bin/spark-1.3.1-bin-hadoop2.6/python/pyspark/sql/types.py in
_verify_type(obj, dataType)
986 "length of fields (%d)" % (len(obj),
len(dataType.fields)))
987 for v, f in zip(obj, dataType.fields):
--> 988 _verify_type(v, f.dataType)
989
990 _cached_cls = weakref.WeakValueDictionary()
/usr/local/bin/spark-1.3.1-bin-hadoop2.6/python/pyspark/sql/types.py in
_verify_type(obj, dataType)
970 if type(obj) not in _acceptable_types[_type]:
971 raise TypeError("%s can not accept object in type %s"
--> 972 % (dataType, type(obj)))
973
974 if isinstance(dataType, ArrayType):
TypeError: StringType can not accept object in type <type 'int'>
But the following code words fine:
fields = [StructField("d", IntegerType(), True),StructField("b", IntegerType(),
True),StructField("c", IntegerType(), True),StructField("a", StringType(),
True)]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
