[
https://issues.apache.org/jira/browse/SPARK-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
holdenk closed SPARK-14017.
---------------------------
Resolution: Won't Fix
Thanks for bringing this issue up - I don't think we necessarily want to add
these alias - the type differences are documented in
http://spark.apache.org/docs/latest/sql-programming-guide.html
> dataframe.dtypes -> pyspark.sql.types aliases
> ---------------------------------------------
>
> Key: SPARK-14017
> URL: https://issues.apache.org/jira/browse/SPARK-14017
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 1.5.0
> Environment: Python 2.7; Spark 1.5; Java 1.7; Hadoop 2.6; Scala 2.10
> Reporter: Ruslan Dautkhanov
> Priority: Minor
> Labels: dataframe, datatypes, pyspark, python
>
> Running following:
> #fix schema for gaid which should not be Double
> from pyspark.sql.types import *
> customSchema = StructType()
> for (col,typ) in tsp_orig.dtypes:
> if col=='Agility_GAID':
> typ='string'
> customSchema.add(col,typ,True)
> Getting
> ValueError: Could not parse datatype: bigint
> Looks like pyspark.sql.types doesn't know anything about bigint..
> Should it be aliased to LongType in pyspark.sql.types?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
