[
https://issues.apache.org/jira/browse/SPARK-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249689#comment-16249689
]
Ruslan Dautkhanov commented on SPARK-22505:
-------------------------------------------
[~hyukjin.kwon] Yep, '1' is of type 'str'. This was made specifically to
demonstrate my point.
As I said in the jira description, I wanted to have the same schema inference
works as expected when reading delimited files (in the old good spark-csv spark
module).
As an example, we read in fixed-width files using sc.binaryRecords(hdfsFile,
recordLength) and then after rdd.map() basically get a very wide modeling
dataset which has all elements / "columns" strings.
We want to engage the same spark-csv type of schema inference so Spark maps
strings by analyzing all strings to come up with actual data types.
We had other scenarios when we want toDF() and/or createDataFrame() API calls
to engage the same schema inference by reading whole dataset and see, like in
example above, '1', '2', '3' "least common" type is type 'int' - again, exactly
what spark-csv logic does. Is this possible in Spark?
> toDF() / createDataFrame() type inference doesn't work as expected
> ------------------------------------------------------------------
>
> Key: SPARK-22505
> URL: https://issues.apache.org/jira/browse/SPARK-22505
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
> Affects Versions: 2.2.0
> Reporter: Ruslan Dautkhanov
> Labels: csvparser, inference, pyspark, schema, spark-sql
>
> {code}
> df =
> sc.parallelize([('1','a'),('2','b'),('3','c')]).toDF(['should_be_int','should_be_str'])
> df.printSchema()
> {code}
> produces
> {noformat}
> root
> |-- should_be_int: string (nullable = true)
> |-- should_be_str: string (nullable = true)
> {noformat}
> Notice `should_be_int` has `string` datatype, according to documentation:
> https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection
> {quote}
> Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the
> datatypes. Rows are constructed by passing a list of key/value pairs as
> kwargs to the Row class. The keys of this list define the column names of the
> table, *and the types are inferred by sampling the whole dataset*, similar to
> the inference that is performed on JSON files.
> {quote}
> Schema inference works as expected when reading delimited files like
> {code}
> spark.read.format('csv').option('inferSchema', True)...
> {code}
> but not when using toDF() / createDataFrame() API calls.
> Spark 2.2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
