[
https://issues.apache.org/jira/browse/SPARK-18270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-18270.
----------------------------------
Resolution: Duplicate
I am resolving this as a duplicate of SPARK-16472. Please reopen this if anyone
thinks this is not the duplicate with some reasons.
> Users schema with non-nullable properties is overidden with true
> ----------------------------------------------------------------
>
> Key: SPARK-18270
> URL: https://issues.apache.org/jira/browse/SPARK-18270
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Jork Zijlstra
>
> Users schema with non-nullable properties is overidden with true in
> CSVRelation.csvParser.
> The schema that is given to the CSVRelation.csvParser(schema: StructType)
> isnt the version that is user specifies.
> All nullable option are set to true
> Specifying the schema:
> StructType(Array(
> StructField("id", IntegerType, nullable = false),
> StructField("underlyingId", IntegerType, true)
> ))
> Read the data:
> sparkSession.read
> .schema(sourceSchema)
> .option("header", "false")
> .option("delimiter", """\t""")
> .csv(files(dates): _*)
> .rdd
> Actual Result:
> schema inside csvParser contains only nullable = true values.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
