[
https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976102#comment-16976102
]
Maxim Gekk commented on SPARK-29575:
------------------------------------
This is intentional behavior. User's schema is forcibly set as nullable. See
SPARK-23173
> from_json can produce nulls for fields which are marked as non-nullable
> -----------------------------------------------------------------------
>
> Key: SPARK-29575
> URL: https://issues.apache.org/jira/browse/SPARK-29575
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.4
> Reporter: Victor Lopez
> Priority: Major
>
> I believe this issue was resolved elsewhere
> (https://issues.apache.org/jira/browse/SPARK-23173), though for Pyspark this
> bug seems to still be there.
> The issue appears when using {{from_json}} to parse a column in a Spark
> dataframe. It seems like {{from_json}} ignores whether the schema provided
> has any {{nullable:False}} property.
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(),
> nullable=False)).add(T.StructField('name', T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name':
> 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user",
> schema)).select("details.*").show()
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
