[
https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201308#comment-15201308
]
François Prunier commented on SPARK-11319:
------------------------------------------
Hi [~yhuai],
I'm not familiar with python, I modified the scala documentation thinking
Python was only a binding, but it seems the types are part of the binding if I
understand correctly. That brings a few questions to mind:
1) should I extend my PR to python doc ? Any pointer to where that would be
done would be appreciated. Do I need to add R in there as well?
2) why not enforce the nullable constraint in scala as well ? And maybe R as
well ?
3) should I drop my PR completely as it would be rendered useless if it's now
enforced across the board ? Or would it be useful for a version 1.7 ?
Thanks,
François
> PySpark silently accepts null values in non-nullable DataFrame fields.
> ----------------------------------------------------------------------
>
> Key: SPARK-11319
> URL: https://issues.apache.org/jira/browse/SPARK-11319
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Reporter: Kevin Cox
>
> Running the following code with a null value in a non-nullable column
> silently works. This makes the code incredibly hard to trust.
> {code}
> In [2]: from pyspark.sql.types import *
> In [3]: sqlContext.createDataFrame([(None,)], StructType([StructField("a",
> TimestampType(), False)])).collect()
> Out[3]: [Row(a=None)]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
