[
https://issues.apache.org/jira/browse/SPARK-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543403#comment-14543403
]
Santiago M. Mola commented on SPARK-7012:
-----------------------------------------
In Spark SQL, every expression can be nullable or not (i.e. values can be null
or not). All Spark SQL and Catalyst internals support specifying this.
See, for example, StructField, which is the relevant class for schemas:
https://github.com/apache/spark/blob/2d6612cc8b98f767d73c4d15e4065bf3d6c12ea7/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructField.scala#L31
Or AttributeReference:
https://github.com/apache/spark/blob/c1080b6fddb22d84694da2453e46a03fbc041576/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala#L166
However, when creating a temporary table through a SQL statement (CREATE
TEMPORARY TABLE), there is no way of specifying if a column is nullable or not
(it will be always nullable by default).
Standard SQL supports a constraint called "NOT NULL" to specify that a column
is not nullable. See:
http://www.w3schools.com/sql/sql_notnull.asp
In order to implement this, the parser for "CREATE TEMPORARY TABLE", that is,
DDLParser, should be modifyed to allow "NOT NULL" and set nullable = false
accordingly in StructField. See:
https://github.com/apache/spark/blob/0595b6de8f1da04baceda082553c2aa1aa2cb006/sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala#L176
> Add support for NOT NULL modifier for column definitions on DDLParser
> ---------------------------------------------------------------------
>
> Key: SPARK-7012
> URL: https://issues.apache.org/jira/browse/SPARK-7012
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.3.0
> Reporter: Santiago M. Mola
> Priority: Minor
> Labels: easyfix
>
> Add support for NOT NULL modifier for column definitions on DDLParser. This
> would add support for the following syntax:
> CREATE TEMPORARY TABLE (field INTEGER NOT NULL) ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
