[
https://issues.apache.org/jira/browse/SPARK-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983963#comment-13983963
]
Andre Schumacher commented on SPARK-1649:
-----------------------------------------
OK, I now understand that this would be a bigger change.
It's not just struct fields for nested types, array element types, map value
value types, etc. IMHO it would be cleaner to have it inside the DataType. But
since this seems to be mostly relevant only for nested types could one have a
special DataType for them, something like "NestedDataType(val nullable:
Boolean) extends DataType?
> DataType should contain nullable bit
> ------------------------------------
>
> Key: SPARK-1649
> URL: https://issues.apache.org/jira/browse/SPARK-1649
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.1.0
> Reporter: Andre Schumacher
> Priority: Critical
>
> For the underlying storage layer it would simplify things such as schema
> conversions, predicate filter determination and such to record in the data
> type itself whether a column can be nullable. So the DataType type could look
> like like this:
> abstract class DataType(nullable: Boolean = true)
> Concrete subclasses could then override the nullable val. Mostly this could
> be left as the default but when types can be contained in nested types one
> could optimize for, e.g., arrays with elements that are nullable and those
> that are not.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
