Interesting. I'm quite glad to read your explanation, it makes some of our work quite a bit more clear. I'll open a ticket in a similar vein to this discussion: https://github.com/apache/spark/pull/11785, contrasting nullability implementation as optimization and and enforcement.
Additionally, shall I go ahead and open a ticket pointing out the missing call to .asNullable in the streaming reader? Thanks, Alek On Thu, Oct 13, 2016 at 4:05 PM Michael Armbrust <mich...@databricks.com> wrote: There is a <https://issues.apache.org/jira/browse/SPARK-15192> lot <https://github.com/apache/spark/pull/15329> of <https://github.com/apache/spark/pull/14124> confusion <https://github.com/apache/spark/pull/13873> around <https://github.com/apache/spark/pull/11785> nullable <https://issues.apache.org/jira/browse/SPARK-11319> in <https://issues.apache.org/jira/browse/SPARK-11868> StructType <https://issues.apache.org/jira/browse/SPARK-13740> and we should definitly come up with a consistent story and make sure we have better documentation. This might even mean deprecating this field. At a high level, I think the key problem is that internally, nullable is used as an optimization, not an enforcement mechanism. This is a lot different than NOT NULL in a traditional database. Specifically, we take "nulllable = false" as a promise that that column will never be null and use that fact to do optimizations like skipping null checks. This means that if you lie to us, you actually can get wrong answers (i.e. 0 instead of null). This is clearly confusing and not ideal. A little bit of explanation for some of the specific cases you brought up: the reason that we call asNullable on file sources is that even if that column is never null in the data, there are cases that can still produce a null result. For example, when parsing JSON, we null out all columns other than _corrupt_record when we fail to parse a line. The fact that its different in streaming is a bug. Would you mind opening up a JIRA ticket, and we discuss the right path forward there? On Thu, Oct 13, 2016 at 12:35 PM, Aleksander Eskilson < aleksander...@gmail.com> wrote: Hi there, Working in the space of custom Encoders/ExpressionEncoders, I've noticed that the StructType schema as set when creating an object of the ExpressionEncoder[T] class [1] is not the schema actually used to set types for the columns of a Dataset, as created by using the .as(encoder) method [2] on read data. Instead, what occurs is that the schema is either inferred through analysis of the data, or a schema can be provided using the .schema(structType) method [3] of the DataFrameReader. However, when using the .schema(..) method of DataFrameReader, potentially undesirable behaviour occurs: while the DataSource is being resolved, all FieldTypes of the a StructType schema have their nullability set to *true* (using the asNullable function of StructTypes) [4] when the data is read from a local file, as opposed to a non-streaming source. Of course, allowing null-values where they shouldn't exist can weaken the type-guarantees for DataSets over certain types of encoded data. Thinking on how this might be resolved, first, if it's a legitimate bug, I'm not sure why "non-streaming file based" datasources need to have their StructFields all rendered nullable. Simply removing the call to asNullable would fix the issue. Second, if it's actually necessary for most filesystem-read data-sources to have their StructFields potentially nullable in this manner, we could instead let the StructType schema provided to the Encoder have the final say in the DataSet's schema. This latter option seems sensible to me: if a client is willing to provide a custom Encoder via the .as(..) method on the reader, presumably in setting the schema field of the encoder they have some legitimate notion of how their object's types should be mapped to DataSet column types. Any failure when resolving their data to a DataSet by means of their Encoder can then be traced to their Encoder for their own debugging. Thoughts? Thanks, Alek Eskilson [1] - https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala#L213 [2] - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L374 [3] - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L62 [4] - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L426
