jeff303 commented on issue #17293: [SPARK-19950][SQL] Fix to ignore nullable when df.load() is executed for file-based data source URL: https://github.com/apache/spark/pull/17293#issuecomment-482328688 Does the closure of this PR imply that setting nullable=false in a custom (user-defined) schema will never have an effect when loading CSV or JSON data from a file? In other words, if someone sets `nullable: false` in a custom JSON schema, as in the following scenario, it will be ignored? ``` val customSchemaJson = <...some custom JSON schema...> val customSchema = DataType.fromJson(customSchemaJson).asInstanceOf[StructType] spark.read.schema(customSchema).json("/path/to/data.json") ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
