eatoncys commented on issue #23010: [SPARK-26012][SQL]Null and '' values should not cause dynamic partition failure of string types URL: https://github.com/apache/spark/pull/23010#issuecomment-481499536 > I just realized that we may create `InsertIntoHadoopFsRelationCommand` on the fly, e.g. in `OptimizedCreateHiveTableAsSelectCommand`. In this case an anlyze rule can not help. > > Another idea is in `FileFormatWriter.write`, we add a project above the input `SparkPlan` to add the `Empty2Null` expression. Sorry for the back and forth. Do you mean to move it back to `FileFormatWriter.write` ? can I submit another PR to do it in `FileFormatWriter.write` to compare with this PR, if it is better, we can close this one.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
