[jira] [Commented] (SPARK-25545) CSV loading with DROPMALFORMED mode doesn't correctly drop rows that do not confirm to non-nullable schema fields

2018-10-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1758#comment-1758 ] Hyukjin Kwon commented on SPARK-25545: -- See the discussion at

[jira] [Commented] (SPARK-25545) CSV loading with DROPMALFORMED mode doesn't correctly drop rows that do not confirm to non-nullable schema fields

2018-09-27 Thread Steven Bakhtiari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630087#comment-16630087 ] Steven Bakhtiari commented on SPARK-25545: -- Hi [~hyukjin.kwon] - is there any rationale for why

[jira] [Commented] (SPARK-25545) CSV loading with DROPMALFORMED mode doesn't correctly drop rows that do not confirm to non-nullable schema fields

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630050#comment-16630050 ] Hyukjin Kwon commented on SPARK-25545: -- The problem is, we convert that into nullable schema. It's

[jira] [Commented] (SPARK-25545) CSV loading with DROPMALFORMED mode doesn't correctly drop rows that do not confirm to non-nullable schema fields

2018-09-26 Thread Steven Bakhtiari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628949#comment-16628949 ] Steven Bakhtiari commented on SPARK-25545: -- Somebody on SO pointed me to this older ticket that