[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456662#comment-17456662 ] Apache Spark commented on SPARK-37575: -- User 'wayneguow' has created a pull request

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456172#comment-17456172 ] Hyukjin Kwon commented on SPARK-37575: -- We don't usually assign issues to someone.

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-08 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456168#comment-17456168 ] Guo Wei commented on SPARK-37575: - I  have made some bug fix in my local repo and add so

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-08 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456167#comment-17456167 ] Guo Wei commented on SPARK-37575: - I also found that if emptyValueInRead set to "\"\"",

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17455055#comment-17455055 ] Hyukjin Kwon commented on SPARK-37575: -- Yeah, it would be great to fix it, and upgr

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-08 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17455053#comment-17455053 ] Guo Wei commented on SPARK-37575: - I think I can make a fast repair to verify my conclus

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-08 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17455043#comment-17455043 ] Guo Wei commented on SPARK-37575: - I think I maybe have found the root cause through deb

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-07 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454989#comment-17454989 ] Guo Wei commented on SPARK-37575: - Spark 3.2.0 has the same behavior. > Empty strings a

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454976#comment-17454976 ] Hyukjin Kwon commented on SPARK-37575: -- Spark 2.4.X is EOL so it won't likely be fi

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-07 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454974#comment-17454974 ] Guo Wei commented on SPARK-37575: - As default writerSettings in CSVOptions,  nullValue i

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454968#comment-17454968 ] Hyukjin Kwon commented on SPARK-37575: -- can you set nullValue and emptyValue option

[jira] [Commented] (SPARK-37575) Empty strings and null values are both saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)

2021-12-07 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454966#comment-17454966 ] Guo Wei commented on SPARK-37575: - related issues: https://issues.apache.org/jira/brows