[jira] [Commented] (SPARK-28779) CSV writer doesn't handle older Mac line endings
[ https://issues.apache.org/jira/browse/SPARK-28779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913151#comment-16913151 ] nicolas paris commented on SPARK-28779: --- good to know thanks > CSV writer doesn't handle older Mac line endings > > > Key: SPARK-28779 > URL: https://issues.apache.org/jira/browse/SPARK-28779 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.4.0 >Reporter: nicolas paris >Priority: Minor > > The spark csv writer does not consider "\r" as a newline in string type > columns. As a result, the resulting csv are not quoted, and they get > corrupted. > All \n, \r\n and \r should be considered as newline to allow robust csv > serialization. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28779) CSV writer doesn't handle older Mac line endings
[ https://issues.apache.org/jira/browse/SPARK-28779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912655#comment-16912655 ] nicolas paris commented on SPARK-28779: --- i cannot find the lineSep option in the dataframeReader/writer csv method API. This exists in the json method, maybe that's what you were thinking about ? https://spark.apache.org/docs/2.4.0/api/scala/index.html#org.apache.spark.sql.DataFrameReader@csv(paths:String*):org.apache.spark.sql.DataFrame > CSV writer doesn't handle older Mac line endings > > > Key: SPARK-28779 > URL: https://issues.apache.org/jira/browse/SPARK-28779 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.4.0 >Reporter: nicolas paris >Priority: Minor > > The spark csv writer does not consider "\r" as a newline in string type > columns. As a result, the resulting csv are not quoted, and they get > corrupted. > All \n, \r\n and \r should be considered as newline to allow robust csv > serialization. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28779) CSV writer doesn't handle older Mac line endings
nicolas paris created SPARK-28779: - Summary: CSV writer doesn't handle older Mac line endings Key: SPARK-28779 URL: https://issues.apache.org/jira/browse/SPARK-28779 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.4.0, 2.3.0 Reporter: nicolas paris The spark csv writer does not consider "\r" as a newline in string type columns. As a result, the resulting csv are not quoted, and they get corrupted. All \n, \r\n and \r should be considered as newline to allow robust csv serialization. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23954) Converting spark dataframe containing int64 fields to R dataframes leads to impredictable errors.
nicolas paris created SPARK-23954: - Summary: Converting spark dataframe containing int64 fields to R dataframes leads to impredictable errors. Key: SPARK-23954 URL: https://issues.apache.org/jira/browse/SPARK-23954 Project: Spark Issue Type: Bug Components: SparkR Affects Versions: 2.3.0 Reporter: nicolas paris Converting spark dataframe containing int64 fields to R dataframes leads to impredictable errors. The problems comes from R that does not handle int64 natively. As a result a good workaround would be to convert bigint as strings when transforming spark dataframes into R dataframes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
