[GitHub] spark pull request #20877: [SPARK-23765][SQL] Supports custom line separator...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20877


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20877: [SPARK-23765][SQL] Supports custom line separator...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20877#discussion_r177721014
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
 ---
@@ -85,6 +86,16 @@ private[sql] class JSONOptions(
 
   val multiLine = 
parameters.get("multiLine").map(_.toBoolean).getOrElse(false)
 
+  val lineSeparator: Option[String] = parameters.get("lineSep").map { sep 
=>
--- End diff --

this can be private?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20877: [SPARK-23765][SQL] Supports custom line separator...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/20877#discussion_r176945815
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala 
---
@@ -268,6 +268,8 @@ final class DataStreamReader private[sql](sparkSession: 
SparkSession) extends Lo
* `java.text.SimpleDateFormat`. This applies to timestamp type.
* `multiLine` (default `false`): parse one record, which may span 
multiple lines,
* per file
+   * `lineSep` (default covers all `\r`, `\r\n` and `\n`): defines the 
line separator
--- End diff --

Add a test case for testing the default covers `\r`, `\r\n` and `\n`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org