[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[justinuang]

Github user justinuang commented on the issue:

https://github.com/apache/spark/pull/20877
  
Sorry, I won't be able to take it over!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
@justinuang are you interested in taking over 
https://github.com/apache/spark/pull/20877#issuecomment-429182740 ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
Ah, happy vacation!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
> ... are you busy? Do you have some time to go for CSV's lineSep? 

@HyukjinKwon I will be on a vacation for 3 weeks but highly likely I will 
be in a place where there is no internet and even mobile networks at all.  Yeh 
there are such places in Russia ;-) . So, even if I prepare a PR, I will be not 
able to response to any comments.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
@MaxGekk, are you busy? Do you have some time to go for CSV's lineSep? I 
think I wouldn't have some time within a couple of weeks. If you have some 
time, I would appreciate if you could go ahead. Otherwise, I will try this one 
after a couple of weeks.

The problem in CSV's lineSep is about multiline support. As you might 
already know, CSV's multiline mode is different with JSON in a way it parses 
line by line from the stream whereas JSON treats it as a whole record in 
general - so we should set the lineSep to Univocity parser as well.

The problem is, `lineSep` at Univocity parser has some limitation 
(https://github.com/apache/spark/pull/18581#issuecomment-314037750 and see also 
`https://github.com/uniVocity/univocity-parsers/issues/170`).

There are some changes made in https://github.com/apache/spark/pull/18581 . 
Might able to extract CSV related change and make some addition and deletion.

If it's difficult to support `lineSep` more than one characters by the 
limitation, I think we can restrict the lineSep only to one character in 
`multiLine` mode.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
thanks all for reviewing this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
yea, I will. I'll be busy for a while but I will make it in the next week 
for sure.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/20877
  
@HyukjinKwon do you wanna send a PR to add `lineSep` for CSV and fix the 
charset problem? thanks!



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/20877
  
thanks, merging to master!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
If this one is merged, I believe it should be easier to review #20885 too.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
There was a discussion about the naming here - 
https://github.com/apache/spark/pull/20727#discussion_r172341859. I am against 
to `recordDelimiter`.

Both PR deal with a different problems. This PR deals with line separator 
only and that PR deals with line separator + flexible option.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20877
  
Since both PRs are ready for review, let us review both and see which one 
is better


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20877
  
Yeah. `recordDelimiter` is better based on the semantics.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88572/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88572 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88572/testReport)**
 for PR 20877 at commit 
[`f5e7d34`](https://github.com/apache/spark/commit/f5e7d34d0e422789fdd979a6a17ee7f48b77d0be).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1744/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88572 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88572/testReport)**
 for PR 20877 at commit 
[`f5e7d34`](https://github.com/apache/spark/commit/f5e7d34d0e422789fdd979a6a17ee7f48b77d0be).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/20877
  
We can also change both if they haven’t been released yet.

On Sun, Mar 25, 2018 at 10:37 AM Maxim Gekk 
wrote:

> @gatorsmile  The PR has been already
> submitted: #20885  . Frankly
> speaking I would prefer another name for the option like we discussed
> before: MaxGekk#1  but similar
> PR for text datasource had been merged already: #20727
>  . And I think it is more
> important to have the same option across all datasource. That's why I
> renamed *recordDelimiter* to *lineSep* in #20885
>  / cc @rxin
> 
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
Correct me if I am wrong. None of renaming or adding more flexible 
functionality to the line separator blocks this PR, right?

Even if we go renaming, we should do it for text datasource too which I 
believe is better to do it separately, and the flexible functionality in the 
line separator looks needing more feedback and discussion.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
@gatorsmile The PR has been already submitted: 
https://github.com/apache/spark/pull/20885 . Frankly speaking I would prefer 
another name for the option like we discussed before: 
https://github.com/MaxGekk/spark-1/pull/1 but similar PR for text datasource 
had been merged already: https://github.com/apache/spark/pull/20727 . And I 
think it is more important to have the same option across all datasource. 
That's why I renamed *recordDelimiter* to *lineSep* in 
https://github.com/apache/spark/pull/20885 / cc @rxin 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
He submitted this - https://github.com/apache/spark/pull/20885 and I 
believe we need more feedback and another review iteration.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20877
  
@MaxGekk Will you submit a PR for addressing the comment 
https://github.com/apache/spark/pull/20877#issuecomment-375622342 in the next 
few weeks? If so, we can hold this PR. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
LGTM


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88570/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88570 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88570/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88570 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88570/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1743/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88568/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88568 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88568/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1741/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88568 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88568/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88567 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88567/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88567/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1740/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88567 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88567/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
@cloud-fan, @MaxGekk, and @hvanhovell, would you mind taking a look please 
when you have some time? I think this is pretty similar with 
https://github.com/apache/spark/pull/20727 except one difference that it uses 
text datasource's `lineSep` option to parse line by line in JSON's schema 
inference.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
I think we are fine to change the behaviour of `lineSep` before the release 
..


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
I have only one concern: if we merge this PR, we close the possibility for 
changing format of `lineSep` and future extensions. Your changes allow any 
sequence of chars. It is not clear for me, how we can restrict it and assign 
different meanings to it in the future.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
Yup, yup. I don't object for now. Shall we merge this one first and talk 
more about it in your PR?
I believe this PR itself proposes a complete option and I saw many the 
requests for this feature here and there like mailing list.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
> Does that fix actual usecases?

I see the following use cases:

1. Jsons coming usually from embedded systems have not-standard separators 
(invisible in some cases). It is very convenient to open a file in hex editor 
and copy bytes between }{ to the lineSep option. This is the use case for the 
format with `'x'` selector like: `x0d 54 45`

2. In Json Streaming, records could be separated in pretty different ways. 
We should leave room for improvement I believe. See `'r'` (for regexp) and 
`'/'` reserved selectors

3. Some UTF-8 chars could cause errors from style (format) checkers. It is 
easier to represent such chars in hexadecimal format instead of disabling the 
checkers.

4. In near future, json datasource will support input json in different 
charsets. If the source code in UTF-8 but input json in different charset, it 
is slightly hard to put such chars as values for the lineSep option. The 
`x` format is more convenient here again. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
@HyukjinKwon We have a few clients who are interested in processing of JSON 
streaming like data. Here is the PR which combines your changes and mine: 
https://github.com/apache/spark/pull/20885


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88518/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88518 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88518/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
I am neutral. Does that fix actual usecases? I can help review anyway. 
Would you like to make a followup separately?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88518 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88518/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1712/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/20877
  
What about to make the option more flexible like in the PR: 
https://github.com/MaxGekk/spark-1/pull/1 ? It would be nice to handle JSON 
Streaming for example: https://en.wikipedia.org/wiki/JSON_streaming


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20877
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88515/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20877
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20877
  
**[Test build #88515 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88515/testReport)**
 for PR 20877 at commit 
[`6cbf1ac`](https://github.com/apache/spark/commit/6cbf1ac2939160eb2b2496e3138a7c96d89877f9).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org