[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13808 FWIW, I remember I had a hard time to figure out https://issues.apache.org/jira/browse/SPARK-14103 where the issue itself was about quote but it ended up reading whole partition as a value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13808 @davies, removed `StringIteratorReader` concatenates the lines in each iterator into reader in each partition IIRC. New line in the column was not supported correctly up to my understanding because rows can spawn across multiple blocks. This is a similar problem that we have not supported multiple JSON lines before up to my knowledge. Currently, we have some open PRs for dealing with multiple lines support by using something like `wholeTextFile` or dealing with each file as a multiple line json, which I think we could solve in that way if any of it is merged. I guess we introduced several regression or behaviour changes when we porting. Would this be acceptable rather than supporting multiple lines one in this way? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user davies commented on the issue: https://github.com/apache/spark/pull/13808 @HyukjinKwon @rxin This patch have a regression: A column that have escaped newline can't be correctly parsed anymore. Should we revert this patch or figure a way to fix that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13808 This looks good. Let me merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13808 @rxin Could you take another look please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13808 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60989/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13808 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13808 **[Test build #60989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60989/consoleFull)** for PR 13808 at commit [`5914689`](https://github.com/apache/spark/commit/59146894f7c8da2d70af984f6aeba14fe9c43ad7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13808 **[Test build #60989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60989/consoleFull)** for PR 13808 at commit [`5914689`](https://github.com/apache/spark/commit/59146894f7c8da2d70af984f6aeba14fe9c43ad7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13808 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13808 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60930/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13808 **[Test build #60930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60930/consoleFull)** for PR 13808 at commit [`978a728`](https://github.com/apache/spark/commit/978a7286255fb16f3ad4270d59cb58fe18c967c9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13808: [SPARK-14480][SQL] Remove meaningless StringIteratorRead...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13808 **[Test build #60930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60930/consoleFull)** for PR 13808 at commit [`978a728`](https://github.com/apache/spark/commit/978a7286255fb16f3ad4270d59cb58fe18c967c9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
