[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855][SQL] Added flatten functions ...
Github user sohum2002 commented on the issue: https://github.com/apache/spark/pull/19454 Thank you all for your comments. I hope to improve in my future PRs. Cheers! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855][SQL] Added flatten functions ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19454 Honestly I don't think it is worth doing this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855][SQL] Added flatten functions ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19454 I actually think this can be confusing on Dataset[T], when the Dataset is just untyped and a DataFrame. Do we throw a runtime exception there? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855][SQL] Added flatten functions ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19454 Is this worth doing? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855][SQL] Added flatten functions ...
Github user sohum2002 commented on the issue: https://github.com/apache/spark/pull/19454 @HyukjinKwon - Thank you for your comments and analysis of this PR. I will also try to improve the `flatMap(identity)` as mentioned by @srowen. Also, will add a python implementation. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19454 BTW, for the answer to https://github.com/apache/spark/pull/19454#issuecomment-335138642, I think you should take a look at, for example, `flatMap` as a reference in `rdd.py` and related tests, for example, see `cd ./python/pyspark && grep -r "flatMap" tests.py` and Python [doctest](https://docs.python.org/2/library/doctest.html). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19454 I think @srowen requested to fix it in a more performant way as well, for example, referring https://github.com/apache/spark/pull/16276, if I understood correctly and otherwise closing it. I don't feel strongly about adding this but I was thinking that we might have to go ahead given this API has been required multiple times without explicit objection IIUC and, looks consistent with Scala's [`flatten`]( https://github.com/scala/scala/blob/05016d9035ab9b1c866bd9f12fdd0491f1ea0cbb/src/library/scala/collection/generic/GenericTraversableTemplate.scala#L169). However, IMHO, it might be worthwhile _only if_ this PR gives a clean shot. I'd suggest to close this if we (you and other reviewers here) have to spend a lot of time. Workaround is quite easy anyway. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82550/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19454 **[Test build #82550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82550/testReport)** for PR 19454 at commit [`cc08623`](https://github.com/apache/spark/commit/cc08623519f4ddfdfcc883557c4cc53f11e6f0f7). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19454 **[Test build #82550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82550/testReport)** for PR 19454 at commit [`cc08623`](https://github.com/apache/spark/commit/cc08623519f4ddfdfcc883557c4cc53f11e6f0f7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19454 Let's fix up the PR title from `[SPARK-18855 ][SQL]` to `[SPARK-18855][SQL]` BTW. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user sohum2002 commented on the issue: https://github.com/apache/spark/pull/19454 Would appreciate some help in the Python implementation of the `flatten` function as I have never used pyspark. Could someone help me out? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19454 **[Test build #82542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82542/testReport)** for PR 19454 at commit [`261e45a`](https://github.com/apache/spark/commit/261e45a9a2298df2d4d1f9adc1ca1ced22e90b60). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82542/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19454 **[Test build #82542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82542/testReport)** for PR 19454 at commit [`261e45a`](https://github.com/apache/spark/commit/261e45a9a2298df2d4d1f9adc1ca1ced22e90b60). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19454 This is missing from Python and Java. It also doesn't bother to implement this more efficiently than flatMap(identity). I am not sure this is worth while? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82541/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19454 **[Test build #82541 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82541/testReport)** for PR 19454 at commit [`075e7ef`](https://github.com/apache/spark/commit/075e7ef3f27af91c5190d039770cf15b08a66c81). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19454 **[Test build #82541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82541/testReport)** for PR 19454 at commit [`075e7ef`](https://github.com/apache/spark/commit/075e7ef3f27af91c5190d039770cf15b08a66c81). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19454 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19454 Could you please add test cases? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19454: [SPARK-22152][SPARK-18855 ][SQL] Added flatten functions...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19454 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
