[GitHub] spark issue #21932: [SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp

2018-07-31 Thread holdensmagicalunicorn
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21932 @cloud-fan, thanks! I am a bot who has found some folks who might be able to help with the review:@gatorsmile, @yhuai and @rxin --- --

[GitHub] spark issue #21892: [SPARK-24945][SQL] Switching to uniVocity 2.7.2

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21892 @jbax Thanks for the info! ping @MaxGekk @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apach

[GitHub] spark pull request #20611: [SPARK-23425][SQL]Support wildcard in HDFS path f...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20611#discussion_r206582602 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -303,94 +303,44 @@ case class LoadDataCommand(

[GitHub] spark issue #20611: [SPARK-23425][SQL]Support wildcard in HDFS path for load...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20611 Any behavior change? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark issue #21893: [SPARK-24965][SQL] Support selecting from partitioned ta...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21893 @krisgeus This breaks one of our major assumptions. To provide a proper support, it requires a lot of code changes. Thus, we do not plan to support it in the short term. For your use ca

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93830/ Test FAILed. ---

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93830/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21930 Yeah I would not worry about it --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #21807: [SPARK-24536] Validate that an evaluated limit clause ca...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21807 @mauropalsgraaf Do you have a JIRA ID? I can assign the ticket to you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spar

[GitHub] spark issue #21807: [SPARK-24536] Validate that an evaluated limit clause ca...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21807 LGTM Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional c

[GitHub] spark pull request #21807: [SPARK-24536] Validate that an evaluated limit cl...

2018-07-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21807 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21930 > `logDebug(s\" + cloning the object $obj of class $` Is this normal? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21883 @wangyum How about our Hive tables? Add a test case for Hive tables? They are created by CREATE TABLE instead of CREATE TABLE USING. You can use csv/json format in the test cases. --- ---

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-07-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21889 If the revert is very risky, we normally change the default from true to false when the bug is identified in the RC stage. --- -

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93828/ Test PASSed. ---

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21930 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21930 **[Test build #93828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93828/testReport)** for PR 21930 at commit [`d466a9c`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-07-31 Thread ajacques
Github user ajacques commented on the issue: https://github.com/apache/spark/pull/21889 Where does that leave both of these PRs? Do we still want this one with the code refactoring or to go back to the original? Are there any comments for this PR that would block merging? I've set the

[GitHub] spark issue #21699: [SPARK-24722][SQL] pivot() with Column type argument

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21699 **[Test build #93832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93832/testReport)** for PR 21699 at commit [`cf55135`](https://github.com/apache/spark/commit/cf

[GitHub] spark issue #21928: [SPARK-24976][PYTHON] Allow None for Decimal type conver...

2018-07-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21928 @icexelloss you mean we should change minimum PyArrow version as well? --- - To unsubscribe, e-mail: reviews-unsubscr...@spar

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21930#discussion_r206562302 --- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala --- @@ -123,7 +123,7 @@ abstract class TaskContext extends Serializable { * *

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21883 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21893: [SPARK-24965][SQL] Support selecting from partitioned ta...

2018-07-31 Thread krisgeus
Github user krisgeus commented on the issue: https://github.com/apache/spark/pull/21893 Sorry to hear you don't like to support this although in Hive this works correctly. If chances of getting this merged are so slim I will wait before implementing a configuration switch and add

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21883 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93829/ Test PASSed. ---

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21883 **[Test build #93829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93829/testReport)** for PR 21883 at commit [`46bfcc6`](https://github.com/apache/spark/commit/4

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21930#discussion_r206549150 --- Diff: core/src/test/scala/org/apache/spark/util/ClosureCleanerSuite2.scala --- @@ -538,17 +543,22 @@ class ClosureCleanerSuite2 extends SparkFunSuite wit

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93831/testReport)** for PR 21103 at commit [`72ef664`](https://github.com/apache/spark/commit/72

[GitHub] spark pull request #21825: [SPARK-18188][DOC][FOLLOW-UP]Add `spark.broadcast...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21825#discussion_r206546611 --- Diff: docs/configuration.md --- @@ -1215,6 +1215,14 @@ Apart from these, the following properties are also available, and may be useful if it is

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1521/

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21930#discussion_r206541904 --- Diff: core/src/test/scala/org/apache/spark/util/ClosureCleanerSuite2.scala --- @@ -538,17 +543,22 @@ class ClosureCleanerSuite2 extends SparkFunSuite wit

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21930#discussion_r206542154 --- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala --- @@ -123,7 +123,7 @@ abstract class TaskContext extends Serializable { * *

[GitHub] spark issue #21887: [SPARK-23633][SQL] Update Pandas UDFs section in sql-pro...

2018-07-31 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21887 Thanks! @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21930#discussion_r206540368 --- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala --- @@ -123,7 +123,7 @@ abstract class TaskContext extends Serializable { *

[GitHub] spark issue #21928: [SPARK-24976][PYTHON] Allow None for Decimal type conver...

2018-07-31 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21928 @HyukjinKwon arrow 0.10.0 release is around the corner. I think Spark 2.4 will very likely to ship with 0.10.0 (where I believe this issue has been fixed, @BryanCutler can you confirm?)

[GitHub] spark issue #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over Struct...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20010 I think this is already fixed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-ma

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r206521448 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala --- @@ -61,6 +61,9 @@ private[spark] trait TaskScheduler { */

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r206520439 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -567,4 +567,14 @@ package object config { .intConf

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r206516620 --- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala --- @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-07-31 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21403 no, sorry, you're right. In the case when it is not a subquery, it is safe to treat them as the same. I got confused checking the result of other DBs as they are all behaving differently in that sc

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21599 ah that's a good point. Then maybe we should keep this "java style behavior" and add "strict mode" later. cc @rxin @rednaxelafx --- -

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21403 The example of non-sub-query looks weird. If `where col1 in ((1, 'a'))` can't work, what is the right query if we do have a struct type column? --- --

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-07-31 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21599 @cloud-fan ok, but then all arithmetic operations should be always `nullable = true`. I am not sure whether this can introduce performance regression (the additional checks are also going to worsen

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21927 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93827/ Test PASSed. ---

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21927 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21927 **[Test build #93827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93827/testReport)** for PR 21927 at commit [`0733bfb`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #21782: [SPARK-24816][SQL] SQL interface support repartitionByRa...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21782 Maybe we can follow https://github.com/apache/spark/pull/21911 and use hint? --- - To unsubscribe, e-mail: reviews-unsubscr...@

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21599 I prefer returning null, and introduce a strict mode in Spark 3.0. We can revisit all the returning null cases and think if we should fail if strict mode is on. ---

[GitHub] spark pull request #21852: [SPARK-24893] [SQL] Remove the entire CaseWhen if...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21852#discussion_r206500527 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -416,6 +416,21 @@ object SimplifyConditionals exte

[GitHub] spark pull request #21852: [SPARK-24893] [SQL] Remove the entire CaseWhen if...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21852#discussion_r206499326 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -416,6 +416,21 @@ object SimplifyConditionals exte

[GitHub] spark issue #19788: [SPARK-9853][Core] Optimize shuffle fetch of contiguous ...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19788 > One possible solution is to read all contiguous partition in one shot and then send each shuffle block one by one, how do you think? We may need benchmark performance in this way. SGTM.

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-07-31 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21403 @cloud-fan the problem is that the change is not only for the case when IN is followed by a listquery. The change is needed also in the other case. And the reason why this change is needed is to de

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93830/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/28

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1520/

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21931 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21782: [SPARK-24816][SQL] SQL interface support repartit...

2018-07-31 Thread wangyum
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/21782 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21931 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21930 cc @JoshRosen too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21103 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21931 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-07-31 Thread holdensmagicalunicorn
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21931 @heary-cao, thanks! I am a bot who has found some folks who might be able to help with the review:@rxin, @cloud-fan and @yhuai --- ---

[GitHub] spark pull request #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggrega...

2018-07-31 Thread heary-cao
GitHub user heary-cao opened a pull request: https://github.com/apache/spark/pull/21931 [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row.max.capacity to configure the capacity of fast aggregation. ## What changes were proposed in this pull request? this pr add a conf

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-07-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21403 I'd like to avoid changing the signature of an existing expression if possible. It's not public so we can change it, but I believe some advanced users use these internal classes and we should keep

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93825/ Test FAILed. ---

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93825/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21883 **[Test build #93829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93829/testReport)** for PR 21883 at commit [`46bfcc6`](https://github.com/apache/spark/commit/46

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21883 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1519/

[GitHub] spark issue #21883: [SPARK-24937][SQL] Datasource partition table should loa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21883 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21883: [SPARK-24937][SQL] Datasource partition table sho...

2018-07-31 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/21883#discussion_r206479294 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala --- @@ -284,6 +284,11 @@ object PartitioningUtils {

[GitHub] spark issue #21699: [SPARK-24722][SQL] pivot() with Column type argument

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21699 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21699: [SPARK-24722][SQL] pivot() with Column type argument

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21699 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93826/ Test FAILed. ---

[GitHub] spark issue #21699: [SPARK-24722][SQL] pivot() with Column type argument

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21699 **[Test build #93826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93826/testReport)** for PR 21699 at commit [`34535a9`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21930 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1518/

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21930 @lrytz @retronym @adriaanm @srowen @felixcheung fyi, pls review. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21930 **[Test build #93828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93828/testReport)** for PR 21930 at commit [`d466a9c`](https://github.com/apache/spark/commit/d4

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread holdensmagicalunicorn
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21930 @skonto, thanks! I am a bot who has found some folks who might be able to help with the review:@pwendell, @cloud-fan and @mateiz --- -

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/21930 [SPARK-14540][Core] Fix remaining major issues for Scala 2.12 Support ## What changes were proposed in this pull request? This PR addresses issues 2,3 in the [document](https://docs.google.com/

[GitHub] spark pull request #21669: [SPARK-23257][K8S][WIP] Kerberos Support for Spar...

2018-07-31 Thread nrchakradhar
Github user nrchakradhar commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r206447871 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala --- @@ -107,7 +109,14

[GitHub] spark pull request #21899: [SPARK-24912][SQL] Don't obscure source of OOM du...

2018-07-31 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21899#discussion_r206442258 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala --- @@ -118,12 +119,19 @@ case class BroadcastExchang

[GitHub] spark pull request #21899: [SPARK-24912][SQL] Don't obscure source of OOM du...

2018-07-31 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21899#discussion_r206441736 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala --- @@ -118,12 +119,19 @@ case class BroadcastExchang

[GitHub] spark pull request #21403: [SPARK-24341][SQL] Support only IN subqueries wit...

2018-07-31 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21403#discussion_r206436489 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1422,11 +1422,26 @@ class Analyzer( res

[GitHub] spark issue #21892: [SPARK-24945][SQL] Switching to uniVocity 2.7.2

2018-07-31 Thread jbax
Github user jbax commented on the issue: https://github.com/apache/spark/pull/21892 Did anyone had a chance to test with the 2.7.3-SNAPSHOT build I released to see if the performance issue has been addressed? If it has then let me know and I'll release the final 2.7.3 build. --- -

[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21929 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21929 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21929 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21929: [Streaming-24970][Kinesis] Create WriteAheadLogBa...

2018-07-31 Thread brucezhao11
GitHub user brucezhao11 opened a pull request: https://github.com/apache/spark/pull/21929 [Streaming-24970][Kinesis] Create WriteAheadLogBackedBlockRDD for Kinesis Streaming if WAL is enabled. ## What changes were proposed in this pull request? By default, KinesisInput

[GitHub] spark issue #21929: [Streaming-24970][Kinesis] Create WriteAheadLogBackedBlo...

2018-07-31 Thread holdensmagicalunicorn
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21929 @brucezhao11, thanks! I am a bot who has found some folks who might be able to help with the review:@tdas, @brkyvz and @huaxingao ---

[GitHub] spark pull request #21669: [SPARK-23257][K8S][WIP] Kerberos Support for Spar...

2018-07-31 Thread rvesse
Github user rvesse commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r204403925 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala --- @@ -107,7 +109,14 @@ pri

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-07-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21889 Im that case, I hope my comments are addressed together before merging it in. They are non trivial and I dom't usually go with them if there isn't special reason for it. --- -

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21927 **[Test build #93827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93827/testReport)** for PR 21927 at commit [`0733bfb`](https://github.com/apache/spark/commit/07

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21927 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1517/

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21927 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21927 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21699: [SPARK-24722][SQL] pivot() with Column type argument

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21699 **[Test build #93826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93826/testReport)** for PR 21699 at commit [`34535a9`](https://github.com/apache/spark/commit/34

[GitHub] spark issue #21699: [SPARK-24722][SQL] pivot() with Column type argument

2018-07-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21699 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93825/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/28

<    1   2   3   4   5   6   7   >