[GitHub] [spark] AmplabJenkins commented on pull request #29045: [SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29045: URL: https://github.com/apache/spark/pull/29045#issuecomment-658900416 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] MaxGekk commented on a change in pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
MaxGekk commented on a change in pull request #27366: URL: https://github.com/apache/spark/pull/27366#discussion_r455238390 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmark.scala ## @@ -508,6 +548,9 @@ object JsonBenchmark

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-658914488 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29115: [SPARK-32315][ML] Provide an explanation error message when calling require

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29115: URL: https://github.com/apache/spark/pull/29115#issuecomment-658914448 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-658914488 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29115: [SPARK-32315][ML] Provide an explanation error message when calling require

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29115: URL: https://github.com/apache/spark/pull/29115#issuecomment-658914448 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658914370 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29126: [SPARK-32324][SQL]Fix error messages during using PIVOT and lateral view

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29126: URL: https://github.com/apache/spark/pull/29126#issuecomment-658914030 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] cloud-fan commented on a change in pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-15 Thread GitBox
cloud-fan commented on a change in pull request #29032: URL: https://github.com/apache/spark/pull/29032#discussion_r455237329 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -715,7 +715,8 @@ class DAGSchedulerSuite extends

[GitHub] [spark] AmplabJenkins commented on pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658914370 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-15 Thread GitBox
cloud-fan commented on a change in pull request #29032: URL: https://github.com/apache/spark/pull/29032#discussion_r455237815 ## File path: core/src/main/scala/org/apache/spark/scheduler/DecommissionInfo.scala ## @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] aokolnychyi commented on pull request #29089: [SPARK-32276][SQL] Remove redundant sorts before repartition nodes

2020-07-15 Thread GitBox
aokolnychyi commented on pull request #29089: URL: https://github.com/apache/spark/pull/29089#issuecomment-658932378 @dongjoon-hyun @viirya @hvanhovell @maropu, what do you think? This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-658940338 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] MaxGekk commented on a change in pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
MaxGekk commented on a change in pull request #27366: URL: https://github.com/apache/spark/pull/27366#discussion_r455259788 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/StructFiltersSuite.scala ## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache

[GitHub] [spark] venkata91 commented on pull request #28287: [SPARK-31418][SCHEDULER] Request more executors in case of dynamic allocation is enabled and a task becomes unschedulable due to spark's bl

2020-07-15 Thread GitBox
venkata91 commented on pull request #28287: URL: https://github.com/apache/spark/pull/28287#issuecomment-658957953 > yes we have a test in TaskSchedulerImplSuite that checks to make sure it aborted, but I don't think it covers when dynamic allocation on, so it doesn't hit your new code.

[GitHub] [spark] dongjoon-hyun commented on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

2020-07-15 Thread GitBox
dongjoon-hyun commented on pull request #29121: URL: https://github.com/apache/spark/pull/29121#issuecomment-658958309 It would be great if you mention that in the PR title and PR description. Otherwise, the PR title is misleading. > By suppressing it,

[GitHub] [spark] MaxGekk commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
MaxGekk commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-658966760 jenkins, retest this, please This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dongjoon-hyun commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
dongjoon-hyun commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-658966540 Hi, @jiangxb1987 . Could you ping someone in your mind explicitly like I did at https://github.com/apache/spark/pull/28708#issuecomment-658965320 ? > Please wait for a

[GitHub] [spark] tgravescs commented on pull request #28287: [SPARK-31418][SCHEDULER] Request more executors in case of dynamic allocation is enabled and a task becomes unschedulable due to spark's bl

2020-07-15 Thread GitBox
tgravescs commented on pull request #28287: URL: https://github.com/apache/spark/pull/28287#issuecomment-658974291 you can make a common function that has most of the code that gets called from 2 separate tests. one test passes with dynamic allocation on, the other with it off. that will

[GitHub] [spark] SparkQA commented on pull request #29123: [SPARK-32283][CORE] Kryo should support multiple user registrators

2020-07-15 Thread GitBox
SparkQA commented on pull request #29123: URL: https://github.com/apache/spark/pull/29123#issuecomment-658983163 **[Test build #125880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125880/testReport)** for PR 29123 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29123: [SPARK-32283][CORE] Kryo should support multiple user registrators

2020-07-15 Thread GitBox
SparkQA removed a comment on pull request #29123: URL: https://github.com/apache/spark/pull/29123#issuecomment-658857296 **[Test build #125880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125880/testReport)** for PR 29123 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

2020-07-15 Thread GitBox
dongjoon-hyun commented on pull request #29111: URL: https://github.com/apache/spark/pull/29111#issuecomment-658998288 Hi, @srowen . You last commit passed the GitHub Action. Please see here. - https://github.com/apache/spark/pull/29111/commits/6390b6c46f5bf35e0c92b140bfbe12f98c35cd8f

[GitHub] [spark] dongjoon-hyun commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

2020-07-15 Thread GitBox
dongjoon-hyun commented on pull request #29111: URL: https://github.com/apache/spark/pull/29111#issuecomment-658998506 Also, here. ![Screen Shot 2020-07-15 at 1 41 21 PM](https://user-images.githubusercontent.com/9700541/87593815-e4910e00-c6a0-11ea-9e09-1c8b68fc8ed2.png)

[GitHub] [spark] SparkQA commented on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-15 Thread GitBox
SparkQA commented on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659020484 **[Test build #125909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125909/testReport)** for PR 29032 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29127: [SPARK-32327][SQL] Introduce UnresolvedTableOrPermanentView for commands that support a table and permanent view, but not a temporary v

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29127: URL: https://github.com/apache/spark/pull/29127#issuecomment-659047837 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] karuppayya commented on a change in pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
karuppayya commented on a change in pull request #28804: URL: https://github.com/apache/spark/pull/28804#discussion_r455402409 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2196,6 +2196,25 @@ object SQLConf {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-659047829 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-659047829 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29045: [SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables

2020-07-15 Thread GitBox
SparkQA removed a comment on pull request #29045: URL: https://github.com/apache/spark/pull/29045#issuecomment-658857875 **[Test build #125892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125892/testReport)** for PR 29045 at commit

[GitHub] [spark] SparkQA commented on pull request #29045: [SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables

2020-07-15 Thread GitBox
SparkQA commented on pull request #29045: URL: https://github.com/apache/spark/pull/29045#issuecomment-659047383 **[Test build #125892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125892/testReport)** for PR 29045 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29045: [SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29045: URL: https://github.com/apache/spark/pull/29045#issuecomment-658900416 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] huaxingao commented on pull request #29112: [SPARK-32310][ML][PySpark] ML params default value parity part 1

2020-07-15 Thread GitBox
huaxingao commented on pull request #29112: URL: https://github.com/apache/spark/pull/29112#issuecomment-658901689 cc @srowen @viirya @zhengruifeng This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29015: URL: https://github.com/apache/spark/pull/29015#issuecomment-654419342 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] cloud-fan commented on pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-15 Thread GitBox
cloud-fan commented on pull request #29015: URL: https://github.com/apache/spark/pull/29015#issuecomment-658915498 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29101: [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658936909 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29101: [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658936909 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] tgravescs commented on pull request #28287: [SPARK-31418][SCHEDULER] Request more executors in case of dynamic allocation is enabled and a task becomes unschedulable due to spark's bl

2020-07-15 Thread GitBox
tgravescs commented on pull request #28287: URL: https://github.com/apache/spark/pull/28287#issuecomment-658951229 yes we have a test in TaskSchedulerImplSuite that checks to make sure it aborted, but I don't think it covers when dynamic allocation on, so it doesn't hit your new code. So

[GitHub] [spark] dongjoon-hyun commented on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
dongjoon-hyun commented on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-658958999 Thank you for pinging me, @cloud-fan . This is an automated message from the Apache Git Service. To

[GitHub] [spark] dongjoon-hyun commented on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
dongjoon-hyun commented on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-658959140 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

2020-07-15 Thread GitBox
dongjoon-hyun edited a comment on pull request #29121: URL: https://github.com/apache/spark/pull/29121#issuecomment-658958309 It would be great if you mention `suppressing` in the PR title and PR description. Otherwise, the PR title is misleading.

[GitHub] [spark] venkata91 commented on pull request #28287: [SPARK-31418][SCHEDULER] Request more executors in case of dynamic allocation is enabled and a task becomes unschedulable due to spark's bl

2020-07-15 Thread GitBox
venkata91 commented on pull request #28287: URL: https://github.com/apache/spark/pull/28287#issuecomment-658977388 > you can make a common function that has most of the code that gets called from 2 separate tests. one test passes with dynamic allocation on, the other with it off. that

[GitHub] [spark] SparkQA commented on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-15 Thread GitBox
SparkQA commented on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-658987615 **[Test build #125881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125881/testReport)** for PR 29120 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-15 Thread GitBox
SparkQA removed a comment on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-658857353 **[Test build #125881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125881/testReport)** for PR 29120 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29021: [WIP][SPARK-32201][SQL] More general skew join pattern matching

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29021: URL: https://github.com/apache/spark/pull/29021#issuecomment-658995807 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-659007597 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #29090: [SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

2020-07-15 Thread GitBox
SparkQA commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-659008171 **[Test build #125885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125885/testReport)** for PR 29090 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29111: URL: https://github.com/apache/spark/pull/29111#issuecomment-659007765 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-658910627 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-659007979 **[Test build #125905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125905/testReport)** for PR 28708 at commit

[GitHub] [spark] viirya commented on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
viirya commented on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-659008506 Jenkins seems not working one this. But GitHub Actions are passed. This is an automated message from the Apache

[GitHub] [spark] tgravescs commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
tgravescs commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455369019 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache

[GitHub] [spark] imback82 opened a new pull request #29127: [SPARK-32327][SQL] Introduce UnresolvedTableOrPermanentView for commands that support a table and permanent view, but not a temporary view

2020-07-15 Thread GitBox
imback82 opened a new pull request #29127: URL: https://github.com/apache/spark/pull/29127 ### What changes were proposed in this pull request? This PR proposes to introduce `UnresolvedTableOrPermanentView` for commands that support a table and a permanent view, but not a

[GitHub] [spark] GuoPhilipse commented on a change in pull request #29056: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs

2020-07-15 Thread GitBox
GuoPhilipse commented on a change in pull request #29056: URL: https://github.com/apache/spark/pull/29056#discussion_r455220561 ## File path: docs/sql-ref-syntax-qry-select-groupby.md ## @@ -38,6 +38,8 @@ GROUP BY GROUPING SETS (grouping_set [ , ...]) While aggregate

[GitHub] [spark] srowen commented on pull request #29112: [SPARK-32310][ML][PySpark] ML params default value parity part 1

2020-07-15 Thread GitBox
srowen commented on pull request #29112: URL: https://github.com/apache/spark/pull/29112#issuecomment-658908251 So in theory this shouldn't change behavior, or if it does, it's fixing an incompatibility that's likely more a bug than anything right?

[GitHub] [spark] cloud-fan commented on a change in pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-15 Thread GitBox
cloud-fan commented on a change in pull request #29032: URL: https://github.com/apache/spark/pull/29032#discussion_r455235801 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -912,7 +912,8 @@ private[spark] class TaskSchedulerImpl(

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #27366: URL: https://github.com/apache/spark/pull/27366#discussion_r455251187 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonFilters.scala ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29115: [SPARK-32315][ML] Provide an explanation error message when calling require

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29115: URL: https://github.com/apache/spark/pull/29115#issuecomment-658929884 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28287: [SPARK-31418][SCHEDULER] Request more executors in case of dynamic allocation is enabled and a task becomes unschedulable due t

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #28287: URL: https://github.com/apache/spark/pull/28287#issuecomment-658960595 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-658960503 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29124: [WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.12

2020-07-15 Thread GitBox
dongjoon-hyun edited a comment on pull request #29124: URL: https://github.com/apache/spark/pull/29124#issuecomment-658956530 The failure looks consistent. Could you take a look at that, @wangyum ? ``` [info] org.apache.spark.serializer.KryoSerializerSuite *** ABORTED *** (324

[GitHub] [spark] AmplabJenkins commented on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-658960503 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-658896338 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] Fokko edited a comment on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

2020-07-15 Thread GitBox
Fokko edited a comment on pull request #29121: URL: https://github.com/apache/spark/pull/29121#issuecomment-658967694 Good point @dongjoon-hyun, I was focusing on getting the CI green again. I've updated the PR description and title. While rereading it. Technically the title is

[GitHub] [spark] SparkQA removed a comment on pull request #29124: [WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.12

2020-07-15 Thread GitBox
SparkQA removed a comment on pull request #29124: URL: https://github.com/apache/spark/pull/29124#issuecomment-658857295 **[Test build #125879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125879/testReport)** for PR 29124 at commit

[GitHub] [spark] SparkQA commented on pull request #29124: [WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.12

2020-07-15 Thread GitBox
SparkQA commented on pull request #29124: URL: https://github.com/apache/spark/pull/29124#issuecomment-658988384 **[Test build #125879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125879/testReport)** for PR 29124 at commit

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455337945 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,29 @@ package object config { .booleanConf

[GitHub] [spark] SparkQA removed a comment on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-15 Thread GitBox
SparkQA removed a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-658874516 **[Test build #125896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125896/testReport)** for PR 28977 at commit

[GitHub] [spark] SparkQA commented on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-15 Thread GitBox
SparkQA commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-659006088 **[Test build #125896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125896/testReport)** for PR 28977 at commit

[GitHub] [spark] SparkQA commented on pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-15 Thread GitBox
SparkQA commented on pull request #29114: URL: https://github.com/apache/spark/pull/29114#issuecomment-659014189 **[Test build #125883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125883/testReport)** for PR 29114 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-15 Thread GitBox
SparkQA removed a comment on pull request #29114: URL: https://github.com/apache/spark/pull/29114#issuecomment-658857494 **[Test build #125883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125883/testReport)** for PR 29114 at commit

[GitHub] [spark] tgravescs commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
tgravescs commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455352852 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -44,9 +47,9 @@ import org.apache.spark.util.Utils //

[GitHub] [spark] tgravescs commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
tgravescs commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455388677 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,29 @@ package object config { .booleanConf

[GitHub] [spark] AmplabJenkins commented on pull request #28287: [SPARK-31418][SCHEDULER] Request more executors in case of dynamic allocation is enabled and a task becomes unschedulable due to spark'

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #28287: URL: https://github.com/apache/spark/pull/28287#issuecomment-659036401 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] karuppayya closed pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
karuppayya closed pull request #28804: URL: https://github.com/apache/spark/pull/28804 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] karuppayya commented on a change in pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
karuppayya commented on a change in pull request #28804: URL: https://github.com/apache/spark/pull/28804#discussion_r455403785 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2196,6 +2196,25 @@ object SQLConf {

[GitHub] [spark] karuppayya opened a new pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
karuppayya opened a new pull request #28804: URL: https://github.com/apache/spark/pull/28804 ### What changes were proposed in this pull request? In case of HashAggregation, a partial aggregation(update) is done followed by final aggregation(merge) During partial aggregation we

[GitHub] [spark] karuppayya commented on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-07-15 Thread GitBox
karuppayya commented on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-659049598 Updated the description with the benchmarks, after the latest changes. This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
SparkQA commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-659048554 **[Test build #125912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125912/testReport)** for PR 27366 at commit

[GitHub] [spark] cloud-fan opened a new pull request #29125: [SPARK-32018][SQL] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
cloud-fan opened a new pull request #29125: URL: https://github.com/apache/spark/pull/29125 partially backport https://github.com/apache/spark/pull/29026 This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] MaxGekk commented on a change in pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
MaxGekk commented on a change in pull request #27366: URL: https://github.com/apache/spark/pull/27366#discussion_r455208221 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonFilters.scala ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache

[GitHub] [spark] cloud-fan commented on pull request #29125: [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

2020-07-15 Thread GitBox
cloud-fan commented on pull request #29125: URL: https://github.com/apache/spark/pull/29125#issuecomment-658894290 cc @dongjoon-hyun @viirya This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] viirya commented on pull request #29112: [SPARK-32310][ML][PySpark] ML params default value parity part 1

2020-07-15 Thread GitBox
viirya commented on pull request #29112: URL: https://github.com/apache/spark/pull/29112#issuecomment-658922003 "classification, regression, clustering and fpm" instead of "part 1" in the title? This is an automated

[GitHub] [spark] cloud-fan commented on a change in pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-15 Thread GitBox
cloud-fan commented on a change in pull request #29015: URL: https://github.com/apache/spark/pull/29015#discussion_r455246869 ## File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala ## @@ -191,4 +191,14 @@ private[spark] object UI { .version("3.0.0")

[GitHub] [spark] cloud-fan commented on a change in pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-15 Thread GitBox
cloud-fan commented on a change in pull request #29015: URL: https://github.com/apache/spark/pull/29015#discussion_r455247627 ## File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala ## @@ -191,4 +191,14 @@ private[spark] object UI { .version("3.0.0")

[GitHub] [spark] aokolnychyi commented on pull request #29089: [SPARK-32276][SQL] Remove redundant sorts before repartition nodes

2020-07-15 Thread GitBox
aokolnychyi commented on pull request #29089: URL: https://github.com/apache/spark/pull/29089#issuecomment-658931940 Yes, my proposal is to optimize cases when we sort the data after the repartition like in the examples I gave above. In those cases, sorts below seem to be redundant.

[GitHub] [spark] holdenk commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
holdenk commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-658944693 All checks pass, I'm going to merge this to our current development branch. This is an automated message from

[GitHub] [spark] holdenk commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-07-15 Thread GitBox
holdenk commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-658954395 The SPIP has been voted on, this has been reviewed extensively, the original design is from 2017, I'm not waiting unless someone wishes to -1 for a valid technical reason.

[GitHub] [spark] frankyin-factual commented on pull request #28898: [SPARK-32059][SQL] Allow nested schema pruning thru window/sort/filter plans

2020-07-15 Thread GitBox
frankyin-factual commented on pull request #28898: URL: https://github.com/apache/spark/pull/28898#issuecomment-658954329 @dongjoon-hyun friendly bump This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] AmplabJenkins commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-658986674 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-658986674 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

2020-07-15 Thread GitBox
dongjoon-hyun edited a comment on pull request #29111: URL: https://github.com/apache/spark/pull/29111#issuecomment-658998506 Also, here. The green checkbox at the commit id. ![Screen Shot 2020-07-15 at 1 41 21

[GitHub] [spark] AmplabJenkins commented on pull request #29090: [SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-659010229 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29114: URL: https://github.com/apache/spark/pull/29114#issuecomment-659015242 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] AmplabJenkins commented on pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29114: URL: https://github.com/apache/spark/pull/29114#issuecomment-659015242 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29101: [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-659026999 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-15 Thread GitBox
AmplabJenkins commented on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-659026829 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-659026829 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29101: [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-659026999 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] SparkQA commented on pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-15 Thread GitBox
SparkQA commented on pull request #29015: URL: https://github.com/apache/spark/pull/29015#issuecomment-659027181 **[Test build #125911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125911/testReport)** for PR 29015 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-15 Thread GitBox
AmplabJenkins removed a comment on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-659026841 Test FAILed. Refer to this link for build results (access rights to CI server needed):

  1   2   3   4   5   6   7   8   9   >