[GitHub] [spark] HyukjinKwon commented on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2

2020-11-25 Thread GitBox
HyukjinKwon commented on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-734052269 Nice! This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gatorsmile commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

2020-11-25 Thread GitBox
gatorsmile commented on pull request #30494: URL: https://github.com/apache/spark/pull/30494#issuecomment-734053423 Thanks! Merged to master This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] gatorsmile closed pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

2020-11-25 Thread GitBox
gatorsmile closed pull request #30494: URL: https://github.com/apache/spark/pull/30494 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA removed a comment on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733922687 **[Test build #131808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131808/testReport)** for PR 30475 at commit

[GitHub] [spark] SparkQA commented on pull request #30312: [SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion

2020-11-25 Thread GitBox
SparkQA commented on pull request #30312: URL: https://github.com/apache/spark/pull/30312#issuecomment-733968379 **[Test build #131809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131809/testReport)** for PR 30312 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733972117 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #30506: [SPARK-33565][BUILD][PYTHON] remove python3.8 and fix breakage

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30506: URL: https://github.com/apache/spark/pull/30506#issuecomment-733966884 **[Test build #131812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131812/testReport)** for PR 30506 at commit

[GitHub] [spark] rdblue commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

2020-11-25 Thread GitBox
rdblue commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733979208 In general, I support the idea of moving away from tracking partition values using String. I'm not convinced that using a Literal would work much more easily because literal

[GitHub] [spark] dongjoon-hyun opened a new pull request #30508: Revert "[SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile"

2020-11-25 Thread GitBox
dongjoon-hyun opened a new pull request #30508: URL: https://github.com/apache/spark/pull/30508 ### What changes were proposed in this pull request? This reverts commit cb3fa6c9368e64184a5f7b19688181d11de9511c. ### Why are the changes needed? According to

[GitHub] [spark] SparkQA commented on pull request #30412: [SPARK-33480][SQL] Support char/varchar type

2020-11-25 Thread GitBox
SparkQA commented on pull request #30412: URL: https://github.com/apache/spark/pull/30412#issuecomment-733985079 **[Test build #131798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131798/testReport)** for PR 30412 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #30508: Revert "[SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile"

2020-11-25 Thread GitBox
dongjoon-hyun commented on pull request #30508: URL: https://github.com/apache/spark/pull/30508#issuecomment-733985043 cc @sunchao This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins commented on pull request #30412: [SPARK-33480][SQL] Support char/varchar type

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30412: URL: https://github.com/apache/spark/pull/30412#issuecomment-733985770 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun commented on pull request #30509: [SPARK-33565][PYTHON][BUILD][3.0] Remove py38 spark3

2020-11-25 Thread GitBox
dongjoon-hyun commented on pull request #30509: URL: https://github.com/apache/spark/pull/30509#issuecomment-733987709 Got it. No problem~ This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] SparkQA commented on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
SparkQA commented on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-734000115 **[Test build #131818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131818/testReport)** for PR 30403 at commit

[GitHub] [spark] sunchao commented on pull request #30508: Revert "[SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile"

2020-11-25 Thread GitBox
sunchao commented on pull request #30508: URL: https://github.com/apache/spark/pull/30508#issuecomment-734011469 Yes I'm fine for reverting this first while we searching for other solutions. Let's hope we can still ship this in Spark 3.1 release.

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30131: [SPARK-33220][CORE]Use `scheduleWithFixedDelay` to avoid repeated unnecessary scheduling for a short time

2020-11-25 Thread GitBox
AngersZh commented on a change in pull request #30131: URL: https://github.com/apache/spark/pull/30131#discussion_r530711869 ## File path: core/src/main/scala/org/apache/spark/Heartbeater.scala ## @@ -45,7 +45,8 @@ private[spark] class Heartbeater( val heartbeatTask =

[GitHub] [spark] dongjoon-hyun commented on pull request #30508: Revert "[SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile"

2020-11-25 Thread GitBox
dongjoon-hyun commented on pull request #30508: URL: https://github.com/apache/spark/pull/30508#issuecomment-734011862 Thank you, @HyukjinKwon and @sunchao . This is still testing to check the feasibility to revert~ This PR will wait until next Monday. :) BTW, I'll update the PR

[GitHub] [spark] HyukjinKwon opened a new pull request #30510: [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with Python 3.8 in GitHub Actions

2020-11-25 Thread GitBox
HyukjinKwon opened a new pull request #30510: URL: https://github.com/apache/spark/pull/30510 ### What changes were proposed in this pull request? This PR proposes to keep the test coverage with Python 3.8 in GitHub Actions. It is not tested for now in Jenkins due to an env issue.

[GitHub] [spark] SparkQA commented on pull request #30131: [SPARK-33220][CORE]Use `scheduleWithFixedDelay` to avoid repeated unnecessary scheduling for a short time

2020-11-25 Thread GitBox
SparkQA commented on pull request #30131: URL: https://github.com/apache/spark/pull/30131#issuecomment-734016301 **[Test build #131820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131820/testReport)** for PR 30131 at commit

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30501: [SPARK-33563][PYTHON][R][SQL] Expose inverse hyperbolic trig functions in PySpark and SparkR

2020-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #30501: URL: https://github.com/apache/spark/pull/30501#discussion_r530719331 ## File path: R/pkg/R/functions.R ## @@ -455,6 +455,19 @@ setMethod("acos", column(jc) }) +#' @details Review comment:

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30501: [SPARK-33563][PYTHON][R][SQL] Expose inverse hyperbolic trig functions in PySpark and SparkR

2020-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #30501: URL: https://github.com/apache/spark/pull/30501#discussion_r530719552 ## File path: python/pyspark/sql/functions.py ## @@ -220,6 +220,19 @@ def acos(col): return _invoke_function_over_column("acos", col) +def

[GitHub] [spark] HyukjinKwon commented on pull request #30501: [SPARK-33563][PYTHON][R][SQL] Expose inverse hyperbolic trig functions in PySpark and SparkR

2020-11-25 Thread GitBox
HyukjinKwon commented on pull request #30501: URL: https://github.com/apache/spark/pull/30501#issuecomment-734018562 Looks fine otherwise. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] rdblue commented on pull request #30452: [SPARK-33509][SQL] List partition by names from a V2 table which supports partition management

2020-11-25 Thread GitBox
rdblue commented on pull request #30452: URL: https://github.com/apache/spark/pull/30452#issuecomment-734021147 > The partition can be a transform like year(ts_col), shall we just partition index in the API instead? If I remember correctly, there should be a schema exposed by the

[GitHub] [spark] stczwd commented on pull request #30452: [SPARK-33509][SQL] List partition by names from a V2 table which supports partition management

2020-11-25 Thread GitBox
stczwd commented on pull request #30452: URL: https://github.com/apache/spark/pull/30452#issuecomment-734020787 > Since `SupportsPartitionManagement` already have the API `partitionSchema`, which means that the implementations will pick a name for partition transforms, I think it's OK to

[GitHub] [spark] SparkQA commented on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
SparkQA commented on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-734025753 **[Test build #131818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131818/testReport)** for PR 30403 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-734000115 **[Test build #131818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131818/testReport)** for PR 30403 at commit

[GitHub] [spark] JQ-Cao commented on pull request #30495: [SPARK-33548][WEBUI] display the jvm peak memory usage on the executor ui

2020-11-25 Thread GitBox
JQ-Cao commented on pull request #30495: URL: https://github.com/apache/spark/pull/30495#issuecomment-734033935 > @JQ-Cao These metrics are already on the executor page. They will show up after select the additional metrics checkbox: >

[GitHub] [spark] JQ-Cao closed pull request #30495: [SPARK-33548][WEBUI] display the jvm peak memory usage on the executor ui

2020-11-25 Thread GitBox
JQ-Cao closed pull request #30495: URL: https://github.com/apache/spark/pull/30495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] HyukjinKwon edited a comment on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively

2020-11-25 Thread GitBox
HyukjinKwon edited a comment on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-734037334 It will be exactly same as `spark.files` and `spark.yarn.dist.files`. To be honest, I am not exactly sure how they will conflict to each other but both work together

[GitHub] [spark] HyukjinKwon commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively

2020-11-25 Thread GitBox
HyukjinKwon commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-734037334 It will be exactly same as `spark.files` and `spark.yarn.dist.files`. To be honest, I am not exactly sure how they will conflictto each other but both work together as far

[GitHub] [spark] SparkQA commented on pull request #29490: [SPARK-32668][SQL] HiveGenericUDTF initialize UDTF should use StructObjectInspector method

2020-11-25 Thread GitBox
SparkQA commented on pull request #29490: URL: https://github.com/apache/spark/pull/29490#issuecomment-734041140 **[Test build #131822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131822/testReport)** for PR 29490 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29490: [SPARK-32668][SQL] HiveGenericUDTF initialize UDTF should use StructObjectInspector method

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #29490: URL: https://github.com/apache/spark/pull/29490#issuecomment-734020610 **[Test build #131822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131822/testReport)** for PR 29490 at commit

[GitHub] [spark] cloud-fan commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

2020-11-25 Thread GitBox
cloud-fan commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-73404 Let's do it step by step, and support typed literal first. We can figure out how to eliminate the string <-> actual value roundtrip in v2 commands later. Let's make

[GitHub] [spark] SparkQA commented on pull request #30131: [SPARK-33220][CORE]Use `scheduleWithFixedDelay` to avoid repeated unnecessary scheduling for a short time

2020-11-25 Thread GitBox
SparkQA commented on pull request #30131: URL: https://github.com/apache/spark/pull/30131#issuecomment-734051712 **[Test build #131820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131820/testReport)** for PR 30131 at commit

[GitHub] [spark] SparkQA commented on pull request #30510: [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with Python 3.8 in GitHub Actions

2020-11-25 Thread GitBox
SparkQA commented on pull request #30510: URL: https://github.com/apache/spark/pull/30510#issuecomment-734052772 **[Test build #131821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131821/testReport)** for PR 30510 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #30511: [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8 in GitHub Actions

2020-11-25 Thread GitBox
HyukjinKwon commented on pull request #30511: URL: https://github.com/apache/spark/pull/30511#issuecomment-734052805 Merged to branch-3.0. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HyukjinKwon closed pull request #30511: [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8 in GitHub Actions

2020-11-25 Thread GitBox
HyukjinKwon closed pull request #30511: URL: https://github.com/apache/spark/pull/30511 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA removed a comment on pull request #30510: [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with Python 3.8 in GitHub Actions

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30510: URL: https://github.com/apache/spark/pull/30510#issuecomment-734017295 **[Test build #131821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131821/testReport)** for PR 30510 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
cloud-fan commented on a change in pull request #30403: URL: https://github.com/apache/spark/pull/30403#discussion_r530756887 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTableExec.scala ## @@ -27,7 +26,6 @@ import

[GitHub] [spark] SparkQA removed a comment on pull request #30398: [SPARK-33452][SQL] Support v2 SHOW PARTITIONS

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30398: URL: https://github.com/apache/spark/pull/30398#issuecomment-733763807 **[Test build #131783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131783/testReport)** for PR 30398 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733899176 **[Test build #131802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131802/testReport)** for PR 30475 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29893: [SPARK-32976][SQL]Support column list in INSERT statement

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #29893: URL: https://github.com/apache/spark/pull/29893#issuecomment-733868465 **[Test build #131796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131796/testReport)** for PR 29893 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733873132 **[Test build #131799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131799/testReport)** for PR 30403 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #30504: [SPARK-33544][SQL] Optimizer should not insert filter when when explode with CreateArray/CreateMap

2020-11-25 Thread GitBox
cloud-fan commented on a change in pull request #30504: URL: https://github.com/apache/spark/pull/30504#discussion_r530602009 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -873,24 +873,30 @@ object

[GitHub] [spark] cloud-fan commented on a change in pull request #30504: [SPARK-33544][SQL] Optimizer should not insert filter when when explode with CreateArray/CreateMap

2020-11-25 Thread GitBox
cloud-fan commented on a change in pull request #30504: URL: https://github.com/apache/spark/pull/30504#discussion_r530603409 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -873,24 +873,30 @@ object

[GitHub] [spark] Victsm commented on a change in pull request #30312: [SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion

2020-11-25 Thread GitBox
Victsm commented on a change in pull request #30312: URL: https://github.com/apache/spark/pull/30312#discussion_r530605991 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1992,4 +1992,32 @@ package object config {

[GitHub] [spark] imback82 commented on a change in pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
imback82 commented on a change in pull request #30403: URL: https://github.com/apache/spark/pull/30403#discussion_r530606256 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala ## @@ -421,7 +421,7 @@ class CachedTableSuite extends QueryTest

[GitHub] [spark] SparkQA commented on pull request #30500: [SPARK-33562][UI] Improve the style of the checkbox in executor page

2020-11-25 Thread GitBox
SparkQA commented on pull request #30500: URL: https://github.com/apache/spark/pull/30500#issuecomment-733913452 **[Test build #131788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131788/testReport)** for PR 30500 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30500: [SPARK-33562][UI] Improve the style of the checkbox in executor page

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30500: URL: https://github.com/apache/spark/pull/30500#issuecomment-733835284 **[Test build #131788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131788/testReport)** for PR 30500 at commit

[GitHub] [spark] Victsm commented on a change in pull request #30312: [SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion

2020-11-25 Thread GitBox
Victsm commented on a change in pull request #30312: URL: https://github.com/apache/spark/pull/30312#discussion_r530605991 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1992,4 +1992,32 @@ package object config {

[GitHub] [spark] cloud-fan commented on a change in pull request #30504: [SPARK-33544][SQL] Optimizer should not insert filter when explode with CreateArray/CreateMap

2020-11-25 Thread GitBox
cloud-fan commented on a change in pull request #30504: URL: https://github.com/apache/spark/pull/30504#discussion_r530606734 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -873,24 +873,30 @@ object

[GitHub] [spark] imback82 commented on a change in pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
imback82 commented on a change in pull request #30403: URL: https://github.com/apache/spark/pull/30403#discussion_r530606256 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala ## @@ -421,7 +421,7 @@ class CachedTableSuite extends QueryTest

[GitHub] [spark] imback82 commented on a change in pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
imback82 commented on a change in pull request #30403: URL: https://github.com/apache/spark/pull/30403#discussion_r530571035 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala ## @@ -421,7 +421,7 @@ class CachedTableSuite extends QueryTest

[GitHub] [spark] cloud-fan commented on a change in pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

2020-11-25 Thread GitBox
cloud-fan commented on a change in pull request #30403: URL: https://github.com/apache/spark/pull/30403#discussion_r530608911 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala ## @@ -421,7 +421,7 @@ class CachedTableSuite extends QueryTest

[GitHub] [spark] Victsm removed a comment on pull request #30480: [SPARK-32921][SHUFFLE][test-maven][test-hadoop2.7] MapOutputTracker extensions to support push-based shuffle

2020-11-25 Thread GitBox
Victsm removed a comment on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-733866872 The javadoc issues leading to the build failures are not related to files changed in this patch. ``` [error]

[GitHub] [spark] SparkQA commented on pull request #30430: [SPARK-33503][SQL] Refactor SortOrder class to allow multiple childrens

2020-11-25 Thread GitBox
SparkQA commented on pull request #30430: URL: https://github.com/apache/spark/pull/30430#issuecomment-733916550 **[Test build #131785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131785/testReport)** for PR 30430 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30430: [SPARK-33503][SQL] Refactor SortOrder class to allow multiple childrens

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30430: URL: https://github.com/apache/spark/pull/30430#issuecomment-733801134 **[Test build #131785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131785/testReport)** for PR 30430 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733924096 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

2020-11-25 Thread GitBox
SparkQA commented on pull request #30494: URL: https://github.com/apache/spark/pull/30494#issuecomment-733927636 **[Test build #131810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131810/testReport)** for PR 30494 at commit

[GitHub] [spark] viirya commented on a change in pull request #30504: [SPARK-33544][SQL] Optimizer should not insert filter when explode with CreateArray/CreateMap

2020-11-25 Thread GitBox
viirya commented on a change in pull request #30504: URL: https://github.com/apache/spark/pull/30504#discussion_r530608975 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ## @@ -30,6 +30,13 @@ import

[GitHub] [spark] AmplabJenkins commented on pull request #30488: [SPARK-33071][SPARK-33536][SQL] Avoid changing dataset_id of LogicalPlan in join() to not break DetectAmbiguousSelfJoin

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30488: URL: https://github.com/apache/spark/pull/30488#issuecomment-733536612 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30470: [SPARK-33495][BUILD] Remove commons-logging.jar's dependency

2020-11-25 Thread GitBox
SparkQA commented on pull request #30470: URL: https://github.com/apache/spark/pull/30470#issuecomment-733536564 **[Test build #131737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131737/testReport)** for PR 30470 at commit

[GitHub] [spark] AngersZhuuuu commented on pull request #30496: [SPARK-33547][SQL] Add usage of typed literal in doc

2020-11-25 Thread GitBox
AngersZh commented on pull request #30496: URL: https://github.com/apache/spark/pull/30496#issuecomment-733536446 FYI @maropu there is many duplicated example between typed literal and corresponding literal, any suggestion?

[GitHub] [spark] AmplabJenkins commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733536613 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733536615 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733536614 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30472: [WIP][SPARK-32221] Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-11-25 Thread GitBox
SparkQA commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-733543655 **[Test build #131757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131757/testReport)** for PR 30472 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-733544301 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2020-11-25 Thread GitBox
SparkQA commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-733544254 **[Test build #131754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131754/testReport)** for PR 30483 at commit

[GitHub] [spark] SparkQA commented on pull request #28026: [SPARK-31257][SQL] Unify create table syntax

2020-11-25 Thread GitBox
SparkQA commented on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-733547825 **[Test build #131736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131736/testReport)** for PR 28026 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-733543655 **[Test build #131757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131757/testReport)** for PR 30472 at commit

[GitHub] [spark] SparkQA commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-11-25 Thread GitBox
SparkQA commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-733550292 **[Test build #131757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131757/testReport)** for PR 30472 at commit

[GitHub] [spark] SparkQA commented on pull request #30480: [SPARK-32921][SHUFFLE][test-maven][test-hadoop2.7] MapOutputTracker extensions to support push-based shuffle

2020-11-25 Thread GitBox
SparkQA commented on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-733551692 **[Test build #131710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131710/testReport)** for PR 30480 at commit

[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views

2020-11-25 Thread GitBox
luluorta commented on a change in pull request #30289: URL: https://github.com/apache/spark/pull/30289#discussion_r530134316 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -361,11 +379,38 @@ object ViewHelper { } } +

[GitHub] [spark] wangyum commented on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2020-11-25 Thread GitBox
wangyum commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-733556681 @LuciferYang It would be great if we had some benchmark numbers. This is an automated message from the Apache

[GitHub] [spark] aminh73 commented on pull request #27380: [SPARK-30669][SS] Introduce AdmissionControl APIs for StructuredStreaming

2020-11-25 Thread GitBox
aminh73 commented on pull request #27380: URL: https://github.com/apache/spark/pull/27380#issuecomment-733557893 We need to use `maxOffsetsPerTrigger` in the Kafka source with `Trigger.Once()` but it seems reads `allAvailable` in spark 3. Is there a way for achieving rate limit in this

[GitHub] [spark] Ngone51 commented on a change in pull request #30312: [SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion

2020-11-25 Thread GitBox
Ngone51 commented on a change in pull request #30312: URL: https://github.com/apache/spark/pull/30312#discussion_r530141471 ## File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala ## @@ -0,0 +1,462 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] zero323 commented on pull request #30382: [SPARK-33457][PYTHON] Adjust mypy configuration

2020-11-25 Thread GitBox
zero323 commented on pull request #30382: URL: https://github.com/apache/spark/pull/30382#issuecomment-733566086 Thanks @Fokko and @HyukjinKwon! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-11-25 Thread GitBox
SparkQA commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-733568741 **[Test build #131761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131761/testReport)** for PR 30472 at commit

[GitHub] [spark] SparkQA commented on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2020-11-25 Thread GitBox
SparkQA commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-733568636 **[Test build #131760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131760/testReport)** for PR 30483 at commit

[GitHub] [spark] SparkQA commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively

2020-11-25 Thread GitBox
SparkQA commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733568538 **[Test build #131759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131759/testReport)** for PR 30486 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30493: [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30493: URL: https://github.com/apache/spark/pull/30493#issuecomment-733568871 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30484: URL: https://github.com/apache/spark/pull/30484#issuecomment-733572799 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-11-25 Thread GitBox
SparkQA commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-733577201 **[Test build #131761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131761/testReport)** for PR 30472 at commit

[GitHub] [spark] SparkQA commented on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast

2020-11-25 Thread GitBox
SparkQA commented on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733576716 **[Test build #131764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131764/testReport)** for PR 30440 at commit

[GitHub] [spark] SparkQA commented on pull request #30212: [SPARK-33308][SQL] Refract current grouping analytics

2020-11-25 Thread GitBox
SparkQA commented on pull request #30212: URL: https://github.com/apache/spark/pull/30212#issuecomment-733576983 **[Test build #131730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131730/testReport)** for PR 30212 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30212: [SPARK-33308][SQL] Refract current grouping analytics

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30212: URL: https://github.com/apache/spark/pull/30212#issuecomment-733578781 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] kiszk commented on pull request #30243: [SPARK-33335][SQL] Support `array_contains_array` func

2020-11-25 Thread GitBox
kiszk commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-733582819 LGTM from code generation perstpective @maropu @cloud-fan @HyukjinKwon @ueshin Any other comments? In particular, regarding the specification of the function

[GitHub] [spark] AngersZhuuuu opened a new pull request #30496: [SPARK-33547][SQL] Add usage of typed literal in doc

2020-11-25 Thread GitBox
AngersZh opened a new pull request #30496: URL: https://github.com/apache/spark/pull/30496 ### What changes were proposed in this pull request? According to https://github.com/apache/spark/pull/30421#discussion_r530024114 Add typed literal in doc ### Why are the

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30496: [SPARK-33547][SQL] Add usage of typed literal in doc

2020-11-25 Thread GitBox
AngersZh commented on a change in pull request #30496: URL: https://github.com/apache/spark/pull/30496#discussion_r530172194 ## File path: docs/sql-ref-literals.md ## @@ -21,14 +21,74 @@ license: | A literal (also known as a constant) represents a fixed data value.

[GitHub] [spark] SparkQA removed a comment on pull request #30470: [SPARK-33495][BUILD] Remove commons-logging.jar's dependency

2020-11-25 Thread GitBox
SparkQA removed a comment on pull request #30470: URL: https://github.com/apache/spark/pull/30470#issuecomment-733468984 **[Test build #131737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131737/testReport)** for PR 30470 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value

2020-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733536615 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission

2020-11-25 Thread GitBox
SparkQA commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733536865 **[Test build #131753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131753/testReport)** for PR 30492 at commit

[GitHub] [spark] SparkQA commented on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast

2020-11-25 Thread GitBox
SparkQA commented on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733537202 **[Test build #131756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131756/testReport)** for PR 30440 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2

2020-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733513051 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast

2020-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733510495 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2

2020-11-25 Thread GitBox
SparkQA commented on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733537096 **[Test build #131755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131755/testReport)** for PR 30478 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30470: [SPARK-33495][BUILD] Remove commons-logging.jar's dependency

2020-11-25 Thread GitBox
AmplabJenkins commented on pull request #30470: URL: https://github.com/apache/spark/pull/30470#issuecomment-733537538 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30496: [SPARK-33547][SQL] Add usage of typed literal in doc

2020-11-25 Thread GitBox
SparkQA commented on pull request #30496: URL: https://github.com/apache/spark/pull/30496#issuecomment-733537090 **[Test build #131752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131752/testReport)** for PR 30496 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30488: [SPARK-33071][SPARK-33536][SQL] Avoid changing dataset_id of LogicalPlan in join() to not break DetectAmbiguousSelfJoin

2020-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #30488: URL: https://github.com/apache/spark/pull/30488#issuecomment-733536612 This is an automated message from the Apache Git Service. To respond to the message, please log on

<    2   3   4   5   6   7   8   9   >