[GitHub] [spark] SparkQA commented on pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries for better reuse

2021-06-29 Thread GitBox
SparkQA commented on pull request #32298: URL: https://github.com/apache/spark/pull/32298#issuecomment-870826390 **[Test build #140386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140386/testReport)** for PR 32298 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-870826301 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44896/ --

[GitHub] [spark] SparkQA commented on pull request #32286: [SPARK-35181][CORE] Use zstd for spark.io.compression.codec by default

2021-06-29 Thread GitBox
SparkQA commented on pull request #32286: URL: https://github.com/apache/spark/pull/32286#issuecomment-870826313 **[Test build #140387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140387/testReport)** for PR 32286 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33106: [SPARK-35876][SQL] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33106: URL: https://github.com/apache/spark/pull/33106#issuecomment-870826056 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44893/ --

[GitHub] [spark] SparkQA commented on pull request #32552: [SPARK-34819][SQL] MapType supports comparable semantics

2021-06-29 Thread GitBox
SparkQA commented on pull request #32552: URL: https://github.com/apache/spark/pull/32552#issuecomment-870825644 **[Test build #140384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140384/testReport)** for PR 32552 at commit

[GitHub] [spark] SparkQA commented on pull request #32365: [SPARK-35228][SQL] Add expression ToHiveString for keep consistent between hive/spark format in df.show

2021-06-29 Thread GitBox
SparkQA commented on pull request #32365: URL: https://github.com/apache/spark/pull/32365#issuecomment-870825574 **[Test build #140385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140385/testReport)** for PR 32365 at commit

[GitHub] [spark] SparkQA commented on pull request #32753: [SPARK-34859][SQL] Handle column index when using vectorized Parquet reader

2021-06-29 Thread GitBox
SparkQA commented on pull request #32753: URL: https://github.com/apache/spark/pull/32753#issuecomment-870825440 **[Test build #140382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140382/testReport)** for PR 32753 at commit

[GitHub] [spark] SparkQA commented on pull request #32610: [SPARK-35460][K8S] verify the content of`spark.kubernetes.executor.podNamePrefix` before post it to k8s api-server

2021-06-29 Thread GitBox
SparkQA commented on pull request #32610: URL: https://github.com/apache/spark/pull/32610#issuecomment-870825506 **[Test build #140383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140383/testReport)** for PR 32610 at commit

[GitHub] [spark] SparkQA commented on pull request #32793: [WIP][SPARK-35430] Switch on "PVs with local storage" integration test on Docker driver

2021-06-29 Thread GitBox
SparkQA commented on pull request #32793: URL: https://github.com/apache/spark/pull/32793#issuecomment-870825358 **[Test build #140380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140380/testReport)** for PR 32793 at commit

[GitHub] [spark] SparkQA commented on pull request #32787: [SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans

2021-06-29 Thread GitBox
SparkQA commented on pull request #32787: URL: https://github.com/apache/spark/pull/32787#issuecomment-870825377 **[Test build #140381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140381/testReport)** for PR 32787 at commit

[GitHub] [spark] SparkQA commented on pull request #32832: [SPARK-35686][SQL] Not allow using auto-generated alias when creating view

2021-06-29 Thread GitBox
SparkQA commented on pull request #32832: URL: https://github.com/apache/spark/pull/32832#issuecomment-870825342 **[Test build #140379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140379/testReport)** for PR 32832 at commit

[GitHub] [spark] SparkQA commented on pull request #32875: [WIP][SPARK-35703] Remove HashClusteredDistribution and relax constraint for bucket join

2021-06-29 Thread GitBox
SparkQA commented on pull request #32875: URL: https://github.com/apache/spark/pull/32875#issuecomment-870825234 **[Test build #140377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140377/testReport)** for PR 32875 at commit

[GitHub] [spark] SparkQA commented on pull request #32902: [SPARK-35754][CORE] Add config to put migrating blocks on disk only

2021-06-29 Thread GitBox
SparkQA commented on pull request #32902: URL: https://github.com/apache/spark/pull/32902#issuecomment-870825203 **[Test build #140375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140375/testReport)** for PR 32902 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the sta

2021-06-29 Thread GitBox
AmplabJenkins removed a comment on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-868924160 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] SparkQA commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE

2021-06-29 Thread GitBox
SparkQA commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-870825238 **[Test build #140378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140378/testReport)** for PR 32850 at commit

[GitHub] [spark] SparkQA commented on pull request #32928: [SPARK-35784][SS] Implementation for RocksDB instance

2021-06-29 Thread GitBox
SparkQA commented on pull request #32928: URL: https://github.com/apache/spark/pull/32928#issuecomment-870825069 **[Test build #140374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140374/testReport)** for PR 32928 at commit

[GitHub] [spark] SparkQA commented on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions

2021-06-29 Thread GitBox
SparkQA commented on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-870825166 **[Test build #140376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140376/testReport)** for PR 32883 at commit

[GitHub] [spark] SparkQA commented on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better

2021-06-29 Thread GitBox
SparkQA commented on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-870825128 **[Test build #140368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140368/testReport)** for PR 33078 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33115: [SPARK-35916][SQL] Support subtraction among Date/Timestamp/TimestampWithoutTZ

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33115: URL: https://github.com/apache/spark/pull/33115#issuecomment-870824971 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44890/ --

[GitHub] [spark] SparkQA commented on pull request #32934: [WIP][SPARK-35788][SS] Metrics support for RocksDB instance

2021-06-29 Thread GitBox
SparkQA commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-870824970 **[Test build #140372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140372/testReport)** for PR 32934 at commit

[GitHub] [spark] SparkQA commented on pull request #32933: [WIP][SPARK-35785][SS] Cleanup support for RocksDB instance

2021-06-29 Thread GitBox
SparkQA commented on pull request #32933: URL: https://github.com/apache/spark/pull/32933#issuecomment-870824903 **[Test build #140373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140373/testReport)** for PR 32933 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33117: [SPARK-35859][PYTHON] Cleanup type hints in pandas-on-Spark

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33117: URL: https://github.com/apache/spark/pull/33117#issuecomment-870824861 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44888/ --

[GitHub] [spark] SparkQA commented on pull request #32943: [SPARK-35735][SQL] Take into account day-time interval fields in cast

2021-06-29 Thread GitBox
SparkQA commented on pull request #32943: URL: https://github.com/apache/spark/pull/32943#issuecomment-870824868 **[Test build #140371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140371/testReport)** for PR 32943 at commit

[GitHub] [spark] SparkQA commented on pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

2021-06-29 Thread GitBox
SparkQA commented on pull request #32980: URL: https://github.com/apache/spark/pull/32980#issuecomment-870824788 **[Test build #140369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140369/testReport)** for PR 32980 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33119: [SPARK-35920][BUILD] Upgrade to Chill 0.10.0

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33119: URL: https://github.com/apache/spark/pull/33119#issuecomment-870824723 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44887/ --

[GitHub] [spark] SparkQA commented on pull request #32958: [SPARK-35065][SQL] Group exception messages in spark/sql (core)

2021-06-29 Thread GitBox
SparkQA commented on pull request #32958: URL: https://github.com/apache/spark/pull/32958#issuecomment-870824743 **[Test build #140370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140370/testReport)** for PR 32958 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33101: [SPARK-35907][CORE] Instead of File#mkdirs, Files#createDirectories is expected.

2021-06-29 Thread GitBox
AmplabJenkins removed a comment on pull request #33101: URL: https://github.com/apache/spark/pull/33101#issuecomment-869080424 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33114: [SPARK-35913][SQL] Create hive permanent function with owner name

2021-06-29 Thread GitBox
AmplabJenkins removed a comment on pull request #33114: URL: https://github.com/apache/spark/pull/33114#issuecomment-870196000 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
AmplabJenkins removed a comment on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-870824450 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140365/

[GitHub] [spark] SparkQA commented on pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress

2021-06-29 Thread GitBox
SparkQA commented on pull request #33091: URL: https://github.com/apache/spark/pull/33091#issuecomment-870824612 **[Test build #140367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140367/testReport)** for PR 33091 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33120: [SPARK-35899][SQL][FOLLOWUP] Utility to convert connector expressions to Catalyst

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33120: URL: https://github.com/apache/spark/pull/33120#issuecomment-870824681 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44886/ --

[GitHub] [spark] SparkQA commented on pull request #33095: [SPARK-35339][PYTHON] Improve unit tests for data-type-based basic operations

2021-06-29 Thread GitBox
SparkQA commented on pull request #33095: URL: https://github.com/apache/spark/pull/33095#issuecomment-870824506 **[Test build #140366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140366/testReport)** for PR 33095 at commit

[GitHub] [spark] SparkQA commented on pull request #33114: [SPARK-35913][SQL] Create hive permanent function with owner name

2021-06-29 Thread GitBox
SparkQA commented on pull request #33114: URL: https://github.com/apache/spark/pull/33114#issuecomment-870824396 **[Test build #140362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140362/testReport)** for PR 33114 at commit

[GitHub] [spark] SparkQA commented on pull request #33101: [SPARK-35907][CORE] Instead of File#mkdirs, Files#createDirectories is expected.

2021-06-29 Thread GitBox
SparkQA commented on pull request #33101: URL: https://github.com/apache/spark/pull/33101#issuecomment-870824488 **[Test build #140364 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140364/testReport)** for PR 33101 at commit

[GitHub] [spark] viirya commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests

2021-06-29 Thread GitBox
viirya commented on pull request #32867: URL: https://github.com/apache/spark/pull/32867#issuecomment-870824460 Ur, this might be what I said before, how we make sure the tests are run... @Yikun Can you check it again? -- This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-870824450 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140365/ -- This

[GitHub] [spark] SparkQA commented on pull request #33105: [SPARK-35908][SQL] Remove repartition if the child maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
SparkQA commented on pull request #33105: URL: https://github.com/apache/spark/pull/33105#issuecomment-870824402 **[Test build #140363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140363/testReport)** for PR 33105 at commit

[GitHub] [spark] SparkQA commented on pull request #33116: [SPARK-35259][SHUFFLE] Rename ExternalBlockHandler Timer metrics without incorrect millis suffix

2021-06-29 Thread GitBox
SparkQA commented on pull request #33116: URL: https://github.com/apache/spark/pull/33116#issuecomment-870824311 **[Test build #140361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140361/testReport)** for PR 33116 at commit

[GitHub] [spark] SparkQA commented on pull request #33121: [SPARK-35921][BUILD] ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT

2021-06-29 Thread GitBox
SparkQA commented on pull request #33121: URL: https://github.com/apache/spark/pull/33121#issuecomment-870824270 **[Test build #140360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140360/testReport)** for PR 33121 at commit

[GitHub] [spark] SparkQA commented on pull request #33136: [SPARK-35932][SQL] Support extracting hour/minute/second from timestamp without time zone

2021-06-29 Thread GitBox
SparkQA commented on pull request #33136: URL: https://github.com/apache/spark/pull/33136#issuecomment-870824155 **[Test build #140356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140356/testReport)** for PR 33136 at commit

[GitHub] [spark] SparkQA commented on pull request #33133: [SPARK-35930][BUILD] Upgrade kinesis-client to 1.14.4

2021-06-29 Thread GitBox
SparkQA commented on pull request #33133: URL: https://github.com/apache/spark/pull/33133#issuecomment-870824122 **[Test build #140357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140357/testReport)** for PR 33133 at commit

[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

2021-06-29 Thread GitBox
SparkQA commented on pull request #33137: URL: https://github.com/apache/spark/pull/33137#issuecomment-870824055 **[Test build #140355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140355/testReport)** for PR 33137 at commit

[GitHub] [spark] SparkQA commented on pull request #33128: [SPARK-35873][PYTHON] Cleanup the version logic from the pandas API on Spark

2021-06-29 Thread GitBox
SparkQA commented on pull request #33128: URL: https://github.com/apache/spark/pull/33128#issuecomment-870824118 **[Test build #140358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140358/testReport)** for PR 33128 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33126: [SPARK-35924][BUILD][TESTS] Add Java 17 ea build test to GitHub action

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33126: URL: https://github.com/apache/spark/pull/33126#issuecomment-870824143 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140359/ -- This

[GitHub] [spark] SparkQA commented on pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode

2021-06-29 Thread GitBox
SparkQA commented on pull request #33138: URL: https://github.com/apache/spark/pull/33138#issuecomment-870823920 **[Test build #140354 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140354/testReport)** for PR 33138 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #30245: [SPARK-33337][SQL] Support subexpression elimination in branches of conditional expressions

2021-06-29 Thread GitBox
cloud-fan commented on a change in pull request #30245: URL: https://github.com/apache/spark/pull/30245#discussion_r660870034 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ## @@ -65,11 +65,82 @@ class

[GitHub] [spark] dongjoon-hyun commented on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-870816136 Maven build also looks good to me. ``` $ build/mvn package -pl common/unsafe --am ... [INFO] Spark Project Parent POM ... SUCCESS [

[GitHub] [spark] dongjoon-hyun closed pull request #33126: [SPARK-35924][BUILD][TESTS] Add Java 17 ea build test to GitHub action

2021-06-29 Thread GitBox
dongjoon-hyun closed pull request #33126: URL: https://github.com/apache/spark/pull/33126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] ueshin edited a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests

2021-06-29 Thread GitBox
ueshin edited a comment on pull request #32867: URL: https://github.com/apache/spark/pull/32867#issuecomment-870809700 I'm sorry the the late review, but is this triggering unit tests? I don't see any test cases running. - https://github.com/apache/spark/runs/2940354315 ```

[GitHub] [spark] dongjoon-hyun commented on pull request #33139: [SPARK-35938][PYTHON] Add deprecation warning for Python 3.6

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33139: URL: https://github.com/apache/spark/pull/33139#issuecomment-870812081 Thank you for working on this, @xinrong-databricks . cc @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] viirya commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre and Hadoop to 3.2.1

2021-06-29 Thread GitBox
viirya commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-870811211 try this again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] viirya opened a new pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre and Hadoop to 3.2.1

2021-06-29 Thread GitBox
viirya opened a new pull request #29326: URL: https://github.com/apache/spark/pull/29326 ### What changes were proposed in this pull request? This PR upgrades Guava to newer 27.0-jre and the dependency version of Hadoop 3.2 line to 3.2.1. ### Why are the changes

[GitHub] [spark] ueshin commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests

2021-06-29 Thread GitBox
ueshin commented on pull request #32867: URL: https://github.com/apache/spark/pull/32867#issuecomment-870809700 Is this triggering unit tests? I don't see any test cases running. https://github.com/apache/spark/runs/2940354315 ```

[GitHub] [spark] viirya commented on a change in pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32980: URL: https://github.com/apache/spark/pull/32980#discussion_r660853410 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -1049,17 +1103,25 @@ class

[GitHub] [spark] xinrong-databricks opened a new pull request #33139: [SPARK-35936][PYTHON] Add deprecation warning for Python 3.6

2021-06-29 Thread GitBox
xinrong-databricks opened a new pull request #33139: URL: https://github.com/apache/spark/pull/33139 ### What changes were proposed in this pull request? Add deprecation warning for Python 3.6. ### Why are the changes needed? According to

[GitHub] [spark] otterc commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

2021-06-29 Thread GitBox
otterc commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r660843950 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -567,7 +598,8 @@ public void

[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-870806007 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun closed pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
dongjoon-hyun closed pull request #33100: URL: https://github.com/apache/spark/pull/33100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-870804816 Thank you, @wangyum , @HyukjinKwon , @maropu , @huaxingao . For the above comment (https://github.com/apache/spark/pull/33100#discussion_r659455547), we can discuss

[GitHub] [spark] viirya commented on a change in pull request #32928: [SPARK-35784][SS] Implementation for RocksDB instance

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32928: URL: https://github.com/apache/spark/pull/32928#discussion_r660848165 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala ## @@ -0,0 +1,451 @@ +/* + * Licensed to the Apache

[GitHub] [spark] SaurabhChawla100 commented on pull request #32972: [SPARK-35756][SQL] unionByName supports struct having same col names but different sequence

2021-06-29 Thread GitBox
SaurabhChawla100 commented on pull request #32972: URL: https://github.com/apache/spark/pull/32972#issuecomment-870803046 > #33040 was merged so this needs to be reworked based off that now Thanks for sharing the details. Will update the PR as per the new change done -- This is an

[GitHub] [spark] otterc commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

2021-06-29 Thread GitBox
otterc commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r660843950 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -567,7 +598,8 @@ public void

[GitHub] [spark] ueshin closed pull request #33117: [SPARK-35859][PYTHON] Cleanup type hints in pandas-on-Spark

2021-06-29 Thread GitBox
ueshin closed pull request #33117: URL: https://github.com/apache/spark/pull/33117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] ueshin commented on pull request #33117: [SPARK-35859][PYTHON] Cleanup type hints in pandas-on-Spark

2021-06-29 Thread GitBox
ueshin commented on pull request #33117: URL: https://github.com/apache/spark/pull/33117#issuecomment-870796934 Thanks! merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] viirya commented on a change in pull request #32921: [SPARK-35779][SQL] Dynamic filtering for Data Source V2

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32921: URL: https://github.com/apache/spark/pull/32921#discussion_r660834539 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsDynamicFiltering.java ## @@ -0,0 +1,55 @@ +/* + * Licensed to the

[GitHub] [spark] viirya commented on a change in pull request #32921: [SPARK-35779][SQL] Dynamic filtering for Data Source V2

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32921: URL: https://github.com/apache/spark/pull/32921#discussion_r660834539 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsDynamicFiltering.java ## @@ -0,0 +1,55 @@ +/* + * Licensed to the

[GitHub] [spark] srowen commented on pull request #33135: [SPARK-35931][CORE][YARN] Ability to override Yarn Cluster Submit Class with Configuration

2021-06-29 Thread GitBox
srowen commented on pull request #33135: URL: https://github.com/apache/spark/pull/33135#issuecomment-870790464 I just don't know enough about YARN to review this one -- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] srowen commented on pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1

2021-06-29 Thread GitBox
srowen commented on pull request #33130: URL: https://github.com/apache/spark/pull/33130#issuecomment-870789567 I'm probably missing something - we don't have / need Jenkins tests, just the Github Actions? I just couldn't see test results here, or for the Chill change. -- This is an

[GitHub] [spark] sunchao commented on a change in pull request #32921: [SPARK-35779][SQL] Dynamic filtering for Data Source V2

2021-06-29 Thread GitBox
sunchao commented on a change in pull request #32921: URL: https://github.com/apache/spark/pull/32921#discussion_r660321387 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeFiltering.java ## @@ -0,0 +1,61 @@ +/* + * Licensed to the

[GitHub] [spark] dongjoon-hyun closed pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1

2021-06-29 Thread GitBox
dongjoon-hyun closed pull request #33130: URL: https://github.com/apache/spark/pull/33130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1

2021-06-29 Thread GitBox
dongjoon-hyun commented on a change in pull request #33130: URL: https://github.com/apache/spark/pull/33130#discussion_r660822494 ## File path: pom.xml ## @@ -2858,6 +2864,18 @@ org.apache.maven.plugins maven-shade-plugin 3.2.4 + +

[GitHub] [spark] dongjoon-hyun commented on pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33130: URL: https://github.com/apache/spark/pull/33130#issuecomment-870781939 Thank you, @viirya . Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] xuanyuanking commented on a change in pull request #32928: [SPARK-35784][SS] Implementation for RocksDB instance

2021-06-29 Thread GitBox
xuanyuanking commented on a change in pull request #32928: URL: https://github.com/apache/spark/pull/32928#discussion_r660815271 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala ## @@ -0,0 +1,451 @@ +/* + * Licensed to the

[GitHub] [spark] gengliangwang opened a new pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode

2021-06-29 Thread GitBox
gengliangwang opened a new pull request #33138: URL: https://github.com/apache/spark/pull/33138 ### What changes were proposed in this pull request? Add a new ANSI type coercion rule: when getting a date field from a Timestamp column, cast the column as Date type.

[GitHub] [spark] akshatb1 commented on pull request #33135: [SPARK-35931][CORE][YARN] Ability to override Yarn Cluster Submit Class with Configuration

2021-06-29 Thread GitBox
akshatb1 commented on pull request #33135: URL: https://github.com/apache/spark/pull/33135#issuecomment-870772962 @srowen @HyukjinKwon @tgravescs Could you kindly help in reviewing this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] viirya commented on a change in pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1

2021-06-29 Thread GitBox
viirya commented on a change in pull request #33130: URL: https://github.com/apache/spark/pull/33130#discussion_r660801520 ## File path: pom.xml ## @@ -2858,6 +2864,18 @@ org.apache.maven.plugins maven-shade-plugin 3.2.4 + +

[GitHub] [spark] otterc commented on a change in pull request #32140: [SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

2021-06-29 Thread GitBox
otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r660751432 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -386,40 +415,53 @@ final class

[GitHub] [spark] MaxGekk opened a new pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

2021-06-29 Thread GitBox
MaxGekk opened a new pull request #33137: URL: https://github.com/apache/spark/pull/33137 ### What changes were proposed in this pull request? In the PR, I propose to catch all non-fatal exceptions coming `refreshTable()` at the final stage of table repairing, and output an error

[GitHub] [spark] xinrong-databricks commented on a change in pull request #32139: [SPARK-35032][PYTHON] Port Koalas Index unit tests into PySpark

2021-06-29 Thread GitBox
xinrong-databricks commented on a change in pull request #32139: URL: https://github.com/apache/spark/pull/32139#discussion_r660786103 ## File path: dev/sparktestsupport/modules.py ## @@ -611,43 +611,47 @@ def __hash__(self): "pyspark.pandas.spark.utils",

[GitHub] [spark] xinrong-databricks commented on pull request #32955: [SPARK-35344][PYTHON] Support creating a Column of numpy literals in pandas API on Spark

2021-06-29 Thread GitBox
xinrong-databricks commented on pull request #32955: URL: https://github.com/apache/spark/pull/32955#issuecomment-870743798 Thanks @ueshin! I will file follow-up tickets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun edited a comment on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-870742689 FYI, in the master branch, I can test the module without this PR. Given that, `unsafe` module is not broken at least. Which module did you hit the failure? ```

[GitHub] [spark] dongjoon-hyun commented on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-870742689 FYI, in the master branch, I can test the module without this PR. ``` $ build/sbt "unsafe/test" ... [info] Run completed in 1 second, 56 milliseconds. [info]

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun commented on a change in pull request #33131: URL: https://github.com/apache/spark/pull/33131#discussion_r660777507 ## File path: common/unsafe/pom.xml ## @@ -56,6 +56,11 @@ chill_${scala.binary.version} + + com.esotericsoftware +

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun commented on a change in pull request #33131: URL: https://github.com/apache/spark/pull/33131#discussion_r660776280 ## File path: pom.xml ## @@ -3317,9 +3322,9 @@ scala-2.12 -

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun commented on a change in pull request #33131: URL: https://github.com/apache/spark/pull/33131#discussion_r660776009 ## File path: core/pom.xml ## @@ -33,7 +33,7 @@ core - + Review comment: Let's not touch irrelevant file. -- This is an

[GitHub] [spark] dongjoon-hyun commented on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-870739505 Thank you for making a PR. The GitHub Action works fine. Could you add how to reproduce the failure into the PR description? > I found that the build failed when I

[GitHub] [spark] otterc commented on a change in pull request #32140: [SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

2021-06-29 Thread GitBox
otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r660774462 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +878,83 @@ final class

[GitHub] [spark] dongjoon-hyun commented on pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1

2021-06-29 Thread GitBox
dongjoon-hyun commented on pull request #33130: URL: https://github.com/apache/spark/pull/33130#issuecomment-870736924 Could you review this please, @srowen and @viirya ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] otterc commented on a change in pull request #32140: [SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

2021-06-29 Thread GitBox
otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r660771829 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +878,83 @@ final class

[GitHub] [spark] viirya commented on a change in pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32980: URL: https://github.com/apache/spark/pull/32980#discussion_r660768427 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -76,24 +76,34 @@ object ExprCode {

[GitHub] [spark] Ngone51 commented on a change in pull request #32140: [SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

2021-06-29 Thread GitBox
Ngone51 commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r660766903 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +878,83 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

2021-06-29 Thread GitBox
otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r660751432 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -386,40 +415,53 @@ final class

[GitHub] [spark] Ngone51 commented on a change in pull request #33034: WIP: [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle

2021-06-29 Thread GitBox
Ngone51 commented on a change in pull request #33034: URL: https://github.com/apache/spark/pull/33034#discussion_r660750549 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java ## @@ -222,7 +223,7 @@ public void

[GitHub] [spark] otterc commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

2021-06-29 Thread GitBox
otterc commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r660749709 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/PushBlockStream.java ## @@ -99,10 +110,11 @@ public void

[GitHub] [spark] viirya commented on a change in pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32980: URL: https://github.com/apache/spark/pull/32980#discussion_r660748186 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -76,24 +76,34 @@ object ExprCode {

[GitHub] [spark] AmplabJenkins commented on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-870712325 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket

2021-06-29 Thread GitBox
AmplabJenkins commented on pull request #33132: URL: https://github.com/apache/spark/pull/33132#issuecomment-870712291 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] viirya commented on a change in pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32980: URL: https://github.com/apache/spark/pull/32980#discussion_r660745976 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -1049,17 +1103,25 @@ class

[GitHub] [spark] viirya commented on a change in pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

2021-06-29 Thread GitBox
viirya commented on a change in pull request #32980: URL: https://github.com/apache/spark/pull/32980#discussion_r660744333 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -1071,14 +1133,13 @@ class

[GitHub] [spark] itholic edited a comment on pull request #33128: [SPARK-35873][PYTHON] Cleanup the version logic from the pandas API on Spark

2021-06-29 Thread GitBox
itholic edited a comment on pull request #33128: URL: https://github.com/apache/spark/pull/33128#issuecomment-870706328 Thanks, @HyukjinKwon . Just fixed the PR description! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

<    2   3   4   5   6   7   8   9   10   >