[GitHub] [spark] yaooqinn commented on a change in pull request #32452: [SPARK-35243][SQL]Support columnar execution on ANSI interval types

yaooqinn commented on a change in pull request #32452: URL: https://github.com/apache/spark/pull/32452#discussion_r629135907 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala ## @@ -823,6 +823,8 @@ private[columnar] object

[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions

SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-836346923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #32491: avoid using deprecated `buildForBatch` and `buildForStreaming`

SparkQA commented on pull request #32491: URL: https://github.com/apache/spark/pull/32491#issuecomment-836346307 **[Test build #138329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138329/testReport)** for PR 32491 at commit

[GitHub] [spark] yaooqinn commented on a change in pull request #32452: [SPARK-35243][SQL]Support columnar execution on ANSI interval types

yaooqinn commented on a change in pull request #32452: URL: https://github.com/apache/spark/pull/32452#discussion_r629135304 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnBuilder.scala ## @@ -181,6 +181,8 @@ private[columnar] object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still co

AmplabJenkins removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836344082 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42849/

[GitHub] [spark] yaooqinn commented on a change in pull request #32452: [SPARK-35243][SQL]Support columnar execution on ANSI interval types

yaooqinn commented on a change in pull request #32452: URL: https://github.com/apache/spark/pull/32452#discussion_r629135012 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnAccessor.scala ## @@ -145,6 +151,8 @@ private[sql] object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

AmplabJenkins removed a comment on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-836344080 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42845/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respects `spark.sql.addPartitionInBatch.size` too

AmplabJenkins removed a comment on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836344079 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42847/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

AmplabJenkins removed a comment on pull request #32490: URL: https://github.com/apache/spark/pull/32490#issuecomment-836344078 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42846/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

AmplabJenkins removed a comment on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836344087 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138318/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions

AmplabJenkins removed a comment on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-836344091 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138326/

[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

AmplabJenkins commented on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836344087 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138318/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue r

AmplabJenkins commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836344082 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42849/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

AmplabJenkins commented on pull request #32490: URL: https://github.com/apache/spark/pull/32490#issuecomment-836344078 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42846/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respects `spark.sql.addPartitionInBatch.size` too

AmplabJenkins commented on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836344079 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42847/ --

[GitHub] [spark] SparkQA commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

SparkQA commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-836335904 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42845/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

SparkQA commented on pull request #32490: URL: https://github.com/apache/spark/pull/32490#issuecomment-83660 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42846/ --

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836332669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

SparkQA commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-836329378 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42845/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respects `spark.sql.addPartitionInBatch.size` too

SparkQA commented on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836328103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] HyukjinKwon commented on pull request #32491: avoid using deprecated `buildForBatch` and `buildForStreaming`

HyukjinKwon commented on pull request #32491: URL: https://github.com/apache/spark/pull/32491#issuecomment-836322976 @linhongliu-db can we file a JIRA? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon closed pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

HyukjinKwon closed pull request #32490: URL: https://github.com/apache/spark/pull/32490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] HyukjinKwon commented on pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

HyukjinKwon commented on pull request #32490: URL: https://github.com/apache/spark/pull/32490#issuecomment-836320878 Thanks Dongjoon. Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions

SparkQA removed a comment on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-836280820 **[Test build #138326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138326/testReport)** for PR 32470 at commit

[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions

SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-836319291 **[Test build #138326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138326/testReport)** for PR 32470 at commit

[GitHub] [spark] linhongliu-db opened a new pull request #32491: avoid using deprecated `buildForBatch` and `buildForStreaming`

linhongliu-db opened a new pull request #32491: URL: https://github.com/apache/spark/pull/32491 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

SparkQA removed a comment on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836185216 **[Test build #138318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138318/testReport)** for PR 32031 at commit

[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

SparkQA commented on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836304888 **[Test build #138318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138318/testReport)** for PR 32031 at commit

[GitHub] [spark] viirya commented on pull request #32482: [SPARK-35332][SQL] Make cache plan disable configs configurable

viirya commented on pull request #32482: URL: https://github.com/apache/spark/pull/32482#issuecomment-836304662 So it sounds like this wants to have an aggressive option when preparing cache plan, right? The config name can be refined actually. The first glance confuses me. -- This is

[GitHub] [spark] dongjoon-hyun commented on pull request #32192: [SPARK-34720][SQL] MERGE ... UPDATE/INSERT * should do by-name resolution

dongjoon-hyun commented on pull request #32192: URL: https://github.com/apache/spark/pull/32192#issuecomment-836301884 Got it for the further explanation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] viirya commented on a change in pull request #32482: [SPARK-35332][SQL] Make cache plan disable configs configurable

viirya commented on a change in pull request #32482: URL: https://github.com/apache/spark/pull/32482#discussion_r629110699 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ## @@ -1069,21 +1069,26 @@ object SparkSession extends Logging { }

[GitHub] [spark] dongjoon-hyun commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

dongjoon-hyun commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836296880 Merged to master/3.1/3.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] cfmcgrady commented on pull request #32488: [SPARK-35316][SQL] UnwrapCastInBinaryComparison support In/InSet predicate

cfmcgrady commented on pull request #32488: URL: https://github.com/apache/spark/pull/32488#issuecomment-836295528 cc @wangyum -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] dongjoon-hyun closed pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

dongjoon-hyun closed pull request #32487: URL: https://github.com/apache/spark/pull/32487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] cloud-fan commented on pull request #32192: [SPARK-34720][SQL] MERGE ... UPDATE/INSERT * should do by-name resolution

cloud-fan commented on pull request #32192: URL: https://github.com/apache/spark/pull/32192#issuecomment-836295271 I've made it clear at the beginning of the PR description: INSERT/UPDATE * is an extension and I can't find it in other mainstream databases. If you open the docs you posted,

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still co

AmplabJenkins removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836290771 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138321/

[GitHub] [spark] AmplabJenkins commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue r

AmplabJenkins commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836290771 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138321/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue

SparkQA removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836192784 **[Test build #138321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138321/testReport)** for PR 32399 at commit

[GitHub] [spark] viirya commented on a change in pull request #32482: [SPARK-35332][SQL] Make cache plan disable configs configurable

viirya commented on a change in pull request #32482: URL: https://github.com/apache/spark/pull/32482#discussion_r629104846 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1090,6 +1090,17 @@ object SQLConf { .booleanConf

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836289096 **[Test build #138321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138321/testReport)** for PR 32399 at commit

[GitHub] [spark] beliefer commented on a change in pull request #32409: [SPARK-35285][SQL] Parse ANSI interval types in SQL schema

beliefer commented on a change in pull request #32409: URL: https://github.com/apache/spark/pull/32409#discussion_r629103567 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala ## @@ -1098,7 +1098,7 @@ class

[GitHub] [spark] viirya commented on a change in pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still conti

viirya commented on a change in pull request #32399: URL: https://github.com/apache/spark/pull/32399#discussion_r629103247 ## File path: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ## @@ -183,11 +190,28 @@ class CrossValidator @Since("1.2.0")

[GitHub] [spark] cloud-fan closed pull request #32481: [SPARK-35261][SQL][TESTS][FOLLOW-UP] Change failOnError to false for NativeAdd in V2FunctionBenchmark

cloud-fan closed pull request #32481: URL: https://github.com/apache/spark/pull/32481 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32481: [SPARK-35261][SQL][TESTS][FOLLOW-UP] Change failOnError to false for NativeAdd in V2FunctionBenchmark

AmplabJenkins removed a comment on pull request #32481: URL: https://github.com/apache/spark/pull/32481#issuecomment-835744112 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138302/

[GitHub] [spark] cloud-fan commented on pull request #32481: [SPARK-35261][SQL][TESTS][FOLLOW-UP] Change failOnError to false for NativeAdd in V2FunctionBenchmark

cloud-fan commented on pull request #32481: URL: https://github.com/apache/spark/pull/32481#issuecomment-836285718 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] c21 commented on pull request #32482: [SPARK-35332][SQL] Make cache plan disable configs configurable

c21 commented on pull request #32482: URL: https://github.com/apache/spark/pull/32482#issuecomment-836285542 cc @viirya as well as he was finding the bug and raised the concern for auto bucketed scan for cached query. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

SparkQA commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-836281556 **[Test build #138328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138328/testReport)** for PR 31776 at commit

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836280942 **[Test build #138327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138327/testReport)** for PR 32399 at commit

[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions

SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-836280820 **[Test build #138326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138326/testReport)** for PR 32470 at commit

[GitHub] [spark] SparkQA commented on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respects `spark.sql.addPartitionInBatch.size` too

SparkQA commented on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836280528 **[Test build #138325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138325/testReport)** for PR 32489 at commit

[GitHub] [spark] SparkQA commented on pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

SparkQA commented on pull request #32490: URL: https://github.com/apache/spark/pull/32490#issuecomment-836280510 **[Test build #138324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138324/testReport)** for PR 32490 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still co

AmplabJenkins removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836278795 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138317/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

AmplabJenkins removed a comment on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836278801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42844/

[GitHub] [spark] AmplabJenkins commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue r

AmplabJenkins commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836278795 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138317/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

AmplabJenkins commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836278801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42844/ --

[GitHub] [spark] AngersZhuuuu commented on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respects `spark.sql.addPartitionInBatch.size` too

AngersZh commented on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836274023 > Need to change the config doc since it mentions the concrete command: > >

[GitHub] [spark] WeichenXu123 commented on a change in pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still

WeichenXu123 commented on a change in pull request #32399: URL: https://github.com/apache/spark/pull/32399#discussion_r629096547 ## File path: python/pyspark/util.py ## @@ -263,6 +264,69 @@ def _parse_memory(s): return int(float(s[:-1]) * units[s[-1].lower()]) +def

[GitHub] [spark] MaxGekk commented on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respect `spark.sql.addPartitionInBatch.size` too

MaxGekk commented on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836272718 RepairTableCommand respect -> RepairTableCommand respects -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

SparkQA commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836266886 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42844/ --

[GitHub] [spark] HyukjinKwon opened a new pull request #32490: [MINOR][INFRA] Add python/.idea into git ignore

HyukjinKwon opened a new pull request #32490: URL: https://github.com/apache/spark/pull/32490 ### What changes were proposed in this pull request? This PR adds `python/.idea` into Git ignore. PyCharm is supposed to be open against `python` directory which contains `pyspark` package

[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

cloud-fan commented on a change in pull request #32448: URL: https://github.com/apache/spark/pull/32448#discussion_r629091515 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -707,32 +710,63 @@ class

[GitHub] [spark] viirya commented on a change in pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still conti

viirya commented on a change in pull request #32399: URL: https://github.com/apache/spark/pull/32399#discussion_r629091240 ## File path: dev/sparktestsupport/modules.py ## @@ -565,6 +565,7 @@ def __hash__(self): "pyspark.ml.tests.test_stat",

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still

HyukjinKwon commented on a change in pull request #32399: URL: https://github.com/apache/spark/pull/32399#discussion_r629086511 ## File path: mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala ## @@ -161,11 +169,26 @@ class TrainValidationSplit

[GitHub] [spark] WeichenXu123 commented on a change in pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still

WeichenXu123 commented on a change in pull request #32399: URL: https://github.com/apache/spark/pull/32399#discussion_r629087709 ## File path: mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala ## @@ -161,11 +169,26 @@ class TrainValidationSplit

[GitHub] [spark] cloud-fan edited a comment on pull request #32482: [SPARK-35332][SQL] Make cache plan disable configs configurable

cloud-fan edited a comment on pull request #32482: URL: https://github.com/apache/spark/pull/32482#issuecomment-836259432 In general, I think it's better to optimize the cached plan more aggressively for better performance, even though it may cause perf regression due to output

[GitHub] [spark] cloud-fan commented on pull request #32482: [SPARK-35332][SQL] Make cache plan disable configs configurable

cloud-fan commented on pull request #32482: URL: https://github.com/apache/spark/pull/32482#issuecomment-836259432 In general, I think it's better to optimize the cached plan more aggressively for better performance, even though it may cause perf regression due to output partitioning

[GitHub] [spark] MaxGekk commented on a change in pull request #32409: [SPARK-35285][SQL] Parse ANSI interval types in SQL schema

MaxGekk commented on a change in pull request #32409: URL: https://github.com/apache/spark/pull/32409#discussion_r629088422 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala ## @@ -1098,7 +1098,7 @@ class

[GitHub] [spark] WeichenXu123 commented on a change in pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still

WeichenXu123 commented on a change in pull request #32399: URL: https://github.com/apache/spark/pull/32399#discussion_r629087709 ## File path: mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala ## @@ -161,11 +169,26 @@ class TrainValidationSplit

[GitHub] [spark] cloud-fan commented on pull request #31269: [SPARK-33933][SQL] Materialize BroadcastQueryStage first to try to avoid broadcast timeout in AQE

cloud-fan commented on pull request #31269: URL: https://github.com/apache/spark/pull/31269#issuecomment-836252782 @zhongyu09 please open a new JIRA, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on a change in pull request #32454: [SPARK-35327][SQL][TESTS] Filters out the TPC-DS queries that can cause flaky test results

cloud-fan commented on a change in pull request #32454: URL: https://github.com/apache/spark/pull/32454#discussion_r629085064 ## File path: sql/core/src/test/scala/org/apache/spark/sql/TPCDSBase.scala ## @@ -24,7 +24,7 @@ import org.apache.spark.sql.test.SharedSparkSession

[GitHub] [spark] AngersZhuuuu commented on pull request #32489: [SPARK-35360][SQL] RepairTableCommand respect `spark.sql.addPartitionInBatch.size` too

AngersZh commented on pull request #32489: URL: https://github.com/apache/spark/pull/32489#issuecomment-836250813 ping @MaxGekk @wangyum @maropu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] MaxGekk closed pull request #32444: [SPARK-35111][SPARK-35112][SQL][FOLLOWUP] Rename ANSI interval patterns and regexps

MaxGekk closed pull request #32444: URL: https://github.com/apache/spark/pull/32444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] SparkQA removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue

SparkQA removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836152449 **[Test build #138317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138317/testReport)** for PR 32399 at commit

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836244182 **[Test build #138317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138317/testReport)** for PR 32399 at commit

[GitHub] [spark] MaxGekk commented on pull request #32444: [SPARK-35111][SPARK-35112][SQL][FOLLOWUP] Rename ANSI interval patterns and regexps

MaxGekk commented on pull request #32444: URL: https://github.com/apache/spark/pull/32444#issuecomment-836243414 +1, LGTM. Merging to master. Thank you, @AngersZh and @cloud-fan for your review. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AngersZhuuuu opened a new pull request #32489: [SPARK-35360][SQL] RepairTableCommand respect `spark.sql.addPartitionInBatch.size` too

AngersZh opened a new pull request #32489: URL: https://github.com/apache/spark/pull/32489 ### What changes were proposed in this pull request? RepairTableCommand respect `spark.sql.addPartitionInBatch.size` too ### Why are the changes needed? Make RepairTableCommand

[GitHub] [spark] SparkQA commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

SparkQA commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836238497 **[Test build #138323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138323/testReport)** for PR 32487 at commit

[GitHub] [spark] SparkQA commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

SparkQA commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836235893 **[Test build #138322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138322/testReport)** for PR 32487 at commit

[GitHub] [spark] viirya commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

viirya commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836235572 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

AmplabJenkins removed a comment on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836233460 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138314/

[GitHub] [spark] HyukjinKwon commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

HyukjinKwon commented on pull request #32448: URL: https://github.com/apache/spark/pull/32448#issuecomment-836233745 Can you also update https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2078-L2082 -- This is an automated message

[GitHub] [spark] AmplabJenkins commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

AmplabJenkins commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836233460 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138314/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

SparkQA removed a comment on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836145492 **[Test build #138314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138314/testReport)** for PR 32487 at commit

[GitHub] [spark] SparkQA commented on pull request #32487: [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM

SparkQA commented on pull request #32487: URL: https://github.com/apache/spark/pull/32487#issuecomment-836231948 **[Test build #138314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138314/testReport)** for PR 32487 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still co

AmplabJenkins removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836230224 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42842/

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

HyukjinKwon commented on a change in pull request #32448: URL: https://github.com/apache/spark/pull/32448#discussion_r629072673 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -707,32 +710,63 @@ class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still co

AmplabJenkins removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836229891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836230158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue r

AmplabJenkins commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836230224 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42842/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

AmplabJenkins removed a comment on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836229893 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42840/

[GitHub] [spark] AmplabJenkins commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue r

AmplabJenkins commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836229892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

AmplabJenkins commented on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836229893 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42840/ --

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836229015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

SparkQA commented on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836228137 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42840/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or

SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-836226769 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42843/ --

[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

SparkQA commented on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-836224236 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42840/ -- This is an automated message from the Apache

<    1   2   3   4   5