[GitHub] [spark] AmplabJenkins commented on pull request #32161: [SPARK-35025][SQL][PYTHON][DOCS] Move Parquet data source options from Python and Scala into a single page.

AmplabJenkins commented on pull request #32161: URL: https://github.com/apache/spark/pull/32161#issuecomment-844978648 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43262/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #32590: [SPARK-35445][SQL] Reduce the execution time of DeduplicateRelations

AmplabJenkins commented on pull request #32590: URL: https://github.com/apache/spark/pull/32590#issuecomment-844978646 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43260/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

AmplabJenkins commented on pull request #32546: URL: https://github.com/apache/spark/pull/32546#issuecomment-844978650 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43264/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

AmplabJenkins commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844978644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For q

[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

SparkQA commented on pull request #32546: URL: https://github.com/apache/spark/pull/32546#issuecomment-844971083 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43264/ -- This is an automated message from the A

[GitHub] [spark] SparkQA removed a comment on pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

SparkQA removed a comment on pull request #32604: URL: https://github.com/apache/spark/pull/32604#issuecomment-844829156 **[Test build #138734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138734/testReport)** for PR 32604 at commit [`3417c22`](https://gi

[GitHub] [spark] SparkQA commented on pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

SparkQA commented on pull request #32604: URL: https://github.com/apache/spark/pull/32604#issuecomment-844969400 **[Test build #138734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138734/testReport)** for PR 32604 at commit [`3417c22`](https://github.co

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32161: [SPARK-35025][SQL][PYTHON][DOCS] Move Parquet data source options from Python and Scala into a single page.

HyukjinKwon commented on a change in pull request #32161: URL: https://github.com/apache/spark/pull/32161#discussion_r635977077 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -860,13 +860,14 @@ final class DataFrameWriter[T] private[sql](

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32161: [SPARK-35025][SQL][PYTHON][DOCS] Move Parquet data source options from Python and Scala into a single page.

HyukjinKwon commented on a change in pull request #32161: URL: https://github.com/apache/spark/pull/32161#discussion_r635976767 ## File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala ## @@ -476,44 +476,21 @@ final class DataStreamReader pri

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32204: URL: https://github.com/apache/spark/pull/32204#discussion_r635976346 ## File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala ## @@ -269,73 +218,20 @@ final class DataStreamReader pri

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32204: URL: https://github.com/apache/spark/pull/32204#discussion_r635975929 ## File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala ## @@ -269,73 +218,20 @@ final class DataStreamReader pri

[GitHub] [spark] Kimahriman edited a comment on pull request #32595: [SPARK-35449][SQL] Only extract common expressions from CaseWhen values if elseValue is set

Kimahriman edited a comment on pull request #32595: URL: https://github.com/apache/spark/pull/32595#issuecomment-844960099 > is it a perf-only issue? Yeah because of the UDF issue I'd consider it more a bug with performance side-effects. Whether those side-effects are positive or neg

[GitHub] [spark] Kimahriman commented on pull request #32595: [SPARK-35449][SQL] Only extract common expressions from CaseWhen values if elseValue is set

Kimahriman commented on pull request #32595: URL: https://github.com/apache/spark/pull/32595#issuecomment-844960099 > is it a perf-only issue? Yeah because of the UDF issue I'd consider it more a bug with performance side-effects. Whether those side-effects are positive or negative l

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635974286 ## File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala ## @@ -453,20 +453,21 @@ final class DataStreamReader pri

[GitHub] [spark] SparkQA commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

SparkQA commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844959921 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43258/ -- This is an automated message from the A

[GitHub] [spark] SparkQA removed a comment on pull request #32607: [SPARK-35459][SQL][TESTS] Move `AvroRowReaderSuite` to a separate file

SparkQA removed a comment on pull request #32607: URL: https://github.com/apache/spark/pull/32607#issuecomment-844933489 **[Test build #138743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138743/testReport)** for PR 32607 at commit [`1d1f246`](https://gi

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635973928 ## File path: docs/sql-data-sources-orc.md ## @@ -172,3 +172,26 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC

[GitHub] [spark] SparkQA commented on pull request #32607: [SPARK-35459][SQL][TESTS] Move `AvroRowReaderSuite` to a separate file

SparkQA commented on pull request #32607: URL: https://github.com/apache/spark/pull/32607#issuecomment-844958983 **[Test build #138743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138743/testReport)** for PR 32607 at commit [`1d1f246`](https://github.co

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635973164 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](

[GitHub] [spark] SparkQA commented on pull request #32590: [SPARK-35445][SQL] Reduce the execution time of DeduplicateRelations

SparkQA commented on pull request #32590: URL: https://github.com/apache/spark/pull/32590#issuecomment-844958069 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43260/ -- This is an automated message from the A

[GitHub] [spark] Kimahriman commented on a change in pull request #32595: [SPARK-35449][SQL] Only extract common expressions from CaseWhen values if elseValue is set

Kimahriman commented on a change in pull request #32595: URL: https://github.com/apache/spark/pull/32595#discussion_r635973028 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala ## @@ -226,6 +226,17 @@ class

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635972738 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](

[GitHub] [spark] SparkQA commented on pull request #32161: [SPARK-35025][SQL][PYTHON][DOCS] Move Parquet data source options from Python and Scala into a single page.

SparkQA commented on pull request #32161: URL: https://github.com/apache/spark/pull/32161#issuecomment-844955297 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43262/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

SparkQA commented on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844954744 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43259/ -- This is an automated message from the A

[GitHub] [spark] SparkQA removed a comment on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

SparkQA removed a comment on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844725650 **[Test build #138730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138730/testReport)** for PR 32602 at commit [`5097247`](https://gi

[GitHub] [spark] HyukjinKwon commented on pull request #32586: [SPARK-35439][SQL] Children subexpr should come first than parent subexpr

HyukjinKwon commented on pull request #32586: URL: https://github.com/apache/spark/pull/32586#issuecomment-844950334 nit but maybe we should also update the PR description: > Replacing HashMap with LinkedHashMap can deal with it. -- This is an automated message from the Apache Git Serv

[GitHub] [spark] SparkQA commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

SparkQA commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844950368 **[Test build #138730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138730/testReport)** for PR 32602 at commit [`5097247`](https://github.co

[GitHub] [spark] JkSelf commented on pull request #32594: [SPARK-35447][SQL] Optimize skew join before coalescing shuffle partitions

JkSelf commented on pull request #32594: URL: https://github.com/apache/spark/pull/32594#issuecomment-844949996 > It's a bit hard to reason about skew join if partitions have been coalesced. A skewed partition needs to be much larger than other partitions and we need to look at the raw siz

[GitHub] [spark] beliefer commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

beliefer commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635965198 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/CommandResultExec.scala ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32586: [SPARK-35439][SQL] Children subexpr should come first than parent subexpr

HyukjinKwon commented on a change in pull request #32586: URL: https://github.com/apache/spark/pull/32586#discussion_r635963399 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ## @@ -167,7 +167,28 @@ class Equival

[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

SparkQA commented on pull request #32546: URL: https://github.com/apache/spark/pull/32546#issuecomment-844943018 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43264/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon closed pull request #32571: [SPARK-35424][SHUFFLE] Remove some useless code in the ExternalBlockHandler.

HyukjinKwon closed pull request #32571: URL: https://github.com/apache/spark/pull/32571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pl

[GitHub] [spark] SparkQA removed a comment on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

SparkQA removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-844933776 **[Test build #138745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138745/testReport)** for PR 32513 at commit [`8e9277d`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

AmplabJenkins removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-844941060 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138745/ -

[GitHub] [spark] HyukjinKwon commented on pull request #32571: [SPARK-35424][SHUFFLE] Remove some useless code in the ExternalBlockHandler.

HyukjinKwon commented on pull request #32571: URL: https://github.com/apache/spark/pull/32571#issuecomment-844941284 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-844940995 **[Test build #138745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138745/testReport)** for PR 32513 at commit [`8e9277d`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

AmplabJenkins commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-844941060 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138745/ -- This

[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

SparkQA commented on pull request #32546: URL: https://github.com/apache/spark/pull/32546#issuecomment-844937566 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43261/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

AmplabJenkins removed a comment on pull request #32204: URL: https://github.com/apache/spark/pull/32204#issuecomment-844935046 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43263/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

AmplabJenkins removed a comment on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844932004 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43257/

[GitHub] [spark] AmplabJenkins commented on pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

AmplabJenkins commented on pull request #32204: URL: https://github.com/apache/spark/pull/32204#issuecomment-844935046 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43263/ -- T

[GitHub] [spark] SparkQA commented on pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

SparkQA commented on pull request #32204: URL: https://github.com/apache/spark/pull/32204#issuecomment-844935022 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43263/ -- This

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32587: [SPARK-35440][SQL] Add language type to `ExpressionInfo` for UDF

HyukjinKwon commented on a change in pull request #32587: URL: https://github.com/apache/spark/pull/32587#discussion_r635953428 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java ## @@ -144,6 +155,11 @@ public ExpressionInfo(

[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-844933776 **[Test build #138745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138745/testReport)** for PR 32513 at commit [`8e9277d`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #32606: [SPARK-35287][SQL] Allow RemoveRedundantProjects to preserve ProjectExec which generates UnsafeRow for DataSourceV2ScanRelation

SparkQA commented on pull request #32606: URL: https://github.com/apache/spark/pull/32606#issuecomment-844933465 **[Test build #138744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138744/testReport)** for PR 32606 at commit [`7ecd073`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #32607: [SPARK-35459][SQL][TESTS] Move `AvroRowReaderSuite` to a separate file

SparkQA commented on pull request #32607: URL: https://github.com/apache/spark/pull/32607#issuecomment-844933489 **[Test build #138743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138743/testReport)** for PR 32607 at commit [`1d1f246`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #32605: [WIP][SPARK-35446] Override getJDBCType in MySQLDialect to map FloatType to FLOAT

AmplabJenkins commented on pull request #32605: URL: https://github.com/apache/spark/pull/32605#issuecomment-844933165 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [spark] AmplabJenkins commented on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

AmplabJenkins commented on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844932004 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43257/ -- T

[GitHub] [spark] SparkQA commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

SparkQA commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844930193 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43258/ -- This is an automated message from the Apache

[GitHub] [spark] MaxGekk commented on a change in pull request #30221: [SPARK-33314][SQL] Avoid dropping rows in Avro reader

MaxGekk commented on a change in pull request #30221: URL: https://github.com/apache/spark/pull/30221#discussion_r635948561 ## File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ## @@ -2005,3 +2027,61 @@ class AvroV2Suite extends AvroSuite with

[GitHub] [spark] MaxGekk opened a new pull request #32607: [SPARK-35459][SQL][TESTS] Move `AvroRowReaderSuite` to a separate file

MaxGekk opened a new pull request #32607: URL: https://github.com/apache/spark/pull/32607 ### What changes were proposed in this pull request? Move `AvroRowReaderSuite` out from `AvroSuite.scala` and place it to `AvroRowReaderSuite.scala`. ### Why are the changes needed? To imp

[GitHub] [spark] SparkQA commented on pull request #32590: [SPARK-35445][SQL] Reduce the execution time of DeduplicateRelations

SparkQA commented on pull request #32590: URL: https://github.com/apache/spark/pull/32590#issuecomment-844928082 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43260/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #32161: [SPARK-35025][SQL][PYTHON][DOCS] Move Parquet data source options from Python and Scala into a single page.

SparkQA commented on pull request #32161: URL: https://github.com/apache/spark/pull/32161#issuecomment-844926743 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43262/ -- This is an automated message from the Apache

[GitHub] [spark] beliefer commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command so that query command with CTE

beliefer commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635944226 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -74,12 +75,26 @@ class QueryExecution( sparkSession

[GitHub] [spark] SparkQA commented on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

SparkQA commented on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844923098 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43259/ -- This is an automated message from the Apache

[GitHub] [spark] beliefer commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute Command so that query command with CTE

beliefer commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635938935 ## File path: sql/core/src/main/scala/org/apache/spark/sql/expressions/CommandResult.scala ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software F

[GitHub] [spark] sarutak opened a new pull request #32606: [SPARK-35287][SQL] Allow RemoveRedundantProjects to preserve ProjectExec which generates UnsafeRow for DataSourceV2ScanRelation

sarutak opened a new pull request #32606: URL: https://github.com/apache/spark/pull/32606 ### What changes were proposed in this pull request? This PR fixes an issue that `RemoveRedundantProjects` removes `ProjectExec` which is for generating `UnsafeRow`. In `DataSourceV2Strategy`

[GitHub] [spark] mariosmeim-db opened a new pull request #32605: [WIP][SPARK-35446] Override getJDBCType in MySQLDialect to map FloatType to FLOAT

mariosmeim-db opened a new pull request #32605: URL: https://github.com/apache/spark/pull/32605 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

[GitHub] [spark] SparkQA commented on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

SparkQA commented on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844905052 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43257/ -- This is an automated message from the A

[GitHub] [spark] cloud-fan commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute Command so that query command with CTE

cloud-fan commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635923354 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -74,12 +75,26 @@ class QueryExecution( sparkSessio

[GitHub] [spark] cloud-fan commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute Command so that query command with CTE

cloud-fan commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635923166 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -223,10 +224,10 @@ class Dataset[T] private[sql]( @transient private[s

[GitHub] [spark] cloud-fan commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute Command so that query command with CTE

cloud-fan commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635922487 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -74,12 +75,26 @@ class QueryExecution( sparkSessio

[GitHub] [spark] cloud-fan commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute Command so that query command with CTE

cloud-fan commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635921988 ## File path: sql/core/src/main/scala/org/apache/spark/sql/expressions/CommandResult.scala ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] cloud-fan commented on a change in pull request #32513: [SPARK-35378][SQL] Eagerly execute Command so that query command with CTE

cloud-fan commented on a change in pull request #32513: URL: https://github.com/apache/spark/pull/32513#discussion_r635921117 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/CommandResultExec.scala ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Softwar

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

AmplabJenkins removed a comment on pull request #32604: URL: https://github.com/apache/spark/pull/32604#issuecomment-844894018 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43256/

[GitHub] [spark] AmplabJenkins commented on pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

AmplabJenkins commented on pull request #32604: URL: https://github.com/apache/spark/pull/32604#issuecomment-844894018 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43256/ -- T

[GitHub] [spark] SparkQA commented on pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

SparkQA commented on pull request #32604: URL: https://github.com/apache/spark/pull/32604#issuecomment-844893976 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43256/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

SparkQA commented on pull request #32546: URL: https://github.com/apache/spark/pull/32546#issuecomment-844889966 **[Test build #138742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138742/testReport)** for PR 32546 at commit [`6358d59`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

SparkQA commented on pull request #32204: URL: https://github.com/apache/spark/pull/32204#issuecomment-844886889 **[Test build #138741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138741/testReport)** for PR 32204 at commit [`f4d9843`](https://github.com

[GitHub] [spark] gengliangwang commented on a change in pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

gengliangwang commented on a change in pull request #32604: URL: https://github.com/apache/spark/pull/32604#discussion_r635913210 ## File path: build/mvn ## @@ -78,7 +78,7 @@ install_app() { echo " ${local_tarball}" >> ${local_checksum} # two spaces + file are importan

[GitHub] [spark] SparkQA commented on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

SparkQA commented on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844882808 **[Test build #138740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138740/testReport)** for PR 32600 at commit [`cb73f13`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32603: [SPARK-35457][BUILD] Bump ANTLR runtime version to 4.8

AmplabJenkins removed a comment on pull request #32603: URL: https://github.com/apache/spark/pull/32603#issuecomment-844877813 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138731/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

AmplabJenkins removed a comment on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844877810 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43252/

[GitHub] [spark] gengliangwang commented on a change in pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

gengliangwang commented on a change in pull request #32204: URL: https://github.com/apache/spark/pull/32204#discussion_r635909326 ## File path: docs/sql-data-sources-json.md ## @@ -94,3 +94,171 @@ SELECT * FROM jsonTable + +## Data Source Option + +Data source options of

[GitHub] [spark] SparkQA commented on pull request #32161: [SPARK-35025][SQL][PYTHON][DOCS] Move Parquet data source options from Python and Scala into a single page.

SparkQA commented on pull request #32161: URL: https://github.com/apache/spark/pull/32161#issuecomment-844880004 **[Test build #138739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138739/testReport)** for PR 32161 at commit [`ffc124c`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

SparkQA commented on pull request #32546: URL: https://github.com/apache/spark/pull/32546#issuecomment-844879518 **[Test build #138738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138738/testReport)** for PR 32546 at commit [`0b0e183`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #32590: [SPARK-35445][SQL] Reduce the execution time of DeduplicateRelations

SparkQA commented on pull request #32590: URL: https://github.com/apache/spark/pull/32590#issuecomment-844879414 **[Test build #138737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138737/testReport)** for PR 32590 at commit [`ef50b77`](https://github.com

[GitHub] [spark] itholic commented on a change in pull request #32204: [SPARK-34494][SQL][DOCS] Move JSON data source options from Python and Scala into a single page

itholic commented on a change in pull request #32204: URL: https://github.com/apache/spark/pull/32204#discussion_r635906645 ## File path: docs/sql-data-sources-json.md ## @@ -94,3 +94,171 @@ SELECT * FROM jsonTable + +## Data Source Option + +Data source options of JSON c

[GitHub] [spark] SparkQA commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

SparkQA commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844879338 **[Test build #138736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138736/testReport)** for PR 32602 at commit [`165077b`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #32603: [SPARK-35457][BUILD] Bump ANTLR runtime version to 4.8

AmplabJenkins commented on pull request #32603: URL: https://github.com/apache/spark/pull/32603#issuecomment-844877813 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138731/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

AmplabJenkins commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844877810 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43252/ -- T

[GitHub] [spark] SparkQA commented on pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

SparkQA commented on pull request #32600: URL: https://github.com/apache/spark/pull/32600#issuecomment-844875871 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43257/ -- This is an automated message from the Apache

[GitHub] [spark] ulysses-you commented on a change in pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

ulysses-you commented on a change in pull request #32602: URL: https://github.com/apache/spark/pull/32602#discussion_r635897104 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateUnnecessaryJoin.scala ## @@ -59,19 +66,33 @@ object EliminateU

[GitHub] [spark] Ngone51 commented on a change in pull request #32590: [SPARK-35445][SQL] Reduce the execution time of DeduplicateRelations

Ngone51 commented on a change in pull request #32590: URL: https://github.com/apache/spark/pull/32590#discussion_r635896472 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala ## @@ -67,61 +81,78 @@ object DeduplicateRela

[GitHub] [spark] Ngone51 commented on a change in pull request #32590: [SPARK-35445][SQL] Reduce the execution time of DeduplicateRelations

Ngone51 commented on a change in pull request #32590: URL: https://github.com/apache/spark/pull/32590#discussion_r635895661 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala ## @@ -17,16 +17,30 @@ package org.apache.

[GitHub] [spark] SparkQA commented on pull request #32604: [SPARK-35458][BUILD] Use ` > /dev/null 2>&1` to replace `-q` in shasum

SparkQA commented on pull request #32604: URL: https://github.com/apache/spark/pull/32604#issuecomment-844865587 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43256/ -- This is an automated message from the Apache

[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

itholic commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635890871 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds:

[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

itholic commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635890871 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds:

[GitHub] [spark] Ngone51 edited a comment on pull request #32136: [SPARK-35022][CORE] Task Scheduling Plugin in Spark

Ngone51 edited a comment on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-844858158 > However I'm not sure if stage level scheduling can deal with executor lost case. Based on above comment, seems it cannot. That will be a major concern for the use-case

[GitHub] [spark] itholic commented on a change in pull request #32546: [SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page

itholic commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635890871 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -881,14 +881,10 @@ final class DataFrameWriter[T] private[sql](ds:

[GitHub] [spark] SparkQA removed a comment on pull request #32603: [SPARK-35457][BUILD] Bump ANTLR runtime version to 4.8

SparkQA removed a comment on pull request #32603: URL: https://github.com/apache/spark/pull/32603#issuecomment-844733128 **[Test build #138731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138731/testReport)** for PR 32603 at commit [`b8f20cc`](https://gi

[GitHub] [spark] SparkQA commented on pull request #32603: [SPARK-35457][BUILD] Bump ANTLR runtime version to 4.8

SparkQA commented on pull request #32603: URL: https://github.com/apache/spark/pull/32603#issuecomment-844861177 **[Test build #138731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138731/testReport)** for PR 32603 at commit [`b8f20cc`](https://github.co

[GitHub] [spark] Ngone51 edited a comment on pull request #32136: [SPARK-35022][CORE] Task Scheduling Plugin in Spark

Ngone51 edited a comment on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-844858158 > However I'm not sure if stage level scheduling can deal with executor lost case. Based on above comment, seems it cannot. That will be a major concern for the use-case

[GitHub] [spark] Ngone51 edited a comment on pull request #32136: [SPARK-35022][CORE] Task Scheduling Plugin in Spark

Ngone51 edited a comment on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-844858158 > However I'm not sure if stage level scheduling can deal with executor lost case. Based on above comment, seems it cannot. That will be a major concern for the use-case

[GitHub] [spark] Ngone51 commented on pull request #32136: [SPARK-35022][CORE] Task Scheduling Plugin in Spark

Ngone51 commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-844858158 > However I'm not sure if stage level scheduling can deal with executor lost case. Based on above comment, seems it cannot. That will be a major concern for the use-case here. D

[GitHub] [spark] yaooqinn commented on a change in pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

yaooqinn commented on a change in pull request #32600: URL: https://github.com/apache/spark/pull/32600#discussion_r635884003 ## File path: core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala ## @@ -104,7 +104,7 @@ private[spark] class TypedConfigBuilder[T]

[GitHub] [spark] maropu commented on pull request #32603: [SPARK-35457][BUILD] Bump ANTLR runtime version to 4.8

maropu commented on pull request #32603: URL: https://github.com/apache/spark/pull/32603#issuecomment-844855140 Thanks! Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [spark] maropu closed pull request #32603: [SPARK-35457][BUILD] Bump ANTLR runtime version to 4.8

maropu closed pull request #32603: URL: https://github.com/apache/spark/pull/32603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [spark] maropu commented on a change in pull request #32600: [SPARK-35456][CORE] Print the invalid value in config validation error message

maropu commented on a change in pull request #32600: URL: https://github.com/apache/spark/pull/32600#discussion_r635882306 ## File path: core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala ## @@ -104,7 +104,7 @@ private[spark] class TypedConfigBuilder[T](

[GitHub] [spark] SparkQA commented on pull request #32602: [SPARK-35455][SQL] Enhance EliminateUnnecessaryJoin

SparkQA commented on pull request #32602: URL: https://github.com/apache/spark/pull/32602#issuecomment-844846689 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43252/ -- This is an automated message from the A

<    2   3   4   5   6   7   8   >