[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

shahidki31 commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642307807 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] yaooqinn commented on pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

yaooqinn commented on pull request #32708: URL: https://github.com/apache/spark/pull/32708#issuecomment-851308784 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] viirya commented on pull request #32703: [SPARK-35566][SS] Fix StateStoreRestoreExec output rows

viirya commented on pull request #32703: URL: https://github.com/apache/spark/pull/32703#issuecomment-851308597 Thanks all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32658: URL: https://github.com/apache/spark/pull/32658#discussion_r642306485 ## File path: docs/sql-data-sources-csv.md ## @@ -38,3 +36,217 @@ Spark SQL provides `spark.read().csv("file_name")` to read a file or directory o

[GitHub] [spark] yaooqinn commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

yaooqinn commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642306212 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala ## @@ -45,10 +45,15

[GitHub] [spark] yaooqinn commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

yaooqinn commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642305798 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStepSuite.scala ## @@ -370,6

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32658: URL: https://github.com/apache/spark/pull/32658#discussion_r642305722 ## File path: docs/sql-data-sources-csv.md ## @@ -38,3 +36,217 @@ Spark SQL provides `spark.read().csv("file_name")` to read a file or directory o

[GitHub] [spark] yaooqinn commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

yaooqinn commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642305041 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStepSuite.scala ## @@ -370,7

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32658: URL: https://github.com/apache/spark/pull/32658#discussion_r642305065 ## File path: docs/sql-data-sources-csv.md ## @@ -38,3 +36,217 @@ Spark SQL provides `spark.read().csv("file_name")` to read a file or directory o

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32658: URL: https://github.com/apache/spark/pull/32658#discussion_r642302955 ## File path: docs/sql-data-sources-csv.md ## @@ -38,3 +36,217 @@ Spark SQL provides `spark.read().csv("file_name")` to read a file or directory o

[GitHub] [spark] SparkQA removed a comment on pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

SparkQA removed a comment on pull request #32658: URL: https://github.com/apache/spark/pull/32658#issuecomment-851151414 **[Test build #139093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139093/testReport)** for PR 32658 at commit

[GitHub] [spark] SparkQA commented on pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

SparkQA commented on pull request #32658: URL: https://github.com/apache/spark/pull/32658#issuecomment-851299022 **[Test build #139093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139093/testReport)** for PR 32658 at commit

[GitHub] [spark] yaooqinn commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

yaooqinn commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642292225 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala ## @@ -45,10 +45,15

[GitHub] [spark] SparkQA removed a comment on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

SparkQA removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851294123 **[Test build #139101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139101/testReport)** for PR 32513 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

AmplabJenkins removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851296997 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139101/

[GitHub] [spark] AmplabJenkins commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

AmplabJenkins commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851296997 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139101/ -- This

[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851296962 **[Test build #139101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139101/testReport)** for PR 32513 at commit

[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851294123 **[Test build #139101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139101/testReport)** for PR 32513 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

SparkQA removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851290880 **[Test build #139100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139100/testReport)** for PR 32513 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

AmplabJenkins removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851293206 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139100/

[GitHub] [spark] AmplabJenkins commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

AmplabJenkins commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851293206 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139100/ -- This

[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851293164 **[Test build #139100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139100/testReport)** for PR 32513 at commit

[GitHub] [spark] zwangsheng commented on pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

zwangsheng commented on pull request #32708: URL: https://github.com/apache/spark/pull/32708#issuecomment-851292043 this SPARK_EXECUTOR_POD_IP_OVERWRITE has nothing to do with this PR, it will be removed in this PR first -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Eagerly execute non-root Command

SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-851290880 **[Test build #139100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139100/testReport)** for PR 32513 at commit

[GitHub] [spark] yaooqinn commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

yaooqinn commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642292225 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala ## @@ -45,10 +45,15

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32658: [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page

HyukjinKwon commented on a change in pull request #32658: URL: https://github.com/apache/spark/pull/32658#discussion_r642291885 ## File path: docs/sql-data-sources-csv.md ## @@ -38,3 +36,217 @@ Spark SQL provides `spark.read().csv("file_name")` to read a file or directory o

[GitHub] [spark] zwangsheng commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

zwangsheng commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642291731 ## File path: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh ## @@ -67,6 +67,14 @@ elif ! [ -z ${SPARK_HOME+x} ]; then

[GitHub] [spark] dongjoon-hyun commented on pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

dongjoon-hyun commented on pull request #32709: URL: https://github.com/apache/spark/pull/32709#issuecomment-851288457 Thanks, @HyukjinKwon ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

dongjoon-hyun commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642290289 ## File path: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh ## @@ -67,6 +67,14 @@ elif ! [ -z ${SPARK_HOME+x} ];

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

dongjoon-hyun commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642290289 ## File path: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh ## @@ -67,6 +67,14 @@ elif ! [ -z ${SPARK_HOME+x} ];

[GitHub] [spark] SparkQA commented on pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

SparkQA commented on pull request #32709: URL: https://github.com/apache/spark/pull/32709#issuecomment-851287479 **[Test build #139099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139099/testReport)** for PR 32709 at commit

[GitHub] [spark] HyukjinKwon removed a comment on pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

HyukjinKwon removed a comment on pull request #32709: URL: https://github.com/apache/spark/pull/32709#issuecomment-851287095 I'll take a look in CI. If it's simple, i will just fix it here. the PR uses my resource in the forked repository anyway :-). -- This is an automated message from

[GitHub] [spark] HyukjinKwon commented on pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

HyukjinKwon commented on pull request #32709: URL: https://github.com/apache/spark/pull/32709#issuecomment-851287531 I'll take a look in CI. If it's simple, i will just fix it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon edited a comment on pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

HyukjinKwon edited a comment on pull request #32709: URL: https://github.com/apache/spark/pull/32709#issuecomment-851287095 I'll take a look in CI. If it's simple, i will just fix it here. the PR uses my resource in the forked repository anyway :-). -- This is an automated message from

[GitHub] [spark] HyukjinKwon commented on pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

HyukjinKwon commented on pull request #32709: URL: https://github.com/apache/spark/pull/32709#issuecomment-851287095 I'll take a look in CI. If it's simple, i will just fix it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon opened a new pull request #32709: [WIP][SPARK-35573][R] Support R 4.1.0 in SparkR

HyukjinKwon opened a new pull request #32709: URL: https://github.com/apache/spark/pull/32709 ### What changes were proposed in this pull request? This PR proposes to support R 4.1.0+ in SparkR. Currently the tests are being failed as below: ``` ══ Failed

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

dongjoon-hyun commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642288879 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStepSuite.scala ## @@

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

dongjoon-hyun commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642288879 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStepSuite.scala ## @@

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

dongjoon-hyun commented on a change in pull request #32708: URL: https://github.com/apache/spark/pull/32708#discussion_r642288323 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala ## @@ -473,6 +473,13 @@ private[spark]

[GitHub] [spark] cloud-fan commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

cloud-fan commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642288216 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

AmplabJenkins removed a comment on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851281858 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139097/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

AmplabJenkins removed a comment on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851281860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139095/

[GitHub] [spark] AmplabJenkins commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

AmplabJenkins commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851281860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139095/ -- This

[GitHub] [spark] SparkQA commented on pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

SparkQA commented on pull request #32558: URL: https://github.com/apache/spark/pull/32558#issuecomment-851281861 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43619/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

AmplabJenkins commented on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851281858 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139097/ -- This

[GitHub] [spark] HyukjinKwon commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

HyukjinKwon commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851281005 I will take that! thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] HyukjinKwon commented on pull request #32703: [SPARK-35566][SS] Fix StateStoreRestoreExec output rows

HyukjinKwon commented on pull request #32703: URL: https://github.com/apache/spark/pull/32703#issuecomment-851280017 Merged to master, branch-3.1 and branch-3.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] HyukjinKwon closed pull request #32703: [SPARK-35566][SS] Fix StateStoreRestoreExec output rows

HyukjinKwon closed pull request #32703: URL: https://github.com/apache/spark/pull/32703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] HyukjinKwon commented on pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

HyukjinKwon commented on pull request #32558: URL: https://github.com/apache/spark/pull/32558#issuecomment-851278594 I think we should also add the parameter at `csv` and `json` at `DataFrameWriter`, `DataFrameReader`, `DataStreamReader`, `DataStreamWriter` (and also `readwriter.py` and

[GitHub] [spark] dongjoon-hyun commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

dongjoon-hyun commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851278632 Thank you, @HyukjinKwon . BTW, for SparkR UT failure on R 4.1.0, I filed SPARK-35573 . -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

HyukjinKwon commented on a change in pull request #32558: URL: https://github.com/apache/spark/pull/32558#discussion_r642281798 ## File path: docs/sql-data-sources-json.md ## @@ -155,6 +155,12 @@ Data source options of JSON can be set via: Allows leading zeros in numbers

[GitHub] [spark] HyukjinKwon commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

HyukjinKwon commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851275244 Thanks @dongjoon-hyun ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

shahidki31 commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642279415 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] SparkQA removed a comment on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

SparkQA removed a comment on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851199336 **[Test build #139097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139097/testReport)** for PR 32693 at commit

[GitHub] [spark] SparkQA commented on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

SparkQA commented on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851273254 **[Test build #139097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139097/testReport)** for PR 32693 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

cloud-fan commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642277656 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

shahidki31 commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642268386 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

shahidki31 commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642265510 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] shahidki31 commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

shahidki31 commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642265510 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] SparkQA removed a comment on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

SparkQA removed a comment on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851169175 **[Test build #139095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139095/testReport)** for PR 32706 at commit

[GitHub] [spark] SparkQA commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

SparkQA commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851253186 **[Test build #139095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139095/testReport)** for PR 32706 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

AmplabJenkins removed a comment on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851249700 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43618/

[GitHub] [spark] AmplabJenkins commented on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

AmplabJenkins commented on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851249700 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43618/ --

[GitHub] [spark] SparkQA commented on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

SparkQA commented on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851249663 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43618/ -- This is an automated message from the

[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

SaurabhChawla100 commented on a change in pull request #32558: URL: https://github.com/apache/spark/pull/32558#discussion_r642259416 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala ## @@ -38,6 +39,13 @@ class CSVInferSchema(val

[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

SaurabhChawla100 commented on a change in pull request #32558: URL: https://github.com/apache/spark/pull/32558#discussion_r642259322 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ## @@ -68,6 +68,8 @@ private[sql] class

[GitHub] [spark] cloud-fan commented on a change in pull request #32699: [SPARK-35560][SQL] Remove redundant subexpression evaluation in nested subexpressions

cloud-fan commented on a change in pull request #32699: URL: https://github.com/apache/spark/pull/32699#discussion_r642258870 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -1068,11 +1072,19 @@ class

[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

SaurabhChawla100 commented on a change in pull request #32558: URL: https://github.com/apache/spark/pull/32558#discussion_r642258661 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ## @@ -68,6 +68,8 @@ private[sql] class

[GitHub] [spark] cloud-fan commented on a change in pull request #32699: [SPARK-35560][SQL] Remove redundant subexpression evaluation in nested subexpressions

cloud-fan commented on a change in pull request #32699: URL: https://github.com/apache/spark/pull/32699#discussion_r642258116 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ## @@ -1039,21 +1039,25 @@ class

[GitHub] [spark] SparkQA commented on pull request #32558: [SPARK-34953][CORE][SQL] Add the code change for adding the DateType in the infer schema while reading in CSV and JSON

SparkQA commented on pull request #32558: URL: https://github.com/apache/spark/pull/32558#issuecomment-851247187 **[Test build #139098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139098/testReport)** for PR 32558 at commit

[GitHub] [spark] cloud-fan commented on pull request #32675: [SPARK-35531][SQL] Can not insert into hive bucket table if create table with upper case schema

cloud-fan commented on pull request #32675: URL: https://github.com/apache/spark/pull/32675#issuecomment-851243531 > the table schema will be stored as lower cases while bucket column info will stay the same with user input. I'm not sure this is true. Table schema and bucketed

[GitHub] [spark] cloud-fan commented on a change in pull request #32675: [SPARK-35531][SQL] Can not insert into hive bucket table if create table with upper case schema

cloud-fan commented on a change in pull request #32675: URL: https://github.com/apache/spark/pull/32675#discussion_r642254313 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ## @@ -1092,14 +1092,23 @@ private[hive] object

[GitHub] [spark] AmplabJenkins commented on pull request #32708: [WIP] [SPARK-35572] [K8S] add hostNetwork feature to executor

AmplabJenkins commented on pull request #32708: URL: https://github.com/apache/spark/pull/32708#issuecomment-851238472 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32686: [WIP][SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules

AmplabJenkins removed a comment on pull request #32686: URL: https://github.com/apache/spark/pull/32686#issuecomment-851237182 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43617/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

AmplabJenkins removed a comment on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851237180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32707: [SPARK-31168][BUILD][FOLLOWUP] Update scala-2.12 profile

AmplabJenkins removed a comment on pull request #32707: URL: https://github.com/apache/spark/pull/32707#issuecomment-851237179 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139091/

[GitHub] [spark] dongjoon-hyun commented on pull request #32697: [SPARK-31168][BUILD] Upgrade Scala to 2.12.14

dongjoon-hyun commented on pull request #32697: URL: https://github.com/apache/spark/pull/32697#issuecomment-851237605 Sure, I agree with you that we need to remove it in Apache Spark 3.2.0 timeframe. Thanks, @iemejia . > My suggestion to remove silencer was to unblock the issue but

[GitHub] [spark] AmplabJenkins commented on pull request #32686: [WIP][SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules

AmplabJenkins commented on pull request #32686: URL: https://github.com/apache/spark/pull/32686#issuecomment-851237182 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43617/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32707: [SPARK-31168][BUILD][FOLLOWUP] Update scala-2.12 profile

AmplabJenkins commented on pull request #32707: URL: https://github.com/apache/spark/pull/32707#issuecomment-851237179 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139091/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

AmplabJenkins commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851237180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] JkSelf commented on a change in pull request #32705: [SPARK-35568][SQL] Avoid UnsupportedOperationException when enabling both AQE and DPP

JkSelf commented on a change in pull request #32705: URL: https://github.com/apache/spark/pull/32705#discussion_r642249067 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala ## @@ -312,7 +312,15 @@ case class

[GitHub] [spark] iemejia commented on pull request #32697: [SPARK-31168][BUILD] Upgrade Scala to 2.12.14

iemejia commented on pull request #32697: URL: https://github.com/apache/spark/pull/32697#issuecomment-851235328 Excellent @dongjoon-hyun! My suggestion to remove silencer was to unblock the issue but this can be definitely be done afterwards. Great to see the new version of Scala just

[GitHub] [spark] chong0929 commented on pull request #32202: [SPARK-28098][SQL]Supporting non-partitioned Hive tables with subdirectories

chong0929 commented on pull request #32202: URL: https://github.com/apache/spark/pull/32202#issuecomment-851229533 > > > I found the same problem with partition Hive tables if they contain subdirectories, so why wasn't it changed in this action? > > > > > > you mean it will hit

[GitHub] [spark] cloud-fan commented on a change in pull request #32704: [SPARK-35567][SQL] Fix: Explain cost is not showing statistics for all the nodes

cloud-fan commented on a change in pull request #32704: URL: https://github.com/apache/spark/pull/32704#discussion_r642244548 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -256,13 +255,9 @@ class QueryExecution( //

[GitHub] [spark] SparkQA commented on pull request #32693: [SPARK-35556][SQL][TESTS] Avoid log NoSuchMethodError when HiveClientImpl.state close

SparkQA commented on pull request #32693: URL: https://github.com/apache/spark/pull/32693#issuecomment-851227301 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43618/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #32686: [WIP][SPARK-35544][SQL] Add tree pattern pruning to Analyzer rules

SparkQA commented on pull request #32686: URL: https://github.com/apache/spark/pull/32686#issuecomment-851222124 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43617/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

SparkQA removed a comment on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851149438 **[Test build #139092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139092/testReport)** for PR 32706 at commit

[GitHub] [spark] SparkQA commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

SparkQA commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851214838 **[Test build #139092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139092/testReport)** for PR 32706 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #32707: [SPARK-31168][BUILD][FOLLOWUP] Update scala-2.12 profile

SparkQA removed a comment on pull request #32707: URL: https://github.com/apache/spark/pull/32707#issuecomment-851149411 **[Test build #139091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139091/testReport)** for PR 32707 at commit

[GitHub] [spark] SparkQA commented on pull request #32706: [SPARK-35507][INFRA] Add Python 3.9 in the docker image for GitHub Action

SparkQA commented on pull request #32706: URL: https://github.com/apache/spark/pull/32706#issuecomment-851211467 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43616/ -- This is an automated message from the

[GitHub] [spark] zwangsheng commented on pull request #32708: SPARK-35572 add hostNetwork feature to executor

zwangsheng commented on pull request #32708: URL: https://github.com/apache/spark/pull/32708#issuecomment-851211308 cc @yaooqinn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #32707: [SPARK-31168][BUILD][FOLLOWUP] Update scala-2.12 profile

SparkQA commented on pull request #32707: URL: https://github.com/apache/spark/pull/32707#issuecomment-851210801 **[Test build #139091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139091/testReport)** for PR 32707 at commit

[GitHub] [spark] zwangsheng opened a new pull request #32708: SPARK-35572 add hostNetwork feature to executor

zwangsheng opened a new pull request #32708: URL: https://github.com/apache/spark/pull/32708 ### What changes were proposed in this pull request? add hostNetwork feature to executor modify BasicExecutorFeatureStep add hostNetwork in executorpod ### Why are the changes needed?

[GitHub] [spark] huskysun commented on a change in pull request #32701: [SPARK-35562][DOC] Fix docs about Kubernetes and Yarn

huskysun commented on a change in pull request #32701: URL: https://github.com/apache/spark/pull/32701#discussion_r642231488 ## File path: docs/submitting-applications.md ## @@ -114,12 +114,12 @@ run it with `--help`. Here are a few examples of common options:

[GitHub] [spark] huskysun commented on a change in pull request #32701: [SPARK-35562][DOC] Fix docs about Kubernetes

huskysun commented on a change in pull request #32701: URL: https://github.com/apache/spark/pull/32701#discussion_r642229347 ## File path: docs/submitting-applications.md ## @@ -114,12 +114,12 @@ run it with `--help`. Here are a few examples of common options:

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32701: [SPARK-35562][DOC] Fix docs about Kubernetes

dongjoon-hyun commented on a change in pull request #32701: URL: https://github.com/apache/spark/pull/32701#discussion_r642229067 ## File path: docs/submitting-applications.md ## @@ -114,12 +114,12 @@ run it with `--help`. Here are a few examples of common options:

<    1   2   3   4   5