[GitHub] [spark] cloud-fan commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command

cloud-fan commented on a change in pull request #34754: URL: https://github.com/apache/spark/pull/34754#discussion_r759935005 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -586,19 +586,13 @@ final class DataFrameWriter[T]

[GitHub] [spark] SparkQA commented on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

SparkQA commented on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983382125 **[Test build #145800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145800/testReport)** for PR 34757 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #34667: [SPARK-36902][SQL] Migrate CreateTableAsSelectStatement to v2 command

cloud-fan commented on a change in pull request #34667: URL: https://github.com/apache/spark/pull/34667#discussion_r759932599 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -170,16 +170,17 @@ class

[GitHub] [spark] cloud-fan commented on a change in pull request #34667: [SPARK-36902][SQL] Migrate CreateTableAsSelectStatement to v2 command

cloud-fan commented on a change in pull request #34667: URL: https://github.com/apache/spark/pull/34667#discussion_r759931289 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -323,18 +323,25 @@ final class DataFrameWriter[T]

[GitHub] [spark] cloud-fan commented on a change in pull request #34667: [SPARK-36902][SQL] Migrate CreateTableAsSelectStatement to v2 command

cloud-fan commented on a change in pull request #34667: URL: https://github.com/apache/spark/pull/34667#discussion_r759930581 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ## @@ -323,18 +323,25 @@ final class DataFrameWriter[T]

[GitHub] [spark] Peng-Lei commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

Peng-Lei commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759926219 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala ## @@ -271,6 +271,14 @@ object CatalogUtils

[GitHub] [spark] Peng-Lei commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

Peng-Lei commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759926219 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala ## @@ -271,6 +271,14 @@ object CatalogUtils

[GitHub] [spark] SparkQA commented on pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

SparkQA commented on pull request #34212: URL: https://github.com/apache/spark/pull/34212#issuecomment-983373300 **[Test build #145803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145803/testReport)** for PR 34212 at commit

[GitHub] [spark] dchvn commented on pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

dchvn commented on pull request #34212: URL: https://github.com/apache/spark/pull/34212#issuecomment-983371360 Jenkins, retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

AmplabJenkins removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983370994 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50267/

[GitHub] [spark] cloud-fan commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

cloud-fan commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759923582 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala ## @@ -271,6 +271,14 @@ object CatalogUtils

[GitHub] [spark] AmplabJenkins commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

AmplabJenkins commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983370994 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50267/ --

[GitHub] [spark] SparkQA commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983370956 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50267/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

SparkQA commented on pull request #34758: URL: https://github.com/apache/spark/pull/34758#issuecomment-983370498 **[Test build #145802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145802/testReport)** for PR 34758 at commit

[GitHub] [spark] Peng-Lei commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

Peng-Lei commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759922889 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -1180,7 +1182,7 @@ class DataSourceV2SQLSuite

[GitHub] [spark] wForget commented on pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

wForget commented on pull request #34489: URL: https://github.com/apache/spark/pull/34489#issuecomment-983370521 > Could you put your findings into the PR description? > > * [[SPARK-37210][SQL] Write to static partition in dynamic write mode  #34489

[GitHub] [spark] SparkQA commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983369353 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50268/ -- This is an automated message from the Apache

[GitHub] [spark] dongjoon-hyun commented on pull request #34264: [SPARK-36462][K8S] Add the ability to selectively disable watching or polling

dongjoon-hyun commented on pull request #34264: URL: https://github.com/apache/spark/pull/34264#issuecomment-983368591 Could you rebase this PR please, @holdenk ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] cloud-fan commented on a change in pull request #34753: [SPARK-37494][SQL] Unify v1 and v2 options output of `SHOW CREATE TABLE` command

cloud-fan commented on a change in pull request #34753: URL: https://github.com/apache/spark/pull/34753#discussion_r759921072 ## File path: sql/core/src/test/scala/org/apache/spark/sql/ShowCreateTableSuite.scala ## @@ -200,6 +200,28 @@ abstract class ShowCreateTableSuite

[GitHub] [spark] wForget commented on a change in pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

wForget commented on a change in pull request #34489: URL: https://github.com/apache/spark/pull/34489#discussion_r759919910 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHiveSingleton.scala ## @@ -27,9 +27,9 @@ import

[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests

cloud-fan commented on a change in pull request #34719: URL: https://github.com/apache/spark/pull/34719#discussion_r759919791 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala ## @@ -0,0 +1,140 @@ +/* + * Licensed to

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

dongjoon-hyun commented on a change in pull request #34489: URL: https://github.com/apache/spark/pull/34489#discussion_r759919225 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSuite.scala ## @@ -168,4 +170,23 @@ class HiveParquetSuite extends

[GitHub] [spark] dongjoon-hyun commented on pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

dongjoon-hyun commented on pull request #34489: URL: https://github.com/apache/spark/pull/34489#issuecomment-983366425 Could you put your findings into the PR description? - https://github.com/apache/spark/pull/34489#issuecomment-967199860 -

[GitHub] [spark] cloud-fan commented on a change in pull request #34718: [SPARK-37460][DOCS] Add the description of ALTER DATABASE SET LOCATION

cloud-fan commented on a change in pull request #34718: URL: https://github.com/apache/spark/pull/34718#discussion_r759919009 ## File path: docs/sql-ref-syntax-ddl-alter-database.md ## @@ -21,25 +21,47 @@ license: | ### Description -You can alter metadata associated with

[GitHub] [spark] cloud-fan commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

cloud-fan commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759918277 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -1180,7 +1182,7 @@ class DataSourceV2SQLSuite

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

dongjoon-hyun commented on a change in pull request #34489: URL: https://github.com/apache/spark/pull/34489#discussion_r759917788 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSuite.scala ## @@ -168,4 +172,30 @@ class HiveParquetSuite extends

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

dongjoon-hyun commented on a change in pull request #34489: URL: https://github.com/apache/spark/pull/34489#discussion_r759917283 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHiveSingleton.scala ## @@ -27,9 +27,9 @@ import

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34489: [SPARK-37210][SQL] Write to static partition in dynamic write mode

dongjoon-hyun commented on a change in pull request #34489: URL: https://github.com/apache/spark/pull/34489#discussion_r759917283 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHiveSingleton.scala ## @@ -27,9 +27,9 @@ import

[GitHub] [spark] dongjoon-hyun commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

dongjoon-hyun commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-983363748 Hi, All. Thank you for making this PR and #34367 . - This PR got +1 already and @zhengruifeng mentioned that `This should be faster than SPARK-37099` before. -

[GitHub] [spark] SparkQA commented on pull request #34673: [SPARK-37343][SQL] Implement createIndex, IndexExists and dropIndex in JDBC (Postgres dialect)

SparkQA commented on pull request #34673: URL: https://github.com/apache/spark/pull/34673#issuecomment-983363812 **[Test build #145801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145801/testReport)** for PR 34673 at commit

[GitHub] [spark] SparkQA commented on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

SparkQA commented on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983363743 **[Test build #145800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145800/testReport)** for PR 34757 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

AmplabJenkins removed a comment on pull request #34212: URL: https://github.com/apache/spark/pull/34212#issuecomment-983362682 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50266/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

AmplabJenkins removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983362685 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145797/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

AmplabJenkins removed a comment on pull request #34766: URL: https://github.com/apache/spark/pull/34766#issuecomment-983362686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145799/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

AmplabJenkins removed a comment on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983362684 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50264/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

AmplabJenkins removed a comment on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-983362683 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50265/

[GitHub] [spark] AmplabJenkins commented on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

AmplabJenkins commented on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983362684 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50264/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

AmplabJenkins commented on pull request #34766: URL: https://github.com/apache/spark/pull/34766#issuecomment-983362686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145799/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

AmplabJenkins commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983362685 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145797/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

AmplabJenkins commented on pull request #34212: URL: https://github.com/apache/spark/pull/34212#issuecomment-983362682 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50266/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

AmplabJenkins commented on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-983362683 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50265/ --

[GitHub] [spark] SparkQA commented on pull request #34764: [SPARK-37330][SQL] Migrate ReplaceTableStatement to v2 command

SparkQA commented on pull request #34764: URL: https://github.com/apache/spark/pull/34764#issuecomment-983360015 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50270/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34763: [SPARK-37389][SQL][FOLLOWUP] SET command shuold not parse comments

SparkQA commented on pull request #34763: URL: https://github.com/apache/spark/pull/34763#issuecomment-983359687 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50271/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34765: [WIP][SPARK-37487][SQL][CORE] Avoid performing CollectMetrics twice if the operation is followed by global sort.

SparkQA commented on pull request #34765: URL: https://github.com/apache/spark/pull/34765#issuecomment-983358731 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50269/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

SparkQA commented on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-983358111 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50265/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34599: [SPARK-37331][K8S] Add the ability to create resources before driverPod creating

dongjoon-hyun commented on a change in pull request #34599: URL: https://github.com/apache/spark/pull/34599#discussion_r759910290 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala ## @@ -142,15

[GitHub] [spark] MaxGekk closed pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

MaxGekk closed pull request #34596: URL: https://github.com/apache/spark/pull/34596 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] mridulm commented on pull request #33896: [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle

mridulm commented on pull request #33896: URL: https://github.com/apache/spark/pull/33896#issuecomment-983356500 It is looking mostly good @venkata91. I would also like @Ngone51 to take a pass through it. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] mridulm commented on a change in pull request #33896: [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle

mridulm commented on a change in pull request #33896: URL: https://github.com/apache/spark/pull/33896#discussion_r759892630 ## File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala ## @@ -53,14 +54,18 @@ private[spark] class ShuffleBlockPusher(conf:

[GitHub] [spark] MaxGekk commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

MaxGekk commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-983356048 +1, LGTM. Merging to master. Thank you, @sadikovi and @gengliangwang @HyukjinKwon @cloud-fan for review. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] mridulm commented on a change in pull request #33896: [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle

mridulm commented on a change in pull request #33896: URL: https://github.com/apache/spark/pull/33896#discussion_r759905638 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -3847,6 +3887,76 @@ class DAGSchedulerSuite extends

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34599: [SPARK-37331][K8S] Add the ability to create resources before driverPod creating

dongjoon-hyun commented on a change in pull request #34599: URL: https://github.com/apache/spark/pull/34599#discussion_r759909487 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala ## @@ -70,7

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34599: [SPARK-37331][K8S] Add the ability to create resources before driverPod creating

dongjoon-hyun commented on a change in pull request #34599: URL: https://github.com/apache/spark/pull/34599#discussion_r759909040 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala ## @@ -381,4 +382,21 @@ object

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34599: [SPARK-37331][K8S] Add the ability to create resources before driverPod creating

dongjoon-hyun commented on a change in pull request #34599: URL: https://github.com/apache/spark/pull/34599#discussion_r759908300 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala ## @@ -381,4 +382,21 @@ object

[GitHub] [spark] MaxGekk commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

MaxGekk commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r759908169 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ## @@ -1012,6 +1012,196 @@ abstract class CSVSuite

[GitHub] [spark] sarutak commented on pull request #34765: [SPARK-37487][SQL][CORE] Avoid performing CollectMetrics twice if the operation is followed by global sort.

sarutak commented on pull request #34765: URL: https://github.com/apache/spark/pull/34765#issuecomment-983352196 > Does this bug also impact the metrics reported by other nodes? For example numOutputRows in FilterExec. Seems so. We need a more comprehensive solution... -- This is

[GitHub] [spark] SparkQA removed a comment on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983329673 **[Test build #145797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145797/testReport)** for PR 34657 at commit

[GitHub] [spark] SparkQA commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983350947 **[Test build #145797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145797/testReport)** for PR 34657 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #34599: [SPARK-37331][K8S] Add the ability to create resources before driverPod creating

dongjoon-hyun commented on pull request #34599: URL: https://github.com/apache/spark/pull/34599#issuecomment-983350835 Sorry for being late. I'll review now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] SparkQA commented on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

SparkQA commented on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983348737 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50264/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

SparkQA removed a comment on pull request #34766: URL: https://github.com/apache/spark/pull/34766#issuecomment-983336928 **[Test build #145799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145799/testReport)** for PR 34766 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

cloud-fan commented on a change in pull request #34766: URL: https://github.com/apache/spark/pull/34766#discussion_r759902213 ## File path: docs/sql-migration-guide.md ## @@ -133,6 +133,8 @@ license: | - In Spark 3.2, create/alter view will fail if the input query output

[GitHub] [spark] SparkQA commented on pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

SparkQA commented on pull request #34766: URL: https://github.com/apache/spark/pull/34766#issuecomment-983344028 **[Test build #145799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145799/testReport)** for PR 34766 at commit

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34706: [SPARK-37458][SS] Remove unnecessary SerializeFromObject from the plan of foreachBatch

dongjoon-hyun edited a comment on pull request #34706: URL: https://github.com/apache/spark/pull/34706#issuecomment-983342994 cc @cloud-fan, too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] dongjoon-hyun commented on pull request #34706: [SPARK-37458][SS] Remove unnecessary SerializeFromObject from the plan of foreachBatch

dongjoon-hyun commented on pull request #34706: URL: https://github.com/apache/spark/pull/34706#issuecomment-983342994 cc @cloud-fan and @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] SparkQA commented on pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

SparkQA commented on pull request #34212: URL: https://github.com/apache/spark/pull/34212#issuecomment-983341397 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50266/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on pull request #34734: [SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md

dongjoon-hyun commented on pull request #34734: URL: https://github.com/apache/spark/pull/34734#issuecomment-983340912 Merged to master. If you don't mind, could you make a backporting PR to branch-3.2 after excluding `3.3.0` configs? -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun closed pull request #34734: [SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md

dongjoon-hyun closed pull request #34734: URL: https://github.com/apache/spark/pull/34734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] tanelk commented on pull request #34765: [SPARK-37487][SQL][CORE] Avoid performing CollectMetrics twice if the operation is followed by global sort.

tanelk commented on pull request #34765: URL: https://github.com/apache/spark/pull/34765#issuecomment-983339082 Does this bug also impact the metrics reported by other nodes? For example `numOutputRows` in `FilterExec`. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] advancedxy commented on pull request #34640: [SPARK-31585][SQL] Introduce Z-order expression

advancedxy commented on pull request #34640: URL: https://github.com/apache/spark/pull/34640#issuecomment-983338047 > @advancedxy - we haven't rolled out this feature in our production yet, still testing so there should be quite some room for further optimization. For storage efficiency,

[GitHub] [spark] SparkQA commented on pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

SparkQA commented on pull request #34766: URL: https://github.com/apache/spark/pull/34766#issuecomment-983336928 **[Test build #145799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145799/testReport)** for PR 34766 at commit

[GitHub] [spark] beliefer opened a new pull request #34766: [SPARK-37513][SQL][DOC] Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

beliefer opened a new pull request #34766: URL: https://github.com/apache/spark/pull/34766 ### What changes were proposed in this pull request? The SQL show below previously returned the date type, now it returns the timestamp type. `select date '2011-11-11' + interval 12 hours;`

[GitHub] [spark] advancedxy commented on a change in pull request #34640: [SPARK-31585][SQL] Introduce Z-order expression

advancedxy commented on a change in pull request #34640: URL: https://github.com/apache/spark/pull/34640#discussion_r759890419 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ZOrder.scala ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache

[GitHub] [spark] SparkQA commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-98827 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50267/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34765: [SPARK-37487][SQL][CORE] Avoid performing CollectMetrics twice if the operation is followed by global sort.

SparkQA commented on pull request #34765: URL: https://github.com/apache/spark/pull/34765#issuecomment-983331523 **[Test build #145798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145798/testReport)** for PR 34765 at commit

[GitHub] [spark] cloud-fan commented on pull request #34763: [SPARK-37389][SQL][FOLLOWUP] SET command shuold not parse comments

cloud-fan commented on pull request #34763: URL: https://github.com/apache/spark/pull/34763#issuecomment-983329910 @beliefer @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983329673 **[Test build #145797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145797/testReport)** for PR 34657 at commit

[GitHub] [spark] sarutak opened a new pull request #34765: [SPARK-37487][SQL][CORE] Avoid performing CollectMetrics twice if the operation is folloed by global sort.

sarutak opened a new pull request #34765: URL: https://github.com/apache/spark/pull/34765 ### What changes were proposed in this pull request? This PR fixes an issue that `CollectMetrics` performs twice if it's followed by global sort like as follows. ``` val df =

[GitHub] [spark] SparkQA commented on pull request #34764: [SPARK-37330][SQL] Migrate ReplaceTableStatement to v2 command

SparkQA commented on pull request #34764: URL: https://github.com/apache/spark/pull/34764#issuecomment-983329462 **[Test build #145795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145795/testReport)** for PR 34764 at commit

[GitHub] [spark] SparkQA commented on pull request #34763: [SPARK-37389][SQL][FOLLOWUP] SET command shuold not parse comments

SparkQA commented on pull request #34763: URL: https://github.com/apache/spark/pull/34763#issuecomment-983329503 **[Test build #145796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145796/testReport)** for PR 34763 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34363: [SPARK-37083][PYTHON] Inline type hints for python/pyspark/accumulators.py

AmplabJenkins commented on pull request #34363: URL: https://github.com/apache/spark/pull/34363#issuecomment-983328712 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50262/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34363: [SPARK-37083][PYTHON] Inline type hints for python/pyspark/accumulators.py

AmplabJenkins removed a comment on pull request #34363: URL: https://github.com/apache/spark/pull/34363#issuecomment-983328712 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50262/

[GitHub] [spark] dchvn opened a new pull request #34764: [SPARK-37330][SQL] Migrate ReplaceTableStatement to v2 command

dchvn opened a new pull request #34764: URL: https://github.com/apache/spark/pull/34764 ### What changes were proposed in this pull request? This PR migrates ReplaceTableStatement to the v2 command ### Why are the changes needed? Migrate to the standard V2 framework ###

[GitHub] [spark] AmplabJenkins commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

AmplabJenkins commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983328707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

AmplabJenkins removed a comment on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983328711 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50261/

[GitHub] [spark] Peng-Lei commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

Peng-Lei commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759885112 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala ## @@ -42,7 +44,12 @@ case class

[GitHub] [spark] Peng-Lei commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

Peng-Lei commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759884718 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -1180,7 +1182,7 @@ class DataSourceV2SQLSuite

[GitHub] [spark] Peng-Lei commented on a change in pull request #34758: [SPARK-37501][SQL] CREATE/REPLACE TABLE should qualify location for v2 command

Peng-Lei commented on a change in pull request #34758: URL: https://github.com/apache/spark/pull/34758#discussion_r759884718 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -1180,7 +1182,7 @@ class DataSourceV2SQLSuite

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-983328709 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50263/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

AmplabJenkins removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983328707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

AmplabJenkins commented on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983328711 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50261/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-983328709 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50263/

[GitHub] [spark] cloud-fan opened a new pull request #34763: [SPARK-37389][SQL][FOLLOWUP] SET command shuold not parse comments

cloud-fan opened a new pull request #34763: URL: https://github.com/apache/spark/pull/34763 ### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/34668 , to fix a breaking change. The SET command uses wildcard

[GitHub] [spark] SparkQA commented on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

SparkQA commented on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-983325259 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50265/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

SparkQA commented on pull request #34212: URL: https://github.com/apache/spark/pull/34212#issuecomment-983320773 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50266/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34757: [SPARK-37504][PYTHON] Pyspark create SparkSession with existed session should not pass static conf

SparkQA commented on pull request #34757: URL: https://github.com/apache/spark/pull/34757#issuecomment-983321062 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50264/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983319286 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50260/ -- This is an automated message from the

[GitHub] [spark] wang-zhun closed pull request #34072: [SPARK-36680][SQL] Supports Dynamic Table Options for Spark SQL

wang-zhun closed pull request #34072: URL: https://github.com/apache/spark/pull/34072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] wang-zhun commented on pull request #34072: [SPARK-36680][SQL] Supports Dynamic Table Options for Spark SQL

wang-zhun commented on pull request #34072: URL: https://github.com/apache/spark/pull/34072#issuecomment-983318289 Close this PR first and look forward to a better proposal implementation -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] SparkQA removed a comment on pull request #34657: [SPARK-37511][PYTHON] Introduce TimedeltaIndex to pandas API on Spark

SparkQA removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-983302382 **[Test build #145794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145794/testReport)** for PR 34657 at commit

  1   2   3   4   5   6   >