[GitHub] [spark] SparkQA commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982372984 **[Test build #145756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145756/testReport)** for PR 34750 at commit

[GitHub] [spark] SparkQA commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982370952 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50221/ -- This is an automated message from the

[GitHub] [spark] beliefer opened a new pull request #31847: [SPARK-34755][SQL] Support the utils for transform number format

2021-11-29 Thread GitBox
beliefer opened a new pull request #31847: URL: https://github.com/apache/spark/pull/31847 ### What changes were proposed in this pull request? Data Type Formatting Functions: `to_number` and `to_char` is very useful. There are some mainstream database support the syntax.

[GitHub] [spark] SparkQA removed a comment on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982354267 **[Test build #145755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145755/testReport)** for PR 34751 at commit

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982364378 **[Test build #145755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145755/testReport)** for PR 34751 at commit

[GitHub] [spark] yangwwei edited a comment on pull request #34672: [SPARK-37394][CORE] Skip registering with ESS if a customized shuffle manager is configured

2021-11-29 Thread GitBox
yangwwei edited a comment on pull request #34672: URL: https://github.com/apache/spark/pull/34672#issuecomment-982363102 @mridulm , @attilapiros , @tgravescs could you pls help to review the changes again? Per @attilapiros 's suggestion, I have added a method in the ShuffleManager

[GitHub] [spark] yangwwei commented on pull request #34672: [SPARK-37394][CORE] Skip registering with ESS if a customized shuffle manager is configured

2021-11-29 Thread GitBox
yangwwei commented on pull request #34672: URL: https://github.com/apache/spark/pull/34672#issuecomment-982363102 @mridulm , @attilapiros , @tgravescs could you pls help to review the changes again? Per @attilapiros 's suggestion, I have added a method in the ShuffleManager trait and

[GitHub] [spark] guiyanakuang commented on a change in pull request #34743: [SPARK-37488][CORE] When `TaskLocation` is `HDFSCacheTaskLocation` or `HostTaskLocation`, check if executor is alive on the

2021-11-29 Thread GitBox
guiyanakuang commented on a change in pull request #34743: URL: https://github.com/apache/spark/pull/34743#discussion_r759001312 ## File path: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ## @@ -291,6 +291,21 @@ class TaskSetManagerSuite

[GitHub] [spark] sadikovi commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-29 Thread GitBox
sadikovi commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r75908 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala ## @@ -160,6 +169,17 @@ class CSVInferSchema(val

[GitHub] [spark] SparkQA commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-29 Thread GitBox
SparkQA commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-982357268 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50225/ -- This is an automated message from the Apache

[GitHub] [spark] LuciferYang commented on a change in pull request #34743: [SPARK-37488][CORE] When `TaskLocation` is `HDFSCacheTaskLocation` or `HostTaskLocation`, check if executor is alive on the h

2021-11-29 Thread GitBox
LuciferYang commented on a change in pull request #34743: URL: https://github.com/apache/spark/pull/34743#discussion_r758996701 ## File path: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ## @@ -291,6 +291,21 @@ class TaskSetManagerSuite

[GitHub] [spark] SparkQA commented on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluster environmen

2021-11-29 Thread GitBox
SparkQA commented on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982356662 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50223/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34568: [SPARK-37287][SQL] Pull out dynamic partition and bucket sort from FileFormatWriter

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34568: URL: https://github.com/apache/spark/pull/34568#issuecomment-982354923 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145740/

[GitHub] [spark] summaryzb commented on a change in pull request #34749: [SPARK-37493][CORE] show driver's gc time and duration time in executors page

2021-11-29 Thread GitBox
summaryzb commented on a change in pull request #34749: URL: https://github.com/apache/spark/pull/34749#discussion_r758995760 ## File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ## @@ -89,7 +89,44 @@ private[spark] class AppStatusStore( } else {

[GitHub] [spark] SparkQA commented on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluster environmen

2021-11-29 Thread GitBox
SparkQA commented on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982355152 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50224/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #34568: [SPARK-37287][SQL] Pull out dynamic partition and bucket sort from FileFormatWriter

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34568: URL: https://github.com/apache/spark/pull/34568#issuecomment-982354923 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145740/ -- This

[GitHub] [spark] SparkQA commented on pull request #34753: [SPARK-37494][SQL] Unify v1 and v2 options output of `SHOW CREATE TABLE` command

2021-11-29 Thread GitBox
SparkQA commented on pull request #34753: URL: https://github.com/apache/spark/pull/34753#issuecomment-982354219 **[Test build #145754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145754/testReport)** for PR 34753 at commit

[GitHub] [spark] SparkQA commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982354345 **[Test build #145756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145756/testReport)** for PR 34750 at commit

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982354267 **[Test build #145755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145755/testReport)** for PR 34751 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34568: [SPARK-37287][SQL] Pull out dynamic partition and bucket sort from FileFormatWriter

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34568: URL: https://github.com/apache/spark/pull/34568#issuecomment-982226440 **[Test build #145740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145740/testReport)** for PR 34568 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34752: [SPARK][STREAMING] minRatePerPartition should be multiplied with secsPerBatch

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34752: URL: https://github.com/apache/spark/pull/34752#issuecomment-982353991 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] SparkQA commented on pull request #34568: [SPARK-37287][SQL] Pull out dynamic partition and bucket sort from FileFormatWriter

2021-11-29 Thread GitBox
SparkQA commented on pull request #34568: URL: https://github.com/apache/spark/pull/34568#issuecomment-982353646 **[Test build #145740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145740/testReport)** for PR 34568 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982353320 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145750/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34723: [SPARK-37492][SQL] Optimize Orc test code with withAllNativeOrcReaders

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34723: URL: https://github.com/apache/spark/pull/34723#issuecomment-982353318 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145738/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-clus

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982353319 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145752/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34213: URL: https://github.com/apache/spark/pull/34213#issuecomment-982353317 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50219/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982353202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluster envi

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982353319 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145752/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34723: [SPARK-37492][SQL] Optimize Orc test code with withAllNativeOrcReaders

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34723: URL: https://github.com/apache/spark/pull/34723#issuecomment-982353318 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145738/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982353203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins commented on pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34213: URL: https://github.com/apache/spark/pull/34213#issuecomment-982353317 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50219/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982353320 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145750/ -- This

[GitHub] [spark] summaryzb commented on a change in pull request #34749: [SPARK-37493][CORE] show driver's gc time and duration time in executors page

2021-11-29 Thread GitBox
summaryzb commented on a change in pull request #34749: URL: https://github.com/apache/spark/pull/34749#discussion_r758993284 ## File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ## @@ -137,7 +138,9 @@ case object GarbageCollectionMetrics extends

[GitHub] [spark] LuciferYang commented on a change in pull request #34745: [WIP][SPARK-37391][SQL] JdbcConnectionProvider must indicate if it needs lock

2021-11-29 Thread GitBox
LuciferYang commented on a change in pull request #34745: URL: https://github.com/apache/spark/pull/34745#discussion_r758992879 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProviderSuite.scala ## @@ -68,12 +69,20

[GitHub] [spark] dchvn commented on pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
dchvn commented on pull request #34213: URL: https://github.com/apache/spark/pull/34213#issuecomment-982350720 Thanks! @itholic @Yikun @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA commented on pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
SparkQA commented on pull request #34213: URL: https://github.com/apache/spark/pull/34213#issuecomment-982349531 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50219/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982348283 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50221/ -- This is an automated message from the Apache

[GitHub] [spark] MaxGekk commented on pull request #31847: [SPARK-34755][SQL] Support the utils for transform number format

2021-11-29 Thread GitBox
MaxGekk commented on pull request #31847: URL: https://github.com/apache/spark/pull/31847#issuecomment-982348272 Since the functions are broadly used in other systems, it makes sense to support them in Spark that can make migration to Spark easier, I believe. @beliefer Could you re-open

[GitHub] [spark] huaxingao commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework

2021-11-29 Thread GitBox
huaxingao commented on pull request #34060: URL: https://github.com/apache/spark/pull/34060#issuecomment-982348104 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] SparkQA removed a comment on pull request #34723: [SPARK-37492][SQL] Optimize Orc test code with withAllNativeOrcReaders

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34723: URL: https://github.com/apache/spark/pull/34723#issuecomment-982219652 **[Test build #145738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145738/testReport)** for PR 34723 at commit

[GitHub] [spark] SparkQA commented on pull request #34723: [SPARK-37492][SQL] Optimize Orc test code with withAllNativeOrcReaders

2021-11-29 Thread GitBox
SparkQA commented on pull request #34723: URL: https://github.com/apache/spark/pull/34723#issuecomment-982344119 **[Test build #145738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145738/testReport)** for PR 34723 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluster en

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982330102 **[Test build #145752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145752/testReport)** for PR 34635 at commit

[GitHub] [spark] SparkQA commented on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluster environmen

2021-11-29 Thread GitBox
SparkQA commented on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982342499 **[Test build #145752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145752/testReport)** for PR 34635 at commit

[GitHub] [spark] Peng-Lei opened a new pull request #34753: [SPARK-37494][SQL] Unify v1 and v2 options output of `SHOW CREATE TABLE` command

2021-11-29 Thread GitBox
Peng-Lei opened a new pull request #34753: URL: https://github.com/apache/spark/pull/34753 ### What changes were proposed in this pull request? 1. Change the v1 `SHOW CREATE TABLE` command behaviors that options output match v2. eg: `'key' = 'value'` 2. sort the order of

[GitHub] [spark] SparkQA removed a comment on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982329931 **[Test build #145751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145751/testReport)** for PR 34751 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982324974 **[Test build #145750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145750/testReport)** for PR 34750 at commit

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-29 Thread GitBox
HyukjinKwon commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r758980984 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala ## @@ -38,6 +39,13 @@ class CSVInferSchema(val

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982338326 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50222/ --

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-29 Thread GitBox
HyukjinKwon commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r758979898 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala ## @@ -160,6 +169,17 @@ class CSVInferSchema(val

[GitHub] [spark] SparkQA commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982337997 **[Test build #145750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145750/testReport)** for PR 34750 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-29 Thread GitBox
dongjoon-hyun commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-982335508 Although it looks good to me, gentle ping once more, @cloud-fan @rdblue @viirya @huaxingao . -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982335210 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50220/ --

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982334949 **[Test build #145751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145751/testReport)** for PR 34751 at commit

[GitHub] [spark] HyukjinKwon closed pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
HyukjinKwon closed pull request #34213: URL: https://github.com/apache/spark/pull/34213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
HyukjinKwon commented on pull request #34213: URL: https://github.com/apache/spark/pull/34213#issuecomment-982334604 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-29 Thread GitBox
dongjoon-hyun commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-982334173 Thank you, @sunchao ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on pull request #34264: [SPARK-36462][K8S] Add the ability to selectively disable watching or polling

2021-11-29 Thread GitBox
dongjoon-hyun commented on pull request #34264: URL: https://github.com/apache/spark/pull/34264#issuecomment-982333916 BTW, in general, I agree with your demands and requirements in this PR. The only concerns are - the better backward compatibility - the visibility of these

[GitHub] [spark] sungpeo opened a new pull request #34752: [SPARK][STREAMING] minRatePerPartition should be multiplied with secsPerBatch

2021-11-29 Thread GitBox
sungpeo opened a new pull request #34752: URL: https://github.com/apache/spark/pull/34752 ### What changes were proposed in this pull request? `maxRatePerPartition` means "max messages per partition per second". But minRatePerPartition does not. ("max messages per

[GitHub] [spark] dongjoon-hyun commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
dongjoon-hyun commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982332947 Thank you for review, @viirya . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-29 Thread GitBox
SparkQA commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-982331806 **[Test build #145753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145753/testReport)** for PR 34656 at commit

[GitHub] [spark] sunchao commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-29 Thread GitBox
sunchao commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-982331198 @dongjoon-hyun done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon closed pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions

2021-11-29 Thread GitBox
HyukjinKwon closed pull request #34732: URL: https://github.com/apache/spark/pull/34732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon closed pull request #34746: [SPARK-37489][PYTHON] Skip hasnans check in numops if eager_check disable

2021-11-29 Thread GitBox
HyukjinKwon closed pull request #34746: URL: https://github.com/apache/spark/pull/34746 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions

2021-11-29 Thread GitBox
HyukjinKwon commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-982330696 Merged to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon commented on pull request #34746: [SPARK-37489][PYTHON] Skip hasnans check in numops if eager_check disable

2021-11-29 Thread GitBox
HyukjinKwon commented on pull request #34746: URL: https://github.com/apache/spark/pull/34746#issuecomment-982330232 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluster environmen

2021-11-29 Thread GitBox
SparkQA commented on pull request #34635: URL: https://github.com/apache/spark/pull/34635#issuecomment-982330102 **[Test build #145752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145752/testReport)** for PR 34635 at commit

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982329931 **[Test build #145751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145751/testReport)** for PR 34751 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982324903 **[Test build #145749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145749/testReport)** for PR 34751 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982329599 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145749/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982329599 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145749/

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982329574 **[Test build #145749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145749/testReport)** for PR 34751 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34264: [SPARK-36462][K8S] Add the ability to selectively disable watching or polling

2021-11-29 Thread GitBox
dongjoon-hyun commented on a change in pull request #34264: URL: https://github.com/apache/spark/pull/34264#discussion_r758971094 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala ## @@

[GitHub] [spark] sunchao commented on a change in pull request #34635: [SPARK-37205][YARN] Introduce a new config 'spark.yarn.am.tokenConfRegex' to support renewing delegation tokens in a multi-cluste

2021-11-29 Thread GitBox
sunchao commented on a change in pull request #34635: URL: https://github.com/apache/spark/pull/34635#discussion_r758970173 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -340,6 +344,40 @@ private[spark] class Client(

[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests

2021-11-29 Thread GitBox
cloud-fan commented on a change in pull request #34719: URL: https://github.com/apache/spark/pull/34719#discussion_r758970183 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala ## @@ -0,0 +1,141 @@ +/* + * Licensed to

[GitHub] [spark] dongjoon-hyun commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
dongjoon-hyun commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982326438 cc @holdenk , @shrutig , @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] Yikun commented on pull request #34746: [SPARK-37489][PYTHON] Skip hasnans check in numops if eager_check disable

2021-11-29 Thread GitBox
Yikun commented on pull request #34746: URL: https://github.com/apache/spark/pull/34746#issuecomment-982325271 ``` ERROR [2.132s]: test_reuse_worker_of_parallelize_range (pyspark.tests.test_worker.WorkerReuseTest) --

[GitHub] [spark] SparkQA commented on pull request #34750: [SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

2021-11-29 Thread GitBox
SparkQA commented on pull request #34750: URL: https://github.com/apache/spark/pull/34750#issuecomment-982324974 **[Test build #145750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145750/testReport)** for PR 34750 at commit

[GitHub] [spark] ChenMichael commented on a change in pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-29 Thread GitBox
ChenMichael commented on a change in pull request #34684: URL: https://github.com/apache/spark/pull/34684#discussion_r758965590 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ## @@ -237,7 +238,23 @@ case class

[GitHub] [spark] SparkQA commented on pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
SparkQA commented on pull request #34751: URL: https://github.com/apache/spark/pull/34751#issuecomment-982324903 **[Test build #145749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145749/testReport)** for PR 34751 at commit

[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests

2021-11-29 Thread GitBox
Peng-Lei commented on a change in pull request #34719: URL: https://github.com/apache/spark/pull/34719#discussion_r758966959 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala ## @@ -0,0 +1,140 @@ +/* + * Licensed to the

[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests

2021-11-29 Thread GitBox
Peng-Lei commented on a change in pull request #34719: URL: https://github.com/apache/spark/pull/34719#discussion_r758966706 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala ## @@ -0,0 +1,140 @@ +/* + * Licensed to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34746: [SPARK-37489][PYTHON] Skip hasnans check in numops if eager_check disable

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34746: URL: https://github.com/apache/spark/pull/34746#issuecomment-982324060 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50217/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-982324154 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145735/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-982324056 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50216/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34657: [WIP] Support TimedeltaIndex in pandas API on Spark

2021-11-29 Thread GitBox
AmplabJenkins removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-982324057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50218/

[GitHub] [spark] AmplabJenkins commented on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-982324154 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145735/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34746: [SPARK-37489][PYTHON] Skip hasnans check in numops if eager_check disable

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34746: URL: https://github.com/apache/spark/pull/34746#issuecomment-982324060 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50217/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34657: [WIP] Support TimedeltaIndex in pandas API on Spark

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-982324057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50218/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py

2021-11-29 Thread GitBox
AmplabJenkins commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-982324056 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50216/ --

[GitHub] [spark] dongjoon-hyun opened a new pull request #34751: [SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

2021-11-29 Thread GitBox
dongjoon-hyun opened a new pull request #34751: URL: https://github.com/apache/spark/pull/34751 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

[GitHub] [spark] SparkQA removed a comment on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

2021-11-29 Thread GitBox
SparkQA removed a comment on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-982193103 **[Test build #145735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145735/testReport)** for PR 34741 at commit

[GitHub] [spark] SparkQA commented on pull request #34741: [SPARK-37463][SQL] Read/Write Timestamp ntz from/to Orc uses UTC time zone

2021-11-29 Thread GitBox
SparkQA commented on pull request #34741: URL: https://github.com/apache/spark/pull/34741#issuecomment-982322853 **[Test build #145735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145735/testReport)** for PR 34741 at commit

[GitHub] [spark] ChenMichael commented on a change in pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-29 Thread GitBox
ChenMichael commented on a change in pull request #34684: URL: https://github.com/apache/spark/pull/34684#discussion_r758965590 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ## @@ -237,7 +238,23 @@ case class

[GitHub] [spark] SparkQA commented on pull request #34657: [WIP] Support TimedeltaIndex in pandas API on Spark

2021-11-29 Thread GitBox
SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-982322524 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50218/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34213: [SPARK-36396][PYTHON] Implement DataFrame.cov

2021-11-29 Thread GitBox
SparkQA commented on pull request #34213: URL: https://github.com/apache/spark/pull/34213#issuecomment-982319967 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50219/ -- This is an automated message from the Apache

[GitHub] [spark] Yikun commented on a change in pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.5

2021-11-29 Thread GitBox
Yikun commented on a change in pull request #34717: URL: https://github.com/apache/spark/pull/34717#discussion_r758898358 ## File path: python/docs/source/user_guide/sql/arrow_pandas.rst ## @@ -387,7 +387,7 @@ working with timestamps in ``pandas_udf``\s to get the best

[GitHub] [spark] cloud-fan commented on a change in pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-29 Thread GitBox
cloud-fan commented on a change in pull request #34684: URL: https://github.com/apache/spark/pull/34684#discussion_r758962344 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ## @@ -237,7 +238,23 @@ case class

[GitHub] [spark] SparkQA commented on pull request #34746: [SPARK-37489][PYTHON] Skip hasnans check in numops if eager_check disable

2021-11-29 Thread GitBox
SparkQA commented on pull request #34746: URL: https://github.com/apache/spark/pull/34746#issuecomment-982313723 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50217/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py

2021-11-29 Thread GitBox
SparkQA commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-982309084 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50216/ -- This is an automated message from the

[GitHub] [spark] LuciferYang commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests

2021-11-29 Thread GitBox
LuciferYang commented on a change in pull request #34719: URL: https://github.com/apache/spark/pull/34719#discussion_r758953291 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala ## @@ -0,0 +1,140 @@ +/* + * Licensed to

  1   2   3   4   5   6   7   8   9   >