[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721198104 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35169/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins removed a comment on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721196324 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721196324 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721196293 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35168/

[GitHub] [spark] imback82 commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

imback82 commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516749544 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzePartitionCommand.scala ## @@ -75,6 +75,9 @@ case class

[GitHub] [spark] jkleckner commented on a change in pull request #29496: [SPARK-24266][K8S][2.4] Restart the watcher when we receive a version changed from k8s

jkleckner commented on a change in pull request #29496: URL: https://github.com/apache/spark/pull/29496#discussion_r516741734 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala ## @@ -177,4 +188,27

[GitHub] [spark] HyukjinKwon edited a comment on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

HyukjinKwon edited a comment on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-720965482 BTW, @dongjoon-hyun, seems like Ubuntu 20.04 changed its default Python version to Python 3 (?). I think that's why the error message shows a bit weird like

[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721182558 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35168/

[GitHub] [spark] srowen commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

srowen commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516730762 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object ExtractPythonUDFs

[GitHub] [spark] cloud-fan commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

cloud-fan commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516728706 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ## @@ -175,7 +175,10 @@ abstract class SQLViewSuite extends

[GitHub] [spark] cloud-fan commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

cloud-fan commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516727702 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -280,6 +280,9 @@ class

[GitHub] [spark] cloud-fan commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

cloud-fan commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516726862 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzePartitionCommand.scala ## @@ -75,6 +75,9 @@ case class

[GitHub] [spark] srowen commented on pull request #30185: [SPARK-33152][SQL] This PR proposes a new logic to maintain & track constraints which solves the OOM or performance issues in query compilatio

srowen commented on pull request #30185: URL: https://github.com/apache/spark/pull/30185#issuecomment-721169846 This still doesn't look right. I'd squash your branch changes and rebase on master. This is an automated

[GitHub] [spark] srowen commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

srowen commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721165959 Hoo boy that is a lot of code to touch. It sounds like eventually it has to happen for Scala 3 and avoids some deprecation noise. I'm not against it though as usual there is a

[GitHub] [spark] cloud-fan commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

cloud-fan commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516725745 ## File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala ## @@ -419,17 +410,16 @@ class

[GitHub] [spark] srowen closed pull request #30182: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost

srowen closed pull request #30182: URL: https://github.com/apache/spark/pull/30182 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] srowen commented on pull request #30182: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost

srowen commented on pull request #30182: URL: https://github.com/apache/spark/pull/30182#issuecomment-721162096 Merged to master/3.0 This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] cloud-fan commented on a change in pull request #30193: [SPARK-33293][SQL] Refactor WriteToDataSourceV2Exec and reduce code duplication

cloud-fan commented on a change in pull request #30193: URL: https://github.com/apache/spark/pull/30193#discussion_r516722182 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala ## @@ -479,15 +432,16 @@ object

[GitHub] [spark] cloud-fan commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

cloud-fan commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516720258 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object ExtractPythonUDFs

[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721156123 **[Test build #130568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130568/testReport)** for PR 30234 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

AmplabJenkins removed a comment on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721154544 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] akiyamaneko edited a comment on pull request #30182: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost

akiyamaneko edited a comment on pull request #30182: URL: https://github.com/apache/spark/pull/30182#issuecomment-721154044 Jenkins test this please This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

AmplabJenkins commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721154544 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

SparkQA removed a comment on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721027459 **[Test build #130563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130563/testReport)** for PR 30151 at commit

[GitHub] [spark] akiyamaneko commented on pull request #30182: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost

akiyamaneko commented on pull request #30182: URL: https://github.com/apache/spark/pull/30182#issuecomment-721154044 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] SparkQA commented on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

SparkQA commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721153265 **[Test build #130563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130563/testReport)** for PR 30151 at commit

[GitHub] [spark] HyukjinKwon closed pull request #30227: [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*)

HyukjinKwon closed pull request #30227: URL: https://github.com/apache/spark/pull/30227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721144076 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins removed a comment on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721144076 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] LuciferYang opened a new pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

LuciferYang opened a new pull request #30234: URL: https://github.com/apache/spark/pull/30234 ### What changes were proposed in this pull request? The purpose of this pr is to partial resolve SPARK-33285. `Auto-application` is dropped in Scala 3 and deprecated in 2.13

[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721142854 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] LuciferYang commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

LuciferYang commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721140762 cc @srowen this is the first part to fix this compilation warnings, if it's too big, I can split multiple small prs.

[GitHub] [spark] SparkQA removed a comment on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA removed a comment on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721142854 **[Test build #130567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130567/testReport)** for PR 30234 at commit

[GitHub] [spark] zero323 commented on pull request #30227: [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*)

zero323 commented on pull request #30227: URL: https://github.com/apache/spark/pull/30227#issuecomment-721137978 Thanks @HyukjinKwon! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] SparkQA removed a comment on pull request #30038: [SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect)

SparkQA removed a comment on pull request #30038: URL: https://github.com/apache/spark/pull/30038#issuecomment-721069806 **[Test build #130566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130566/testReport)** for PR 30038 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

AmplabJenkins removed a comment on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721055509 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] ScrapCodes commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

ScrapCodes commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721070338 Hi @holdenk and @dongjoon-hyun, Hope you are doing good ! Do you have a view around this one way or the other? Thanks !

[GitHub] [spark] gengliangwang commented on pull request #30221: [SPARK-33314][SQL] Avoid dropping rows in Avro reader

gengliangwang commented on pull request #30221: URL: https://github.com/apache/spark/pull/30221#issuecomment-720397753 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] tgravescs commented on a change in pull request #30186: [SPARK-23432][UI] Add executor peak jvm memory metrics in executors page

tgravescs commented on a change in pull request #30186: URL: https://github.com/apache/spark/pull/30186#discussion_r516080247 ## File path: core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html ## @@ -86,6 +86,22 @@ Executors

[GitHub] [spark] dongjoon-hyun closed pull request #30226: [SPARK-33299][SQL][DOCS] Don't mention schemas in JSON format in docs for `from_json`

dongjoon-hyun closed pull request #30226: URL: https://github.com/apache/spark/pull/30226 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

HyukjinKwon commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516423241 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30227: [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*)

AmplabJenkins removed a comment on pull request #30227: URL: https://github.com/apache/spark/pull/30227#issuecomment-720494098 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] ScrapCodes commented on pull request #30038: [SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect)

ScrapCodes commented on pull request #30038: URL: https://github.com/apache/spark/pull/30038#issuecomment-720278520 Hi @huaxingao and @cloud-fan , This is ready for review ! This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30223: [SPARK-33306][SQL][FOLLOWUP] Group DateType and TimestampType together in `needsTimeZone()`

AmplabJenkins removed a comment on pull request #30223: URL: https://github.com/apache/spark/pull/30223#issuecomment-720382265 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HeartSaVioR commented on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

HeartSaVioR commented on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-720970588 @viirya Kindly reminder. @gaborgsomogyi and @xuanyuanking gave +1 - would it be good to persuade your call?

[GitHub] [spark] dongjoon-hyun closed pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

dongjoon-hyun closed pull request #29533: URL: https://github.com/apache/spark/pull/29533 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #30230: [SPARK-33323][SQL] Add query resolved check before convert hive relation

AmplabJenkins commented on pull request #30230: URL: https://github.com/apache/spark/pull/30230#issuecomment-720890323 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30216: [SPARK-33304][R][SQL] Add from_avro and to_avro functions to SparkR

AmplabJenkins commented on pull request #30216: URL: https://github.com/apache/spark/pull/30216#issuecomment-720328716 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] ulysses-you opened a new pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand

ulysses-you opened a new pull request #28647: URL: https://github.com/apache/spark/pull/28647 ### What changes were proposed in this pull request? At CreateTableLikeCommand, we use the new tblproperties with merge source tblproperties. ### Why are the changes needed?

[GitHub] [spark] cloud-fan commented on a change in pull request #30202: [SPARK-33248][SQL][FOLLOWUP] Update migration guide to make clear what behavior changed and make variable names and configurati

cloud-fan commented on a change in pull request #30202: URL: https://github.com/apache/spark/pull/30202#discussion_r515753866 ## File path: docs/sql-migration-guide.md ## @@ -52,7 +52,7 @@ license: | - In Spark 3.1, the `schema_of_json` and `schema_of_csv` functions

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30224: [SPARK-33316][SQL] Support user provided nullable Avro schema for non-nullable catalyst schema in Avro writing

AmplabJenkins removed a comment on pull request #30224: URL: https://github.com/apache/spark/pull/30224#issuecomment-720385125 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on pull request #30231: [SPARK-33156][INFRA][3.0] Upgrade GithubAction image from 18.04 to 20.04

dongjoon-hyun commented on pull request #30231: URL: https://github.com/apache/spark/pull/30231#issuecomment-720885674 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for merging partitions

SparkQA commented on pull request #30164: URL: https://github.com/apache/spark/pull/30164#issuecomment-720725184 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] cloud-fan commented on a change in pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

cloud-fan commented on a change in pull request #30144: URL: https://github.com/apache/spark/pull/30144#discussion_r515765822 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -3691,6 +3691,32 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] sunchao removed a comment on pull request #30176: [SQL][MINOR] Update from_unixtime doc

sunchao removed a comment on pull request #30176: URL: https://github.com/apache/spark/pull/30176#issuecomment-720671002 @HyukjinKwon @Obbay2 is there a JIRA for this? This is an automated message from the Apache Git

[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

HeartSaVioR commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-720900095 Sigh. Now it has conflicts so I can't go with giving +1 and merging. @cchighman Could you please fix the conflicts? It would be totally OK if you'd like to let me

[GitHub] [spark] AmplabJenkins commented on pull request #26319: [SPARK-29594][SQL] Provide better error message when creating a Dataset from a Sequence of Case class where a field name started with a

AmplabJenkins commented on pull request #26319: URL: https://github.com/apache/spark/pull/26319#issuecomment-720287298 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #26319: [SPARK-29594][SQL] Provide better error message when creating a Dataset from a Sequence of Case class where a field name started with

SparkQA removed a comment on pull request #26319: URL: https://github.com/apache/spark/pull/26319#issuecomment-720206210 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] xkrogen commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature

xkrogen commented on pull request #29906: URL: https://github.com/apache/spark/pull/29906#issuecomment-720575364 Huge thanks for pushing this through @tgravescs ! It was no small effort! This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand

SparkQA removed a comment on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-720570503 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for merging par

SparkQA removed a comment on pull request #30164: URL: https://github.com/apache/spark/pull/30164#issuecomment-720725184 **[Test build #130533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130533/testReport)** for PR 30164 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29935: [SPARK-33055][PYTHON][SQL] Add Python CalendarIntervalType

AmplabJenkins removed a comment on pull request #29935: URL: https://github.com/apache/spark/pull/29935#issuecomment-702872843 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HeartSaVioR commented on a change in pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

HeartSaVioR commented on a change in pull request #30151: URL: https://github.com/apache/spark/pull/30151#discussion_r515757520 ## File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala ## @@ -126,6 +126,123 @@ private[ui]

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30009: [SPARK-32907][ML] adaptively blockify instances - LinearSVC

AmplabJenkins removed a comment on pull request #30009: URL: https://github.com/apache/spark/pull/30009#issuecomment-720876636 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

AmplabJenkins removed a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-720292747 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] otterc commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-720702382 With the latest commit, all the review comments have been addressed. @Ngone51 @tgravescs @attilapiros @mridulm @Victsm @jiangxb1987 Please take another look at the changes.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30230: [SPARK-33323][SQL] Add query resolved check before convert hive relation

AmplabJenkins removed a comment on pull request #30230: URL: https://github.com/apache/spark/pull/30230#issuecomment-720890323 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] Ngone51 commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

Ngone51 commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516467454 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + * Licensed

[GitHub] [spark] SparkQA removed a comment on pull request #29082: [SPARK-32288][UI] Add exception summary for failed tasks in stage page

SparkQA removed a comment on pull request #29082: URL: https://github.com/apache/spark/pull/29082#issuecomment-720852047 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on pull request #30177: [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends.

HyukjinKwon commented on pull request #30177: URL: https://github.com/apache/spark/pull/30177#issuecomment-720243217 Let me monitor a bit more before reverting it. I just saw it once. This is an automated message from the

[GitHub] [spark] Victsm commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

Victsm commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516167838 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/PushBlockStream.java ## @@ -23,24 +23,31 @@ import

[GitHub] [spark] HyukjinKwon commented on pull request #30227: [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*)

HyukjinKwon commented on pull request #30227: URL: https://github.com/apache/spark/pull/30227#issuecomment-720813038 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] AmplabJenkins commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

AmplabJenkins commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-720895587 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] bozhang2820 opened a new pull request #30224: [SPARK-33316][SQL] Support user provided nullable Avro schema for non-nullable catalyst schema in Avro writing

bozhang2820 opened a new pull request #30224: URL: https://github.com/apache/spark/pull/30224 ### What changes were proposed in this pull request? This change is to support user provided nullable Avro schema for data with non-nullable catalyst schema in Avro writing. Without

[GitHub] [spark] SparkQA commented on pull request #30221: [SPARK-33314][SQL] Avoid dropping rows in Avro reader

SparkQA commented on pull request #30221: URL: https://github.com/apache/spark/pull/30221#issuecomment-720879566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516175694 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/PushBlockStream.java ## @@ -23,24 +23,31 @@ import

[GitHub] [spark] zero323 commented on a change in pull request #30216: [SPARK-33304][R][SQL] Add from_avro and to_avro functions to SparkR

zero323 commented on a change in pull request #30216: URL: https://github.com/apache/spark/pull/30216#discussion_r515789891 ## File path: R/pkg/DESCRIPTION ## @@ -42,6 +42,7 @@ Collate: 'context.R' 'deserialize.R' 'functions.R' +'functions_avro.R' Review

[GitHub] [spark] zero323 opened a new pull request #30227: [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*)

zero323 opened a new pull request #30227: URL: https://github.com/apache/spark/pull/30227 ### What changes were proposed in this pull request? This PR adds support for passing `Column`s as input to PySpark sorting functions. ### Why are the changes needed?

[GitHub] [spark] zhengruifeng commented on pull request #30009: [SPARK-32907][ML] adaptively blockify instances - LinearSVC

zhengruifeng commented on pull request #30009: URL: https://github.com/apache/spark/pull/30009#issuecomment-720851384 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins commented on pull request #30038: [SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect)

AmplabJenkins commented on pull request #30038: URL: https://github.com/apache/spark/pull/30038#issuecomment-720294931 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #26319: [SPARK-29594][SQL] Provide better error message when creating a Dataset from a Sequence of Case class where a field name starte

AmplabJenkins removed a comment on pull request #26319: URL: https://github.com/apache/spark/pull/26319#issuecomment-720287298 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on pull request #30233: [SPARK-33324][K8S][BUILD] Upgrade kubernetes-client to 4.11.1

dongjoon-hyun commented on pull request #30233: URL: https://github.com/apache/spark/pull/30233#issuecomment-720928756 Thank you, @HyukjinKwon . Merged to master for Apache Spark 3.1. This is an automated message from the

[GitHub] [spark] attilapiros commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

attilapiros commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516643629 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + *

[GitHub] [spark] HeartSaVioR commented on a change in pull request #30221: [SPARK-33314][SQL] Avoid dropping rows in Avro reader

HeartSaVioR commented on a change in pull request #30221: URL: https://github.com/apache/spark/pull/30221#discussion_r515822838 ## File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala ## @@ -171,9 +171,15 @@ private[sql] object AvroUtils extends

[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

AmplabJenkins commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-720292747 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun opened a new pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

dongjoon-hyun opened a new pull request #30232: URL: https://github.com/apache/spark/pull/30232 ### What changes were proposed in this pull request? This PR aims to upgrade `Github Action` runner image from `Ubuntu 18.04 (LTS)` to `Ubuntu 20.04 (LTS)`. ### Why are the changes

[GitHub] [spark] HeartSaVioR edited a comment on pull request #30210: [SPARK-33259][SS] Disable streaming query with possible correctness issue by default

HeartSaVioR edited a comment on pull request #30210: URL: https://github.com/apache/spark/pull/30210#issuecomment-720876379 No I don't have real case for knowing and taking the risk. Probably I could create some query which could evade the issue, but I agree that's more likely in theory

[GitHub] [spark] SparkQA commented on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

SparkQA commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721027459 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins commented on pull request #30225: [SPARK-33187][SQL] Add a check on the number of returned metastore pa…

AmplabJenkins commented on pull request #30225: URL: https://github.com/apache/spark/pull/30225#issuecomment-720385116 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HeartSaVioR commented on pull request #30210: [SPARK-33259][SS] Disable streaming query with possible correctness issue by default

HeartSaVioR commented on pull request #30210: URL: https://github.com/apache/spark/pull/30210#issuecomment-720876379 No I don't have real case for knowing and taking the risk. Probably I could create some query which could evade the issue, but I agree that's more likely in theory and not

[GitHub] [spark] cloud-fan closed pull request #30200: [SPARK-33027][SQL] Add DisableUnnecessaryBucketedScan rule to AQE

cloud-fan closed pull request #30200: URL: https://github.com/apache/spark/pull/30200 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] tgravescs commented on pull request #30204: [SPARK-33288][SPARK-32661][K8S] Stage level scheduling support for Kubernetes

tgravescs commented on pull request #30204: URL: https://github.com/apache/spark/pull/30204#issuecomment-720557444 seems like random test failures happening. I tried the recent YarnClusterSuite locally and it works fine.

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

AngersZh commented on a change in pull request #30144: URL: https://github.com/apache/spark/pull/30144#discussion_r515926187 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -3691,6 +3691,32 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] kiszk commented on pull request #26319: [SPARK-29594][SQL] Provide better error message when creating a Dataset from a Sequence of Case class where a field name started with a number

kiszk commented on pull request #26319: URL: https://github.com/apache/spark/pull/26319#issuecomment-720634523 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA removed a comment on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

SparkQA removed a comment on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-720433951 **[Test build #130526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130526/testReport)** for PR 30144 at commit

[GitHub] [spark] SparkQA commented on pull request #30230: [SPARK-33323][SQL] Add query resolved check before convert hive relation

SparkQA commented on pull request #30230: URL: https://github.com/apache/spark/pull/30230#issuecomment-720875095 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30038: [SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer diale

AmplabJenkins removed a comment on pull request #30038: URL: https://github.com/apache/spark/pull/30038#issuecomment-720294931 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30186: [SPARK-23432][UI] Add executor peak jvm memory metrics in executors page

AmplabJenkins removed a comment on pull request #30186: URL: https://github.com/apache/spark/pull/30186#issuecomment-720874715 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30216: [SPARK-33304][R][SQL] Add from_avro and to_avro functions to SparkR

HyukjinKwon commented on a change in pull request #30216: URL: https://github.com/apache/spark/pull/30216#discussion_r515852857 ## File path: R/pkg/R/functions_avro.R ## @@ -0,0 +1,117 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] [spark] SparkQA commented on pull request #30228: [SPARK-33319][SQL][TEST] Add all built-in SerDes to HiveSerDeReadWriteSuite

SparkQA commented on pull request #30228: URL: https://github.com/apache/spark/pull/30228#issuecomment-720515948 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

<    1   2   3   4   5   6   7   >