[GitHub] [spark] dongjoon-hyun opened a new pull request #29293: [SPARK-32487][CORE] Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource

dongjoon-hyun opened a new pull request #29293: URL: https://github.com/apache/spark/pull/29293 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[GitHub] [spark] ueshin opened a new pull request #29294: [SPARK-32160][CORE][PYSPARK][3.0] Add configs to switch allow/disallow to create SparkContext in executors.

ueshin opened a new pull request #29294: URL: https://github.com/apache/spark/pull/29294 ### What changes were proposed in this pull request? This is a backport of #29278, but with allowing to create `SparkContext` in executors by default. This PR adds configs to switch

[GitHub] [spark] AmplabJenkins commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning

AmplabJenkins commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-665973679 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun opened a new pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

dongjoon-hyun opened a new pull request #29295: URL: https://github.com/apache/spark/pull/29295 ### What changes were proposed in this pull request? This PR aims to recover Java 11 build in `GitHub Action`. ### Why are the changes needed? This test coverage is removed

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29278: [WIP][SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.

AmplabJenkins removed a comment on pull request #29278: URL: https://github.com/apache/spark/pull/29278#issuecomment-665971227 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

SparkQA commented on pull request #29067: URL: https://github.com/apache/spark/pull/29067#issuecomment-665827301 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] WinkerDu commented on pull request #29000: [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode

WinkerDu commented on pull request #29000: URL: https://github.com/apache/spark/pull/29000#issuecomment-666098220 Gentle ping @Ngone51 for further review, thanks :) This is an automated message from the Apache Git Service.

[GitHub] [spark] c21 commented on a change in pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

c21 commented on a change in pull request #29277: URL: https://github.com/apache/spark/pull/29277#discussion_r462677816 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala ## @@ -316,6 +318,387 @@ trait HashJoin extends BaseJoinExec {

[GitHub] [spark] SparkQA commented on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

SparkQA commented on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666047693 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

AmplabJenkins removed a comment on pull request #29276: URL: https://github.com/apache/spark/pull/29276#issuecomment-665903672 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun closed pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

dongjoon-hyun closed pull request #29295: URL: https://github.com/apache/spark/pull/29295 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] yaooqinn commented on pull request #29297: [SPARK-32406][SQL][FOLLOWUP] Make RESET fail against static and core configs

yaooqinn commented on pull request #29297: URL: https://github.com/apache/spark/pull/29297#issuecomment-666151668 cc @gatorsmile @cloud-fan @dongjoon-hyun @maropu thanks very much. This is an automated message from the

[GitHub] [spark] maropu commented on a change in pull request #29146: [WIP][SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

maropu commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r462610033 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -244,11 +258,31 @@ statement | SET TIME ZONE

[GitHub] [spark] SparkQA commented on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

SparkQA commented on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666042867 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] dot-vlad commented on pull request #25575: [SPARK-28818][SQL] Respect source column nullability in the arrays created by `freqItems()`

dot-vlad commented on pull request #25575: URL: https://github.com/apache/spark/pull/25575#issuecomment-665931156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] zhengruifeng commented on pull request #29255: [SPARK-32455][ML] LogisticRegressionModel prediction optimization

zhengruifeng commented on pull request #29255: URL: https://github.com/apache/spark/pull/29255#issuecomment-666060052 Thanks for reviewing! @huaxingao @srowen This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

AmplabJenkins removed a comment on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666050070 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29294: [SPARK-32160][CORE][PYSPARK][3.0] Add configs to switch allow/disallow to create SparkContext in executors.

AmplabJenkins removed a comment on pull request #29294: URL: https://github.com/apache/spark/pull/29294#issuecomment-665989640 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] yaooqinn opened a new pull request #29297: [SPARK-32406][SQL][FOLLOWUP] Make RESET fail against static and core configs

yaooqinn opened a new pull request #29297: URL: https://github.com/apache/spark/pull/29297 ### What changes were proposed in this pull request? This followup addresses comments from https://github.com/apache/spark/pull/29202#discussion_r462054784 1. make RESET static

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

HyukjinKwon commented on a change in pull request #29283: URL: https://github.com/apache/spark/pull/29283#discussion_r462692911 ## File path: docs/sparkr.md ## @@ -681,12 +681,12 @@ The current supported minimum version is 1.0.0; however, this might change betwe Arrow

[GitHub] [spark] holdenk commented on a change in pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning

holdenk commented on a change in pull request #29211: URL: https://github.com/apache/spark/pull/29211#discussion_r462638989 ## File path: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ## @@ -277,12 +282,52 @@ private[spark] class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

AmplabJenkins removed a comment on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665995022 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

AmplabJenkins removed a comment on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666020525 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maryannxue commented on a change in pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

maryannxue commented on a change in pull request #29276: URL: https://github.com/apache/spark/pull/29276#discussion_r462650727 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -695,7 +696,7 @@ private[spark] class TaskSetManager( def

[GitHub] [spark] SparkQA removed a comment on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

SparkQA removed a comment on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665703985 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

SparkQA commented on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666019778 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] LuciferYang opened a new pull request #29299: [SPARK-32490][BUILD] Upgrade netty-all to 4.1.51.Final

LuciferYang opened a new pull request #29299: URL: https://github.com/apache/spark/pull/29299 ### What changes were proposed in this pull request? This PR aims to bring the bug fixes from the latest netty version. ### Why are the changes needed? - 4.1.48.Final:

[GitHub] [spark] SparkQA removed a comment on pull request #29293: [SPARK-32487][CORE] Remove j.w.r.NotFoundException from `import` in [Stages|OneApplication]Resource

SparkQA removed a comment on pull request #29293: URL: https://github.com/apache/spark/pull/29293#issuecomment-665941955 **[Test build #126787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126787/testReport)** for PR 29293 at commit

[GitHub] [spark] viirya commented on a change in pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

viirya commented on a change in pull request #29276: URL: https://github.com/apache/spark/pull/29276#discussion_r462618883 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -695,7 +696,7 @@ private[spark] class TaskSetManager( def

[GitHub] [spark] cloud-fan commented on a change in pull request #29297: [SPARK-32406][SQL][FOLLOWUP] Make RESET fail against static and core configs

cloud-fan commented on a change in pull request #29297: URL: https://github.com/apache/spark/pull/29297#discussion_r462773434 ## File path: sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala ## @@ -142,9 +142,12 @@ class SQLConfSuite extends QueryTest

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

AmplabJenkins removed a comment on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665850566 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r462761942 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -244,11 +258,31 @@ statement | SET TIME ZONE

[GitHub] [spark] gatorsmile commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

gatorsmile commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r462777127 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala ## @@ -61,6 +63,64 @@ class SparkSqlParserSuite

[GitHub] [spark] SparkQA commented on pull request #29278: [WIP][SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.

SparkQA commented on pull request #29278: URL: https://github.com/apache/spark/pull/29278#issuecomment-665970816 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] cloud-fan commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

cloud-fan commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666201062 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] cloud-fan closed pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

cloud-fan closed pull request #29296: URL: https://github.com/apache/spark/pull/29296 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA removed a comment on pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

SparkQA removed a comment on pull request #29234: URL: https://github.com/apache/spark/pull/29234#issuecomment-665941972 **[Test build #126788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126788/testReport)** for PR 29234 at commit

[GitHub] [spark] cloud-fan closed pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

cloud-fan closed pull request #29234: URL: https://github.com/apache/spark/pull/29234 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] viirya commented on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

viirya commented on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666050771 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

HyukjinKwon commented on a change in pull request #29298: URL: https://github.com/apache/spark/pull/29298#discussion_r462787946 ## File path: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala ## @@ -258,6 +260,33 @@ class SubmitRestProtocolSuite

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

AmplabJenkins removed a comment on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666043783 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

AmplabJenkins commented on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666020525 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] c21 commented on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

c21 commented on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666027985 @cloud-fan - updated the PR with addressing comments, and it is ready for review. Also updated the PR description for latest codegen code of example query. Thanks.

[GitHub] [spark] Ngone51 commented on a change in pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

Ngone51 commented on a change in pull request #29270: URL: https://github.com/apache/spark/pull/29270#discussion_r462769211 ## File path: sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] HyukjinKwon commented on pull request #29300: [SPARK-32491][INFRA] Do not install SparkR in test-only mode in testing script

HyukjinKwon commented on pull request #29300: URL: https://github.com/apache/spark/pull/29300#issuecomment-666201596 The fix here should partially fix the build when R is not needed. Looks it fails when R is needed too, for example, at

[GitHub] [spark] maropu commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

maropu commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666201603 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] leanken opened a new pull request #29301: [SPARK-32474][SQL][FOLLOWUP] NullAwareAntiJoin multi-column support

leanken opened a new pull request #29301: URL: https://github.com/apache/spark/pull/29301 ### What changes were proposed in this pull request? This is a follow up issue of [SPARK-32290](https://issues.apache.org/jira/browse/SPARK-32290). In SPARK-32290, We only support Single

[GitHub] [spark] AmplabJenkins commented on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

AmplabJenkins commented on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666043783 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29294: [SPARK-32160][CORE][PYSPARK][3.0] Add configs to switch allow/disallow to create SparkContext in executors.

SparkQA commented on pull request #29294: URL: https://github.com/apache/spark/pull/29294#issuecomment-665989295 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] tgravescs commented on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

tgravescs commented on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665855971 I merged this to master, unfortunately wouldn't pick clean to branch-3.0. @cloud-fan would you want to put up PR for branch-3.0? Otherwise Andy or myself can.

[GitHub] [spark] cloud-fan closed pull request #29204: [SPARK-32412][SQL] Unify error handling for spark thrift server operations

cloud-fan closed pull request #29204: URL: https://github.com/apache/spark/pull/29204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] maropu edited a comment on pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

maropu edited a comment on pull request #29270: URL: https://github.com/apache/spark/pull/29270#issuecomment-666012926 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

AmplabJenkins commented on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666050070 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

AmplabJenkins removed a comment on pull request #29234: URL: https://github.com/apache/spark/pull/29234#issuecomment-665942755 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu opened a new pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

maropu opened a new pull request #29296: URL: https://github.com/apache/spark/pull/29296 ### What changes were proposed in this pull request? This PR aims to update `SqlBse.g4` for avoiding generating unused code. Currently, ANTLR generates unused methods and variables;

[GitHub] [spark] zsxwing commented on a change in pull request #28986: [SPARK-32160][CORE][PYSPARK] Disallow to create SparkContext in executors.

zsxwing commented on a change in pull request #28986: URL: https://github.com/apache/spark/pull/28986#discussion_r462679180 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -2554,6 +2557,19 @@ object SparkContext extends Logging { } } +

[GitHub] [spark] AmplabJenkins commented on pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

AmplabJenkins commented on pull request #29276: URL: https://github.com/apache/spark/pull/29276#issuecomment-665903672 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

AmplabJenkins commented on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-666049254 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on pull request #29286: [WIP}[SPARK-21708][Build] Migrate build to sbt 1.x

HyukjinKwon commented on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-666020324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] AmplabJenkins commented on pull request #29294: [SPARK-32160][CORE][PYSPARK][3.0] Add configs to switch allow/disallow to create SparkContext in executors.

AmplabJenkins commented on pull request #29294: URL: https://github.com/apache/spark/pull/29294#issuecomment-665989640 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

AmplabJenkins commented on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665850566 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning

SparkQA commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-665973326 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA commented on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

SparkQA commented on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-666048818 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins commented on pull request #29278: [WIP][SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.

AmplabJenkins commented on pull request #29278: URL: https://github.com/apache/spark/pull/29278#issuecomment-665971227 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun closed pull request #29293: [SPARK-32487][CORE] Remove j.w.r.NotFoundException from `import` in [Stages|OneApplication]Resource

dongjoon-hyun closed pull request #29293: URL: https://github.com/apache/spark/pull/29293 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] cloud-fan commented on a change in pull request #20176: [SPARK-22981][SQL] Fix incorrect results of Casting Struct to String

cloud-fan commented on a change in pull request #20176: URL: https://github.com/apache/spark/pull/20176#discussion_r462767193 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala ## @@ -259,6 +259,29 @@ case class Cast(child:

[GitHub] [spark] cloud-fan commented on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

cloud-fan commented on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-666158610 Does it qualify a backport? It's kind of a new feature. This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon commented on pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

HyukjinKwon commented on pull request #29298: URL: https://github.com/apache/spark/pull/29298#issuecomment-666170687 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] SparkQA removed a comment on pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

SparkQA removed a comment on pull request #29067: URL: https://github.com/apache/spark/pull/29067#issuecomment-665690606 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-666133472 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

AmplabJenkins removed a comment on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-666049254 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] attilapiros commented on pull request #29090: [SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

attilapiros commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-665952684 Thanks @holdenk for looking into this. And what about logging out a warning when no unit is given? Like: "Memory setting without explicit unit (${value})

[GitHub] [spark] HyukjinKwon commented on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

HyukjinKwon commented on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-666014185 Thank you @dongjoon-hyun. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] huaxingao closed pull request #29255: [SPARK-32455][ML] LogisticRegressionModel prediction optimization

huaxingao closed pull request #29255: URL: https://github.com/apache/spark/pull/29255 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA commented on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

SparkQA commented on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665994152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA removed a comment on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

SparkQA removed a comment on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665994152 **[Test build #126795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126795/testReport)** for PR 29295 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

SparkQA removed a comment on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-666048818 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] uncleGen commented on a change in pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

uncleGen commented on a change in pull request #28781: URL: https://github.com/apache/spark/pull/28781#discussion_r462696730 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/StreamingQueryStatusStore.scala ## @@ -0,0 +1,60 @@ +/* + * Licensed to the

[GitHub] [spark] HyukjinKwon closed pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

HyukjinKwon closed pull request #28968: URL: https://github.com/apache/spark/pull/28968 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] huaxingao commented on pull request #29255: [SPARK-32455][ML] LogisticRegressionModel prediction optimization

huaxingao commented on pull request #29255: URL: https://github.com/apache/spark/pull/29255#issuecomment-666058041 Merged to master. Thanks @zhengruifeng @srowen This is an automated message from the Apache Git Service. To

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

dongjoon-hyun commented on a change in pull request #29298: URL: https://github.com/apache/spark/pull/29298#discussion_r462780182 ## File path: core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json ## @@ -5,283 +5,283 @@ "scalaVersion" :

[GitHub] [spark] dongjoon-hyun commented on pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

dongjoon-hyun commented on pull request #29298: URL: https://github.com/apache/spark/pull/29298#issuecomment-666169269 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

SparkQA removed a comment on pull request #29276: URL: https://github.com/apache/spark/pull/29276#issuecomment-665792999 **[Test build #126786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126786/testReport)** for PR 29276 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

SparkQA removed a comment on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666047693 **[Test build #126798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126798/testReport)** for PR 29283 at commit

[GitHub] [spark] holdenk commented on pull request #29274: [SPARK-32397][BUILD] Allow specifying of time for build to keep time consistent between modules

holdenk commented on pull request #29274: URL: https://github.com/apache/spark/pull/29274#issuecomment-665819306 Thanks @HyukjinKwon I've added that it impacts `maven deploy` to the description. This is an automated message

[GitHub] [spark] jiangxb1987 commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

jiangxb1987 commented on pull request #29228: URL: https://github.com/apache/spark/pull/29228#issuecomment-666137629 It would be really great if you can list the test cases/suites that could get simplified by this change, thanks!

[GitHub] [spark] cloud-fan commented on pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

cloud-fan commented on pull request #29234: URL: https://github.com/apache/spark/pull/29234#issuecomment-666142201 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] maropu commented on a change in pull request #20176: [SPARK-22981][SQL] Fix incorrect results of Casting Struct to String

maropu commented on a change in pull request #20176: URL: https://github.com/apache/spark/pull/20176#discussion_r462636447 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala ## @@ -259,6 +259,29 @@ case class Cast(child: Expression,

[GitHub] [spark] HeartSaVioR removed a comment on pull request #29272: [SPARK-32468][SS][TESTS] Fix timeout config issue in Kafka connector tests

HeartSaVioR removed a comment on pull request #29272: URL: https://github.com/apache/spark/pull/29272#issuecomment-666160436 retest this, please This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] viirya commented on a change in pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

viirya commented on a change in pull request #29234: URL: https://github.com/apache/spark/pull/29234#discussion_r462557526 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/SchemaUtils.scala ## @@ -42,7 +42,27 @@ private[spark] object SchemaUtils { */

[GitHub] [spark] SparkQA removed a comment on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

SparkQA removed a comment on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666019778 **[Test build #126796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126796/testReport)** for PR 29277 at commit

[GitHub] [spark] cloud-fan commented on pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

cloud-fan commented on pull request #29270: URL: https://github.com/apache/spark/pull/29270#issuecomment-666098964 One of my worries is that: this test generates plans with empty tables, and we lost test coverage for things like SMJ. Can the `variant` feature help to improve the test

[GitHub] [spark] dongjoon-hyun commented on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

dongjoon-hyun commented on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665994816 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28986: [SPARK-32160][CORE][PYSPARK] Disallow to create SparkContext in executors.

HyukjinKwon commented on a change in pull request #28986: URL: https://github.com/apache/spark/pull/28986#discussion_r462673731 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -2554,6 +2557,19 @@ object SparkContext extends Logging { } } +

[GitHub] [spark] HeartSaVioR commented on pull request #29272: [SPARK-32468][SS][TESTS] Fix timeout config issue in Kafka connector tests

HeartSaVioR commented on pull request #29272: URL: https://github.com/apache/spark/pull/29272#issuecomment-666158718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins removed a comment on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666050019 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29146: [WIP][SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

SparkQA commented on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-665949787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins commented on pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

AmplabJenkins commented on pull request #29234: URL: https://github.com/apache/spark/pull/29234#issuecomment-665942755 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] holdenk commented on pull request #29090: [SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

holdenk commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-665972368 That sounds good to me. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] viirya commented on a change in pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

viirya commented on a change in pull request #29283: URL: https://github.com/apache/spark/pull/29283#discussion_r462682167 ## File path: docs/sparkr.md ## @@ -681,12 +681,12 @@ The current supported minimum version is 1.0.0; however, this might change betwe Arrow

  1   2   3   4   5   6   7   8   >