[GitHub] [spark] cloud-fan commented on pull request #29301: [SPARK-32474][SQL][FOLLOWUP] NullAwareAntiJoin multi-column support

cloud-fan commented on pull request #29301: URL: https://github.com/apache/spark/pull/29301#issuecomment-666264155 can you create a new jira ticket? It's a major feature that shouldn't be treated as a followup. This is an

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins removed a comment on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666255678 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666255678 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126800/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins removed a comment on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666254919 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666254919 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] itsvikramagr commented on pull request #28904: [SPARK-30462][SS] Streamline the logic on file stream source and sink metadata log to avoid memory issue

itsvikramagr commented on pull request #28904: URL: https://github.com/apache/spark/pull/28904#issuecomment-666248758 @HeartSaVioR - This is a much-needed fix. Thanks for it. I have an orthogonal question. Why do we need to worry about compacting the file sink metadata? I can think

[GitHub] [spark] beliefer commented on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

beliefer commented on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666240549 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] liangz1 commented on pull request #29284: [SPARK-32479][PYSPARK] Fix the slicing logic in createDataFrame when converting pandas dataframe to arrow table

liangz1 commented on pull request #29284: URL: https://github.com/apache/spark/pull/29284#issuecomment-666238258 This is not a bug. Spark will always create `defaultParallelism` partitions; there could be empty partitions. Closing this PR.

[GitHub] [spark] liangz1 closed pull request #29284: [SPARK-32479][PYSPARK] Fix the slicing logic in createDataFrame when converting pandas dataframe to arrow table

liangz1 closed pull request #29284: URL: https://github.com/apache/spark/pull/29284 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

AmplabJenkins removed a comment on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666234409 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

AmplabJenkins commented on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666234409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126798/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

AmplabJenkins removed a comment on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666233540 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

AmplabJenkins commented on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666233540 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

AmplabJenkins removed a comment on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-666231820 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

AmplabJenkins commented on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-666231820 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126802/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

AmplabJenkins removed a comment on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-666231024 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

AmplabJenkins commented on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-666231024 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] c21 commented on a change in pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

c21 commented on a change in pull request #29277: URL: https://github.com/apache/spark/pull/29277#discussion_r462830406 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoinExec.scala ## @@ -70,4 +74,54 @@ case class ShuffledHashJoinExec(

[GitHub] [spark] beliefer commented on pull request #27507: [SPARK-24884][SQL] Support regexp function regexp_extract_all

beliefer commented on pull request #27507: URL: https://github.com/apache/spark/pull/27507#issuecomment-666227396 retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] beliefer commented on pull request #27507: [SPARK-24884][SQL] Support regexp function regexp_extract_all

beliefer commented on pull request #27507: URL: https://github.com/apache/spark/pull/27507#issuecomment-666227396 retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] beliefer commented on pull request #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression

beliefer commented on pull request #27429: URL: https://github.com/apache/spark/pull/27429#issuecomment-666227176 retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] beliefer commented on pull request #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression

beliefer commented on pull request #27429: URL: https://github.com/apache/spark/pull/27429#issuecomment-666227176 retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HeartSaVioR commented on pull request #29272: [SPARK-32468][SS][TESTS] Fix timeout config issue in Kafka connector tests

HeartSaVioR commented on pull request #29272: URL: https://github.com/apache/spark/pull/29272#issuecomment-666223018 retest this, please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

AmplabJenkins removed a comment on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666222739 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

AmplabJenkins commented on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666222739 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] leanken commented on pull request #29301: [SPARK-32474][SQL][FOLLOWUP] NullAwareAntiJoin multi-column support

leanken commented on pull request #29301: URL: https://github.com/apache/spark/pull/29301#issuecomment-666202715 @cloud-fan @maropu @agrawaldevesh Could you guys have a look at this follow up, See if is it worth to do such trade-off to support multi-column NAAJ.

[GitHub] [spark] cloud-fan closed pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

cloud-fan closed pull request #29296: URL: https://github.com/apache/spark/pull/29296 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

cloud-fan commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666201062 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon commented on pull request #29300: [SPARK-32491][INFRA] Do not install SparkR in test-only mode in testing script

HyukjinKwon commented on pull request #29300: URL: https://github.com/apache/spark/pull/29300#issuecomment-666201596 The fix here should partially fix the build when R is not needed. Looks it fails when R is needed too, for example, at

[GitHub] [spark] maropu commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

maropu commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666201603 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] leanken opened a new pull request #29301: [SPARK-32474][SQL][FOLLOWUP] NullAwareAntiJoin multi-column support

leanken opened a new pull request #29301: URL: https://github.com/apache/spark/pull/29301 ### What changes were proposed in this pull request? This is a follow up issue of [SPARK-32290](https://issues.apache.org/jira/browse/SPARK-32290). In SPARK-32290, We only support Single

[GitHub] [spark] SparkQA commented on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

SparkQA commented on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666042867 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA removed a comment on pull request #29294: [SPARK-32160][CORE][PYSPARK][3.0] Add configs to switch allow/disallow to create SparkContext in executors.

SparkQA removed a comment on pull request #29294: URL: https://github.com/apache/spark/pull/29294#issuecomment-665989295 **[Test build #126794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126794/testReport)** for PR 29294 at commit

[GitHub] [spark] SparkQA commented on pull request #29146: [WIP][SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

SparkQA commented on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-665949787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins removed a comment on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666050019 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HeartSaVioR commented on pull request #29272: [SPARK-32468][SS][TESTS] Fix timeout config issue in Kafka connector tests

HeartSaVioR commented on pull request #29272: URL: https://github.com/apache/spark/pull/29272#issuecomment-666158718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] c21 commented on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

c21 commented on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666027985 @cloud-fan - updated the PR with addressing comments, and it is ready for review. Also updated the PR description for latest codegen code of example query. Thanks.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning

AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-665973679 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on pull request #29293: [SPARK-32487][CORE] Remove j.w.r.NotFoundException from `import` in [Stages|OneApplication]Resource

HyukjinKwon commented on pull request #29293: URL: https://github.com/apache/spark/pull/29293#issuecomment-666017984 Nice, LGTM This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] HyukjinKwon commented on pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

HyukjinKwon commented on pull request #29270: URL: https://github.com/apache/spark/pull/29270#issuecomment-666050065 Hm, do we really need to test all plan as string output? It's going to make backporting very difficult whenever we make changes in the plans. It's more difficult because we

[GitHub] [spark] SparkQA commented on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

SparkQA commented on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665849095 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA removed a comment on pull request #29291: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

SparkQA removed a comment on pull request #29291: URL: https://github.com/apache/spark/pull/29291#issuecomment-666042867 **[Test build #126797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126797/testReport)** for PR 29291 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

cloud-fan commented on a change in pull request #29277: URL: https://github.com/apache/spark/pull/29277#discussion_r462759609 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoinExec.scala ## @@ -70,4 +74,54 @@ case class

[GitHub] [spark] Ngone51 commented on a change in pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

Ngone51 commented on a change in pull request #29270: URL: https://github.com/apache/spark/pull/29270#discussion_r462769211 ## File path: sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala ## @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] SparkQA removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning

SparkQA removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-665973326 **[Test build #126791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126791/testReport)** for PR 29211 at commit

[GitHub] [spark] dbtsai closed pull request #29274: [SPARK-32397][BUILD] Allow specifying of time for build to keep time consistent between modules

dbtsai closed pull request #29274: URL: https://github.com/apache/spark/pull/29274 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] SparkQA commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

SparkQA commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666055382 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] MaxGekk commented on pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

MaxGekk commented on pull request #29234: URL: https://github.com/apache/spark/pull/29234#issuecomment-666132055 @HyukjinKwon @cloud-fan Could you review this PR. This is an automated message from the Apache Git Service. To

[GitHub] [spark] dbtsai commented on pull request #29274: [SPARK-32397][BUILD] Allow specifying of time for build to keep time consistent between modules

dbtsai commented on pull request #29274: URL: https://github.com/apache/spark/pull/29274#issuecomment-665944705 Thanks all for reviewing. This will be very useful to release snapshot jars for people to depend on and try out the snapshot release. Merged into master, 3.0, and 2.4 branches

[GitHub] [spark] AmplabJenkins commented on pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

AmplabJenkins commented on pull request #29067: URL: https://github.com/apache/spark/pull/29067#issuecomment-665828006 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

SparkQA removed a comment on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-665949787 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

AmplabJenkins removed a comment on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665850566 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

cloud-fan commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r462761942 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -244,11 +258,31 @@ statement | SET TIME ZONE

[GitHub] [spark] gatorsmile commented on a change in pull request #29146: [SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

gatorsmile commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r462777127 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala ## @@ -61,6 +63,64 @@ class SparkSqlParserSuite

[GitHub] [spark] SparkQA commented on pull request #29278: [WIP][SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.

SparkQA commented on pull request #29278: URL: https://github.com/apache/spark/pull/29278#issuecomment-665970816 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] HyukjinKwon commented on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

HyukjinKwon commented on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-666014185 Thank you @dongjoon-hyun. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] huaxingao closed pull request #29255: [SPARK-32455][ML] LogisticRegressionModel prediction optimization

huaxingao closed pull request #29255: URL: https://github.com/apache/spark/pull/29255 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] attilapiros commented on pull request #29090: [SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

attilapiros commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-665952684 Thanks @holdenk for looking into this. And what about logging out a warning when no unit is given? Like: "Memory setting without explicit unit (${value})

[GitHub] [spark] SparkQA removed a comment on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

SparkQA removed a comment on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665994152 **[Test build #126795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126795/testReport)** for PR 29295 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

SparkQA removed a comment on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-666048818 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] uncleGen commented on a change in pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

uncleGen commented on a change in pull request #28781: URL: https://github.com/apache/spark/pull/28781#discussion_r462696730 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/StreamingQueryStatusStore.scala ## @@ -0,0 +1,60 @@ +/* + * Licensed to the

[GitHub] [spark] SparkQA commented on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

SparkQA commented on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665994152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] cloud-fan commented on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

cloud-fan commented on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-666158610 Does it qualify a backport? It's kind of a new feature. This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon commented on pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

HyukjinKwon commented on pull request #29298: URL: https://github.com/apache/spark/pull/29298#issuecomment-666170687 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] SparkQA removed a comment on pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

SparkQA removed a comment on pull request #29067: URL: https://github.com/apache/spark/pull/29067#issuecomment-665690606 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-666133472 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support

AmplabJenkins removed a comment on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-666049254 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon closed pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

HyukjinKwon closed pull request #28968: URL: https://github.com/apache/spark/pull/28968 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] holdenk commented on pull request #29274: [SPARK-32397][BUILD] Allow specifying of time for build to keep time consistent between modules

holdenk commented on pull request #29274: URL: https://github.com/apache/spark/pull/29274#issuecomment-665819306 Thanks @HyukjinKwon I've added that it impacts `maven deploy` to the description. This is an automated message

[GitHub] [spark] jiangxb1987 commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

jiangxb1987 commented on pull request #29228: URL: https://github.com/apache/spark/pull/29228#issuecomment-666137629 It would be really great if you can list the test cases/suites that could get simplified by this change, thanks!

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

dongjoon-hyun commented on a change in pull request #29298: URL: https://github.com/apache/spark/pull/29298#discussion_r462780182 ## File path: core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json ## @@ -5,283 +5,283 @@ "scalaVersion" :

[GitHub] [spark] dongjoon-hyun commented on pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

dongjoon-hyun commented on pull request #29298: URL: https://github.com/apache/spark/pull/29298#issuecomment-666169269 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

SparkQA removed a comment on pull request #29276: URL: https://github.com/apache/spark/pull/29276#issuecomment-665792999 **[Test build #126786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126786/testReport)** for PR 29276 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29283: [SPARK-32478][R][SQL] Error message to show the schema mismatch in gapply with Arrow vectorization

SparkQA removed a comment on pull request #29283: URL: https://github.com/apache/spark/pull/29283#issuecomment-666047693 **[Test build #126798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126798/testReport)** for PR 29283 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29146: [WIP][SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

AmplabJenkins commented on pull request #29146: URL: https://github.com/apache/spark/pull/29146#issuecomment-665946574 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29296: [SPARK-32488][SQL] Use @parser::members and @lexer::members to avoid generating unused code

AmplabJenkins commented on pull request #29296: URL: https://github.com/apache/spark/pull/29296#issuecomment-666050019 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

SparkQA commented on pull request #29234: URL: https://github.com/apache/spark/pull/29234#issuecomment-665941972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] gemelen commented on pull request #29286: [WIP}[SPARK-21708][Build] Migrate build to sbt 1.x

gemelen commented on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-666101286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] dongjoon-hyun closed pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

dongjoon-hyun closed pull request #29295: URL: https://github.com/apache/spark/pull/29295 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #29293: [SPARK-32487][CORE] Remove j.w.r.NotFoundException from `import` in [Stages|OneApplication]Resource

SparkQA commented on pull request #29293: URL: https://github.com/apache/spark/pull/29293#issuecomment-665941955 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] viirya commented on a change in pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

viirya commented on a change in pull request #29277: URL: https://github.com/apache/spark/pull/29277#discussion_r462685419 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala ## @@ -903,6 +904,10 @@ case class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

AmplabJenkins removed a comment on pull request #29276: URL: https://github.com/apache/spark/pull/29276#issuecomment-665903672 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] yaooqinn commented on pull request #29297: [SPARK-32406][SQL][FOLLOWUP] Make RESET fail against static and core configs

yaooqinn commented on pull request #29297: URL: https://github.com/apache/spark/pull/29297#issuecomment-666151668 cc @gatorsmile @cloud-fan @dongjoon-hyun @maropu thanks very much. This is an automated message from the

[GitHub] [spark] maropu commented on a change in pull request #29146: [WIP][SPARK-32257][SQL] Reports explicit errors for invalid usage of SET/RESET command

maropu commented on a change in pull request #29146: URL: https://github.com/apache/spark/pull/29146#discussion_r462610033 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -244,11 +258,31 @@ statement | SET TIME ZONE

[GitHub] [spark] dongjoon-hyun commented on pull request #29287: [SPARK-27830][CORE][UI][2.4] Show Spark version at app lists of Spark History UI

dongjoon-hyun commented on pull request #29287: URL: https://github.com/apache/spark/pull/29287#issuecomment-666171021 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

cloud-fan commented on pull request #29270: URL: https://github.com/apache/spark/pull/29270#issuecomment-666098964 One of my worries is that: this test generates plans with empty tables, and we lost test coverage for things like SMJ. Can the `variant` feature help to improve the test

[GitHub] [spark] dongjoon-hyun commented on pull request #29295: [SPARK-32248][BUILD] Recover Java 11 build in Github Actions

dongjoon-hyun commented on pull request #29295: URL: https://github.com/apache/spark/pull/29295#issuecomment-665994816 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on pull request #25575: [SPARK-28818][SQL] Respect source column nullability in the arrays created by `freqItems()`

maropu commented on pull request #25575: URL: https://github.com/apache/spark/pull/25575#issuecomment-665984598 I checked that we couldn't cherry-pick the commit into branch-2.4. Looks okay to backport it (because this is a bug), so could you open a new PR for branch-2.4?

[GitHub] [spark] SparkQA removed a comment on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

SparkQA removed a comment on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666019778 **[Test build #126796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126796/testReport)** for PR 29277 at commit

[GitHub] [spark] dongjoon-hyun opened a new pull request #29298: [SPARK-32489][CORE] Pass `core` module UTs in Scala 2.13

dongjoon-hyun opened a new pull request #29298: URL: https://github.com/apache/spark/pull/29298 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[GitHub] [spark] HeartSaVioR removed a comment on pull request #29272: [SPARK-32468][SS][TESTS] Fix timeout config issue in Kafka connector tests

HeartSaVioR removed a comment on pull request #29272: URL: https://github.com/apache/spark/pull/29272#issuecomment-666160436 retest this, please This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] maropu commented on pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

maropu commented on pull request #29270: URL: https://github.com/apache/spark/pull/29270#issuecomment-666012926 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] Ngone51 commented on pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

Ngone51 commented on pull request #29270: URL: https://github.com/apache/spark/pull/29270#issuecomment-666071795 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] liucht-inspur commented on pull request #29287: [SPARK-27830][CORE][UI][2.4] Show Spark version at app lists of Spark History UI

liucht-inspur commented on pull request #29287: URL: https://github.com/apache/spark/pull/29287#issuecomment-666009260 Hi, @dongjoon-hyun . Thank you for your reminding. Can I resubmit a new JIRA for this 2.4 release issue?

[GitHub] [spark] viirya commented on a change in pull request #29234: [SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources

viirya commented on a change in pull request #29234: URL: https://github.com/apache/spark/pull/29234#discussion_r462557526 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/SchemaUtils.scala ## @@ -42,7 +42,27 @@ private[spark] object SchemaUtils { */

[GitHub] [spark] maryannxue commented on a change in pull request #29276: [SPARK-32470][CORE] Remove task result size check for shuffle map stage

maryannxue commented on a change in pull request #29276: URL: https://github.com/apache/spark/pull/29276#discussion_r462650727 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -695,7 +696,7 @@ private[spark] class TaskSetManager( def

[GitHub] [spark] SparkQA removed a comment on pull request #29262: [SPARK-32332][SQL] Support columnar exchanges

SparkQA removed a comment on pull request #29262: URL: https://github.com/apache/spark/pull/29262#issuecomment-665703985 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29277: [SPARK-32421][SQL] Add code-gen for shuffled hash join

SparkQA commented on pull request #29277: URL: https://github.com/apache/spark/pull/29277#issuecomment-666019778 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] LuciferYang opened a new pull request #29299: [SPARK-32490][BUILD] Upgrade netty-all to 4.1.51.Final

LuciferYang opened a new pull request #29299: URL: https://github.com/apache/spark/pull/29299 ### What changes were proposed in this pull request? This PR aims to bring the bug fixes from the latest netty version. ### Why are the changes needed? - 4.1.48.Final:

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28986: [SPARK-32160][CORE][PYSPARK] Disallow to create SparkContext in executors.

HyukjinKwon commented on a change in pull request #28986: URL: https://github.com/apache/spark/pull/28986#discussion_r462673731 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -2554,6 +2557,19 @@ object SparkContext extends Logging { } } +

<    2   3   4   5   6   7   8   >