[GitHub] [spark] gerashegalov commented on a change in pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode

gerashegalov commented on a change in pull request #28746: URL: https://github.com/apache/spark/pull/28746#discussion_r437592878 ## File path: core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala ## @@ -74,6 +74,10 @@ class LocalSparkCluster( def stop():

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28764: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

AmplabJenkins removed a comment on pull request #28764: URL: https://github.com/apache/spark/pull/28764#issuecomment-641454235 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28764: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

AmplabJenkins commented on pull request #28764: URL: https://github.com/apache/spark/pull/28764#issuecomment-641454235 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28764: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

SparkQA removed a comment on pull request #28764: URL: https://github.com/apache/spark/pull/28764#issuecomment-641245723 **[Test build #123694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123694/testReport)** for PR 28764 at commit

[GitHub] [spark] SparkQA commented on pull request #28764: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

SparkQA commented on pull request #28764: URL: https://github.com/apache/spark/pull/28764#issuecomment-641453142 **[Test build #123694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123694/testReport)** for PR 28764 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF

AmplabJenkins removed a comment on pull request #28645: URL: https://github.com/apache/spark/pull/28645#issuecomment-641450746 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF

AmplabJenkins commented on pull request #28645: URL: https://github.com/apache/spark/pull/28645#issuecomment-641450746 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF

SparkQA removed a comment on pull request #28645: URL: https://github.com/apache/spark/pull/28645#issuecomment-641239491 **[Test build #123693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123693/testReport)** for PR 28645 at commit

[GitHub] [spark] SparkQA commented on pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF

SparkQA commented on pull request #28645: URL: https://github.com/apache/spark/pull/28645#issuecomment-641449469 **[Test build #123693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123693/testReport)** for PR 28645 at commit

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437581083 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala ## @@ -0,0 +1,82 @@ +/* + * Licensed to the

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437580626 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437580889 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] cloud-fan closed pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

cloud-fan closed pull request #28650: URL: https://github.com/apache/spark/pull/28650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437580420 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] cloud-fan commented on pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

cloud-fan commented on pull request #28650: URL: https://github.com/apache/spark/pull/28650#issuecomment-641443909 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437578109 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437578179 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437577822 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] cloud-fan commented on pull request #28751: [SPARK-31926][SQL][test-hive1.2] Fix concurrency issue for ThriftCLIService to getPortNumber

cloud-fan commented on pull request #28751: URL: https://github.com/apache/spark/pull/28751#issuecomment-641435280 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] cloud-fan closed pull request #28751: [SPARK-31926][SQL][test-hive1.2] Fix concurrency issue for ThriftCLIService to getPortNumber

cloud-fan closed pull request #28751: URL: https://github.com/apache/spark/pull/28751 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #28768: [SPARK-31941][CORE]Add the code change to handle the SparkException

SaurabhChawla100 commented on a change in pull request #28768: URL: https://github.com/apache/spark/pull/28768#discussion_r437290253 ## File path: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ## @@ -86,6 +86,7 @@ private[spark] class SparkUI private (

[GitHub] [spark] cloud-fan commented on pull request #28745: [SPARK-31915][SQL][PYTHON] Remove projection that adds grouping keys in grouped and cogrouped pandas UDFs

cloud-fan commented on pull request #28745: URL: https://github.com/apache/spark/pull/28745#issuecomment-641432882 will this cause perf regression? e.g. if grouping expr is expensive, with the Project we only need to evaluate it once.

[GitHub] [spark] cloud-fan closed pull request #27986: [SPARK-31220][SQL] repartition obeys initialPartitionNum when adaptiveExecutionEnabled

cloud-fan closed pull request #27986: URL: https://github.com/apache/spark/pull/27986 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #27986: [SPARK-31220][SQL] repartition obeys initialPartitionNum when adaptiveExecutionEnabled

cloud-fan commented on pull request #27986: URL: https://github.com/apache/spark/pull/27986#issuecomment-641404167 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] cloud-fan closed pull request #28765: [SPARK-29295][SQL][FOLLOWUP] Dynamic partition map parsed from partition path should be case insensitive

cloud-fan closed pull request #28765: URL: https://github.com/apache/spark/pull/28765 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA commented on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

SparkQA commented on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-641405101 **[Test build #123699 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123699/testReport)** for PR 27246 at commit

[GitHub] [spark] sarutak commented on a change in pull request #28768: [SPARK-31941][CORE][WEBUI] Handle SparkException in SparkUI

sarutak commented on a change in pull request #28768: URL: https://github.com/apache/spark/pull/28768#discussion_r437537787 ## File path: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ## @@ -86,7 +86,7 @@ private[spark] class SparkUI private (

[GitHub] [spark] AmplabJenkins commented on pull request #28751: [SPARK-31926][SQL][test-hive1.2] Fix concurrency issue for ThriftCLIService to getPortNumber

AmplabJenkins commented on pull request #28751: URL: https://github.com/apache/spark/pull/28751#issuecomment-641405232 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28751: [SPARK-31926][SQL][test-hive1.2] Fix concurrency issue for ThriftCLIService to getPortNumber

AmplabJenkins removed a comment on pull request #28751: URL: https://github.com/apache/spark/pull/28751#issuecomment-641405232 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #27983: [SPARK-31936][SQL]Implement ScriptTransformation in sql/core

AngersZh commented on a change in pull request #27983: URL: https://github.com/apache/spark/pull/27983#discussion_r437558157 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/script/ScriptTransformationExec.scala ## @@ -0,0 +1,239 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA commented on pull request #28768: [SPARK-31941][CORE][WEBUI] Handle SparkException in SparkUI

SparkQA commented on pull request #28768: URL: https://github.com/apache/spark/pull/28768#issuecomment-641406226 **[Test build #123700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123700/testReport)** for PR 28768 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

AmplabJenkins commented on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-641405643 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun closed pull request #28742: [SPARK-31921][CORE] Fix the wrong warning: "App app-xxx requires more resource than any of Workers could have"

dongjoon-hyun closed pull request #28742: URL: https://github.com/apache/spark/pull/28742 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

AmplabJenkins commented on pull request #28650: URL: https://github.com/apache/spark/pull/28650#issuecomment-641414274 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on pull request #28765: [SPARK-29295][SQL][FOLLOWUP] Dynamic partition map parsed from partition path should be case insensitive

cloud-fan commented on pull request #28765: URL: https://github.com/apache/spark/pull/28765#issuecomment-641398316 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA removed a comment on pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

SparkQA removed a comment on pull request #28650: URL: https://github.com/apache/spark/pull/28650#issuecomment-641202467 **[Test build #123689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123689/testReport)** for PR 28650 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

AmplabJenkins removed a comment on pull request #28650: URL: https://github.com/apache/spark/pull/28650#issuecomment-641414274 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

SparkQA removed a comment on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-641312310 **[Test build #123699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123699/testReport)** for PR 27246 at commit

[GitHub] [spark] yaooqinn edited a comment on pull request #28751: [SPARK-31926][SQL] Fix concurrency issue for ThriftCLIService to getPortNumber

yaooqinn edited a comment on pull request #28751: URL: https://github.com/apache/spark/pull/28751#issuecomment-640457288 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

AmplabJenkins removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-640957576 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] yaooqinn commented on a change in pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

yaooqinn commented on a change in pull request #28650: URL: https://github.com/apache/spark/pull/28650#discussion_r437232828 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -56,6 +54,26 @@ trait

[GitHub] [spark] SparkQA commented on pull request #28650: [SPARK-31830][SQL] Consistent error handling for datetime formatting and parsing functions

SparkQA commented on pull request #28650: URL: https://github.com/apache/spark/pull/28650#issuecomment-641412878 **[Test build #123689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123689/testReport)** for PR 28650 at commit

[GitHub] [spark] jkpl closed pull request #28753: [SPARK-31927][K8S] Bump Fabric8 to 4.10.2 to support k8s 1.18

jkpl closed pull request #28753: URL: https://github.com/apache/spark/pull/28753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] srowen commented on pull request #27805: [SPARK-31056][SQL] Add CalendarIntervals division

srowen commented on pull request #27805: URL: https://github.com/apache/spark/pull/27805#issuecomment-640611674 Right, "24 hours" is too simplistic an example. I get that "1 year 2 months" is just months. I think the arguments still stand: non-standard, limited scope, and achievable with

[GitHub] [spark] wangyum commented on pull request #28734: [SPARK-31912][SQL][TESTS] Normalize all binary comparison expressions

wangyum commented on pull request #28734: URL: https://github.com/apache/spark/pull/28734#issuecomment-641355129 I encountered this issue when inferring from inequality attributes before, Its order may change:

[GitHub] [spark] tgravescs commented on pull request #27636: [SPARK-30873][CORE][YARN]Handling Node Decommissioning for Yarn cluster manger in Spark

tgravescs commented on pull request #27636: URL: https://github.com/apache/spark/pull/27636#issuecomment-640883318 sorry I haven't had time to look at this more. It will not go into spark 3.0 as that is very close to shipping.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28743: [SPARK-31920][PYTHON][WIP] Fix pandas conversion using Arrow with __arrow_array__ columns

AmplabJenkins removed a comment on pull request #28743: URL: https://github.com/apache/spark/pull/28743#issuecomment-640721541 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan closed pull request #28763: [SPARK-28199][SS][FOLLOWUP] Mention the change of SPARK-28199 into the SS migration guide

cloud-fan closed pull request #28763: URL: https://github.com/apache/spark/pull/28763 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] AmplabJenkins commented on pull request #28759: [SPARK-31934][BUILD] Remove set -x from docker image tool

AmplabJenkins commented on pull request #28759: URL: https://github.com/apache/spark/pull/28759#issuecomment-640931144 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on a change in pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

maropu commented on a change in pull request #27246: URL: https://github.com/apache/spark/pull/27246#discussion_r437068044 ## File path: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java ## @@ -47,55 +47,48 @@ private int

[GitHub] [spark] bart-samwel commented on pull request #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp

bart-samwel commented on pull request #24195: URL: https://github.com/apache/spark/pull/24195#issuecomment-640702776 @MaxGekk How are we on the rebase function ("get me the same date + hands of the clock in a different timezone") that we talked about at some point?

[GitHub] [spark] fqaiser94 commented on pull request #27066: [SPARK-31317][SQL] Add withField method to Column

fqaiser94 commented on pull request #27066: URL: https://github.com/apache/spark/pull/27066#issuecomment-640747267 @cloud-fan Thanks for your PR. I've merged your changes into this branch with some modifications in order to pass the tests. > Remove the SQL API. It's unclear

[GitHub] [spark] srowen commented on pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode

srowen commented on pull request #28258: URL: https://github.com/apache/spark/pull/28258#issuecomment-640626064 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] attilapiros commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

attilapiros commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r436664435 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -148,6 +170,87 @@ private[spark] class

[GitHub] [spark] iRakson commented on a change in pull request #28748: [WIP][SPARK-30119][WEBUI]Support pagination for streaming tab

iRakson commented on a change in pull request #28748: URL: https://github.com/apache/spark/pull/28748#discussion_r437339866 ## File path: streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala ## @@ -125,24 +125,37 @@ class UISeleniumSuite //

[GitHub] [spark] wangyum commented on pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

wangyum commented on pull request #18323: URL: https://github.com/apache/spark/pull/18323#issuecomment-640956580 @maropu Please take it over. Thank you. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] SaurabhChawla100 commented on pull request #28768: [SPARK-31941][CORE]Add the code change to handle the SparkException

SaurabhChawla100 commented on pull request #28768: URL: https://github.com/apache/spark/pull/28768#issuecomment-641181106 cc @HeartSaVioR, @srowen @HyukjinKwon @dongjoon-hyun Please review this PR This is an

[GitHub] [spark] HeartSaVioR opened a new pull request #28763: [SPARK-28199][SS][FOLLOWUP] Mention the change of SPARK-28199 into the SS migration guide

HeartSaVioR opened a new pull request #28763: URL: https://github.com/apache/spark/pull/28763 ### What changes were proposed in this pull request? SPARK-28199 (#24996) made the trigger related public API to be exposed only from static methods of Trigger class. This is backward

[GitHub] [spark] SparkQA removed a comment on pull request #28743: [SPARK-31920][PYTHON] Fix pandas conversion using Arrow with __arrow_array__ columns

SparkQA removed a comment on pull request #28743: URL: https://github.com/apache/spark/pull/28743#issuecomment-640724799 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28750: [SPARK-31916][SQL] StringConcat can overflow , leads to StringIndexOutOfBoundsException

SparkQA removed a comment on pull request #28750: URL: https://github.com/apache/spark/pull/28750#issuecomment-640450409 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] prakharjain09 edited a comment on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors

prakharjain09 edited a comment on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-641023747 @holdenk @jiangxb1987 @Dooyoung-Hwang Please review the changes. > LGTM expect 2 minor comments, cc @holdenk @jiangxb1987

[GitHub] [spark] HyukjinKwon edited a comment on pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

HyukjinKwon edited a comment on pull request #28752: URL: https://github.com/apache/spark/pull/28752#issuecomment-641018513 @gengliangwang too FYI This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] maropu opened a new pull request #28764: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

maropu opened a new pull request #28764: URL: https://github.com/apache/spark/pull/28764 ### What changes were proposed in this pull request? This PR intends to add a build-in SQL function - `WIDTH_BUCKET`. It is the rework of #28148 Closes #18323 The

[GitHub] [spark] HyukjinKwon commented on a change in pull request #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users

HyukjinKwon commented on a change in pull request #24996: URL: https://github.com/apache/spark/pull/24996#discussion_r437126007 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Triggers.scala ## @@ -17,13 +17,94 @@ package

[GitHub] [spark] yaooqinn commented on pull request #28751: [SPARK-31926][SQL][test-hive1.2] Fix concurrency issue for ThriftCLIService to getPortNumber

yaooqinn commented on pull request #28751: URL: https://github.com/apache/spark/pull/28751#issuecomment-641246775 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] tgravescs commented on a change in pull request #28742: [SPARK-31921][CORE] Fix the wrong warning: "App app-xxx requires more resource than any of Workers could have"

tgravescs commented on a change in pull request #28742: URL: https://github.com/apache/spark/pull/28742#discussion_r436746988 ## File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ## @@ -715,7 +715,9 @@ private[deploy] class Master( val

[GitHub] [spark] zhengruifeng commented on a change in pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

zhengruifeng commented on a change in pull request #28710: URL: https://github.com/apache/spark/pull/28710#discussion_r437111620 ## File path: mllib/src/main/scala/org/apache/spark/ml/classification/ClassificationSummary.scala ## @@ -0,0 +1,265 @@ +/* + * Licensed to the

[GitHub] [spark] SparkQA removed a comment on pull request #27986: [SPARK-31220][SQL] repartition obeys initialPartitionNum when adaptiveExecutionEnabled

SparkQA removed a comment on pull request #27986: URL: https://github.com/apache/spark/pull/27986#issuecomment-640462670 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28742: [SPARK-31921][CORE] Fix the wrong warning: "App app-xxx requires more resource than any of Workers could have"

SparkQA commented on pull request #28742: URL: https://github.com/apache/spark/pull/28742#issuecomment-640763282 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] wangyum closed pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH_BUCKET

wangyum closed pull request #18323: URL: https://github.com/apache/spark/pull/18323 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] SparkQA commented on pull request #28743: [SPARK-31920][PYTHON] Fix pandas conversion using Arrow with __arrow_array__ columns

SparkQA commented on pull request #28743: URL: https://github.com/apache/spark/pull/28743#issuecomment-640724799 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] Ngone51 opened a new pull request #28756: [SPARK-8981]

Ngone51 opened a new pull request #28756: URL: https://github.com/apache/spark/pull/28756 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

redsanket commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r437529862 ## File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ## @@ -1197,6 +1213,71 @@ private[history] class

[GitHub] [spark] yaooqinn opened a new pull request #28766: [SPARK-31939][SQL] Fix Parsing day of year when year field pattern is missing

yaooqinn opened a new pull request #28766: URL: https://github.com/apache/spark/pull/28766 ### What changes were proposed in this pull request? If a datetime pattern contains no year field, the day of year field should not be ignored if exists e.g. ``` spark-sql>

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28763: [SPARK-28199][SS][FOLLOWUP] Mention the change of SPARK-28199 into the SS migration guide

AmplabJenkins removed a comment on pull request #28763: URL: https://github.com/apache/spark/pull/28763#issuecomment-641003776 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

SparkQA removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-640957296 **[Test build #123661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123661/testReport)** for PR 28616 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

AmplabJenkins removed a comment on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-641313089 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28755: [SPARK-29394][SQL]Support ISO 8601 format for intervals

AmplabJenkins removed a comment on pull request #28755: URL: https://github.com/apache/spark/pull/28755#issuecomment-640563320 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #27518: [SPARK-30768][SQL] Constraints inferred from inequality attributes

SparkQA commented on pull request #27518: URL: https://github.com/apache/spark/pull/27518#issuecomment-640641799 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] gengliangwang commented on a change in pull request #28733: [SPARK-31705][SQL] Push more possible predicates through Join via CNF conversion

gengliangwang commented on a change in pull request #28733: URL: https://github.com/apache/spark/pull/28733#discussion_r437275039 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala ## @@ -198,6 +199,88 @@ trait PredicateHelper

[GitHub] [spark] SparkQA commented on pull request #28750: [SPARK-31916][SQL] StringConcat can overflow , leads to StringIndexOutOfBoundsException

SparkQA commented on pull request #28750: URL: https://github.com/apache/spark/pull/28750#issuecomment-640582030 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

SparkQA commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-640957296 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] asfgit closed pull request #28744: [SPARK-31923][Core]Ignore internal accumulators that use unrecognized types rather than crashing

asfgit closed pull request #28744: URL: https://github.com/apache/spark/pull/28744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] srowen commented on pull request #25715: [SPARK-28969][PYTHON][ML] OneVsRestParams parity between scala and python

srowen commented on pull request #25715: URL: https://github.com/apache/spark/pull/25715#issuecomment-640768812 I'd guess release notes for things like this (and it's marked with release notes). Migration guide is more for when there is a non-obvious action you need to take to deal with a

[GitHub] [spark] SparkQA commented on pull request #28762: [SPARK-31849][PYTHON][SQL][FOLLOW-UP] More correct error message in Python UDF exception message

SparkQA commented on pull request #28762: URL: https://github.com/apache/spark/pull/28762#issuecomment-640964944 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA commented on pull request #28755: [SPARK-29394][SQL]Support ISO 8601 format for intervals

SparkQA commented on pull request #28755: URL: https://github.com/apache/spark/pull/28755#issuecomment-640950793 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28768: [SPARK-31941][CORE]Add the code change to handle the SparkException

AmplabJenkins removed a comment on pull request #28768: URL: https://github.com/apache/spark/pull/28768#issuecomment-641176523 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors

SparkQA commented on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-641022846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] maropu commented on a change in pull request #28750: [SPARK-31916][SQL] StringConcat can overflow , leads to StringIndexOutOfBoundsException

maropu commented on a change in pull request #28750: URL: https://github.com/apache/spark/pull/28750#discussion_r437059411 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -3503,6 +3504,25 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] cloud-fan commented on pull request #28770: [SPARK-31942] Revert "[SPARK-31864][SQL] Adjust AQE skew join trigger condition

cloud-fan commented on pull request #28770: URL: https://github.com/apache/spark/pull/28770#issuecomment-641244156 OK to test This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28768: [SPARK-31941][CORE]Add the code change to handle the SparkException

AmplabJenkins commented on pull request #28768: URL: https://github.com/apache/spark/pull/28768#issuecomment-641176523 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on a change in pull request #28737: [SPARK-31913][SQL]: Fix StackOverflowError in FileScanRDD

maropu commented on a change in pull request #28737: URL: https://github.com/apache/spark/pull/28737#discussion_r437383349 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileScanRDDSuite.scala ## @@ -0,0 +1,57 @@ +/* + * Licensed to the

[GitHub] [spark] amit-hitachi commented on pull request #17953: [SPARK-20680][SQL] Spark-sql do not support for void column datatype …

amit-hitachi commented on pull request #17953: URL: https://github.com/apache/spark/pull/17953#issuecomment-641068212 @LantaoJin @gatorsmile Any plans to merge this fix? This is an automated message from the Apache Git

[GitHub] [spark] wangyum commented on a change in pull request #28733: [SPARK-31705][SQL] Push more possible predicates through Join via CNF conversion

wangyum commented on a change in pull request #28733: URL: https://github.com/apache/spark/pull/28733#discussion_r437528590 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -545,6 +545,19 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

cloud-fan commented on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-641310017 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] manuzhang opened a new pull request #28770: [SPARK-31942] Revert "[SPARK-31864][SQL] Adjust AQE skew join trigger condition

manuzhang opened a new pull request #28770: URL: https://github.com/apache/spark/pull/28770 ### What changes were proposed in this pull request? This reverts commit https://github.com/apache/spark/commit/b9737c3c228f465d332e41f1ea0cece2a5f7667e while keeping following changes *

[GitHub] [spark] prakharjain09 commented on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors

prakharjain09 commented on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-641023747 @holdenk @jiangxb1987 Please review the changes. > LGTM expect 2 minor comments, cc @holdenk @jiangxb1987

[GitHub] [spark] Fokko commented on pull request #28754: [SPARK-10520][SQL] Allow average out of DateType

Fokko commented on pull request #28754: URL: https://github.com/apache/spark/pull/28754#issuecomment-640558015 Required for: https://github.com/apache/spark/pull/28754 This is an automated message from the Apache Git

[GitHub] [spark] cloud-fan commented on a change in pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF

cloud-fan commented on a change in pull request #28645: URL: https://github.com/apache/spark/pull/28645#discussion_r437349734 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -2847,6 +2848,45 @@ class Analyzer( } }

[GitHub] [spark] HyukjinKwon commented on a change in pull request #27598: [SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn

HyukjinKwon commented on a change in pull request #27598: URL: https://github.com/apache/spark/pull/27598#discussion_r437088507 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -635,7 +635,12 @@ private[spark] class Client(

<    1   2   3   4   5   6   7   >