[GitHub] [spark] MaxGekk opened a new pull request #28842: [WIP][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

MaxGekk opened a new pull request #28842: URL: https://github.com/apache/spark/pull/28842 ### What changes were proposed in this pull request? Move creation of `dateFormatter` and `timestampFormatter` from the constructor of the `HiveResult` object to `HiveResult. hiveResultString()`.

[GitHub] [spark] cloud-fan commented on pull request #28833: [SPARK-20680][SQL] Make null type in Spark sql to be compatible with Hive void datatype

cloud-fan commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644713571 To confirm: In Hive, people can't create tables with the void type (including void type inside struct/array/map). The only way is CTAS. Is this true? And how about

[GitHub] [spark] AmplabJenkins commented on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins commented on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644639855 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins removed a comment on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644639855 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

SparkQA removed a comment on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644579384 **[Test build #124109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124109/testReport)** for PR 28799 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28837: [SPARK-31996][BUILD] Specify the version of ChromeDriver and RemoteWebDriver which can work with guava 14.0.1

AmplabJenkins removed a comment on pull request #28837: URL: https://github.com/apache/spark/pull/28837#issuecomment-644648785 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644649807 **[Test build #124110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124110/testReport)** for PR 28710 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

SparkQA removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644579440 **[Test build #124110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124110/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28833: [SPARK-20680][SQL] Make null type in Spark sql to be compatible with Hive void datatype

AmplabJenkins commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644680800 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28833: [SPARK-20680][SQL] Make null type in Spark sql to be compatible with Hive void datatype

AmplabJenkins removed a comment on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644680800 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on a change in pull request #28123: [SPARK-31350][SQL] Coalesce bucketed tables for sort merge join if applicable

cloud-fan commented on a change in pull request #28123: URL: https://github.com/apache/spark/pull/28123#discussion_r440779860 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -165,6 +166,7 @@ case class FileSourceScanExec(

[GitHub] [spark] SparkQA removed a comment on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

SparkQA removed a comment on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644645073 **[Test build #124117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124117/testReport)** for PR 28799 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins commented on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644718711 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #27066: [SPARK-31317][SQL] Add withField method to Column

cloud-fan commented on a change in pull request #27066: URL: https://github.com/apache/spark/pull/27066#discussion_r440807378 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ## @@ -539,3 +541,82 @@ case class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644734177 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

SparkQA removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644663229 **[Test build #124118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124118/testReport)** for PR 28839 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644734177 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #27066: [SPARK-31317][SQL] Add withField method to Column

cloud-fan commented on a change in pull request #27066: URL: https://github.com/apache/spark/pull/27066#discussion_r440821924 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ## @@ -539,3 +541,82 @@ case class

[GitHub] [spark] sarutak commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

sarutak commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644740119 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] ulysses-you opened a new pull request #28840: [SPARK-31999][SQL] Add refresh function command

ulysses-you opened a new pull request #28840: URL: https://github.com/apache/spark/pull/28840 ### What changes were proposed in this pull request? In Hive mode, permanent functions are shared with Hive metastore so that functions may be modified by other Hive client.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28838: [SPARK-31997][SQL][TESTS] Drop test_udtf table when SingleSessionSuite test completed

AmplabJenkins removed a comment on pull request #28838: URL: https://github.com/apache/spark/pull/28838#issuecomment-644636724 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins commented on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644645738 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

SparkQA commented on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644645073 **[Test build #124117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124117/testReport)** for PR 28799 at commit

[GitHub] [spark] SparkQA commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

SparkQA commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644645074 **[Test build #124116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124116/testReport)** for PR 28839 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644650512 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path

AmplabJenkins commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-64455 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] maropu commented on pull request #28810: [SPARK-31705][SQL][FOLLOWUP] Avoid the unnecessary CNF computation for full-outer joins

maropu commented on pull request #28810: URL: https://github.com/apache/spark/pull/28810#issuecomment-644683866 Thanks, all! Pending, Jenkins. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins commented on pull request #28810: [SPARK-31705][SQL][FOLLOWUP] Avoid the unnecessary CNF computation for full-outer joins

AmplabJenkins commented on pull request #28810: URL: https://github.com/apache/spark/pull/28810#issuecomment-644684027 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28810: [SPARK-31705][SQL][FOLLOWUP] Avoid the unnecessary CNF computation for full-outer joins

AmplabJenkins removed a comment on pull request #28810: URL: https://github.com/apache/spark/pull/28810#issuecomment-644684027 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] LuciferYang commented on pull request #28838: [SPARK-31997][SQL][TESTS] Drop test_udtf table when SingleSessionSuite test completed

LuciferYang commented on pull request #28838: URL: https://github.com/apache/spark/pull/28838#issuecomment-644706929 thx ~ @wangyum @HyukjinKwon @xuanyuanking This is an automated message from the Apache Git Service. To

[GitHub] [spark] attilapiros commented on a change in pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata

attilapiros commented on a change in pull request #28618: URL: https://github.com/apache/spark/pull/28618#discussion_r440799753 ## File path: core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java ## @@ -273,19 +280,24 @@ void forceSorterToSpill() throws

[GitHub] [spark] bart-samwel commented on pull request #27805: [SPARK-31056][SQL] Add CalendarIntervals division

bart-samwel commented on pull request #27805: URL: https://github.com/apache/spark/pull/27805#issuecomment-644732209 > I am not aware of any SQL standard going this direction. I agree that the suggested UDFs give me the same result. FWIW, the SQL 2016 standard (section 4.6.4

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28824: [SPARK-31984][SQL] Make micros rebasing functions via local timestamps pure

AmplabJenkins removed a comment on pull request #28824: URL: https://github.com/apache/spark/pull/28824#issuecomment-644731927 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins removed a comment on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644639866 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] sarutak commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

sarutak commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644641005 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] sarutak commented on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

sarutak commented on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644641321 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] SparkQA commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

SparkQA commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-644640717 **[Test build #124115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124115/testReport)** for PR 28840 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644641588 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644641588 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644646541 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins removed a comment on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644645738 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

SparkQA commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644646523 **[Test build #124116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124116/testReport)** for PR 28839 at commit

[GitHub] [spark] HyukjinKwon closed pull request #28838: [SPARK-31997][SQL][TESTS] Drop test_udtf table when SingleSessionSuite test completed

HyukjinKwon closed pull request #28838: URL: https://github.com/apache/spark/pull/28838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AngersZhuuuu commented on pull request #27983: [SPARK-31936][SQL]Implement ScriptTransformation in sql/core

AngersZh commented on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-644675736 Is there anything need to update ？ This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] HyukjinKwon removed a comment on pull request #28824: [SPARK-31984][SQL] Make micros rebasing functions via local timestamps pure

HyukjinKwon removed a comment on pull request #28824: URL: https://github.com/apache/spark/pull/28824#issuecomment-644680044 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To

[GitHub] [spark] maropu commented on pull request #28833: [SPARK-20680][SQL] Make null type in Spark sql to be compatible with Hive void datatype

maropu commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644680233 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28799: [SPARK-31871][CORE][WEBUI][2.4] Display the canvas element icon for sorting column

AmplabJenkins removed a comment on pull request #28799: URL: https://github.com/apache/spark/pull/28799#issuecomment-644718711 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28824: [SPARK-31984][SQL] Make micros rebasing functions via local timestamps pure

SparkQA commented on pull request #28824: URL: https://github.com/apache/spark/pull/28824#issuecomment-644730917 **[Test build #124105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124105/testReport)** for PR 28824 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644734182 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644741981 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on pull request #28824: [SPARK-31984][SQL] Make micros rebasing functions via local timestamps pure

cloud-fan commented on pull request #28824: URL: https://github.com/apache/spark/pull/28824#issuecomment-644745025 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] MaxGekk commented on pull request #28593: [SPARK-31710][SQL] Fail casting numeric to timestamp by default

MaxGekk commented on pull request #28593: URL: https://github.com/apache/spark/pull/28593#issuecomment-644784156 After rebasing on the recent master, I faced to failures of `DateTimeBenchmark` because of this PR. I fixed the issue in the PR https://github.com/apache/spark/pull/28843

[GitHub] [spark] cloud-fan commented on pull request #28843: [SPARK-31710][SQL][FOLLOWUP] Replace CAST by TIMESTAMP_SECONDS in benchmarks

cloud-fan commented on pull request #28843: URL: https://github.com/apache/spark/pull/28843#issuecomment-644787724 thanks for fixing it! It's benchmark only so we don't need to wait for jenkins. merging to master!

[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path

cchighman commented on a change in pull request #28841: URL: https://github.com/apache/spark/pull/28841#discussion_r440891685 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala ## @@ -176,9 +180,27 @@ object

[GitHub] [spark] Ngone51 commented on a change in pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode

Ngone51 commented on a change in pull request #28746: URL: https://github.com/apache/spark/pull/28746#discussion_r440898639 ## File path: core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala ## @@ -63,23 +65,34 @@ class LocalSparkCluster( /* Start the

[GitHub] [spark] AmplabJenkins commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP][test-maven] Cleanup the thread local variable of hive metastore

AmplabJenkins commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-644827975 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP][test-maven] Cleanup the thread local variable of hive metastore

AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-644827975 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644839635 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644839635 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp

AmplabJenkins removed a comment on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644843670 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on pull request #28773: [SPARK-26905][SQL] Add `TYPE` in the ANSI non-reserved list

dongjoon-hyun commented on pull request #28773: URL: https://github.com/apache/spark/pull/28773#issuecomment-644844363 +1, late LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

SparkQA commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644844435 **[Test build #124124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124124/testReport)** for PR 28839 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp

AmplabJenkins commented on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644843670 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644845858 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] dongjoon-hyun commented on pull request #28807: [SPARK-26905][SQL] Follow the SQL:2016 reserved keywords

dongjoon-hyun commented on pull request #28807: URL: https://github.com/apache/spark/pull/28807#issuecomment-644847678 No problem. It looks okay, @maropu . This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

baohe-zhang commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r440955307 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala ## @@ -0,0 +1,82 @@ +/* + * Licensed to the

[GitHub] [spark] ulysses-you removed a comment on pull request #28778: [SPARK-31949][SQL] Add spark.default.parallelism in SQLConf for isolated across session

ulysses-you removed a comment on pull request #28778: URL: https://github.com/apache/spark/pull/28778#issuecomment-644768163 Yeah, parallelism is a physical concept, but it is also shared among sessions. I used a long-lived Spark application with enough core and memory (means

[GitHub] [spark] SparkQA commented on pull request #28843: [SPARK-31710][SQL][FOLLOWUP] Replace CAST by TIMESTAMP_SECONDS in benchmarks

SparkQA commented on pull request #28843: URL: https://github.com/apache/spark/pull/28843#issuecomment-644785511 **[Test build #124125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124125/testReport)** for PR 28843 at commit

[GitHub] [spark] SparkQA commented on pull request #28842: [WIP][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

SparkQA commented on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-644789874 **[Test build #124126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124126/testReport)** for PR 28842 at commit

[GitHub] [spark] cloud-fan closed pull request #28843: [SPARK-31710][SQL][FOLLOWUP] Replace CAST by TIMESTAMP_SECONDS in benchmarks

cloud-fan closed pull request #28843: URL: https://github.com/apache/spark/pull/28843 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28788: [SPARK-31960][Yarn][Build] Only populate Hadoop classpath for no-hadoop build

dongjoon-hyun commented on a change in pull request #28788: URL: https://github.com/apache/spark/pull/28788#discussion_r440899419 ## File path: docs/running-on-yarn.md ## @@ -82,6 +82,19 @@ In `cluster` mode, the driver runs on a different machine than the client, so `S

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644830265 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp

SparkQA commented on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644843087 **[Test build #124130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124130/testReport)** for PR 28821 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644842211 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

SparkQA removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644839019 **[Test build #124129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124129/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644842199 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] cloud-fan commented on pull request #28629: [SPARK-31769] Add MDC support for driver threads

cloud-fan commented on pull request #28629: URL: https://github.com/apache/spark/pull/28629#issuecomment-644747028 e.g. you set a custom MDC property via `SparkContext.setLocalProperty`, and change the log4j config file to use this MDC property. Then run a Spark application and check if

[GitHub] [spark] gaborgsomogyi edited a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files

gaborgsomogyi edited a comment on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-644757113 I agree, confusion comes from `latestFirst` basically. > But then should we really open the possibility to trace back older files? I see a use-case where

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28840: [SPARK-31999][SQL] Add refresh function command

AmplabJenkins removed a comment on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-644812026 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

AmplabJenkins commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-644812026 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

tgravescs commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r440918495 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala ## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache

[GitHub] [spark] SparkQA commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP][test-maven] Cleanup the thread local variable of hive metastore

SparkQA commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-644826816 **[Test build #124108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124108/testReport)** for PR 28835 at commit

[GitHub] [spark] huaxingao commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

huaxingao commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644827487 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] SparkQA removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP][test-maven] Cleanup the thread local variable of hive metastore

SparkQA removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-644579388 **[Test build #124108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124108/testReport)** for PR 28835 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644831728 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

SparkQA removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644829409 **[Test build #124128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124128/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644831728 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

AmplabJenkins removed a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-644845847 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] gaborgsomogyi commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files

gaborgsomogyi commented on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-644757113 I agree, confusion comes from `latestFirst` basically. > But then should we really open the possibility to trace back older files? I see a use-case where it's

[GitHub] [spark] gaborgsomogyi edited a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files

gaborgsomogyi edited a comment on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-644757113 I agree, confusion comes from `latestFirst` basically. > But then should we really open the possibility to trace back older files? I see a use-case where

[GitHub] [spark] AmplabJenkins commented on pull request #28836: [SPARK-31561][SQL] Add QUALIFY Clause

AmplabJenkins commented on pull request #28836: URL: https://github.com/apache/spark/pull/28836#issuecomment-644763548 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28836: [SPARK-31561][SQL] Add QUALIFY Clause

AmplabJenkins removed a comment on pull request #28836: URL: https://github.com/apache/spark/pull/28836#issuecomment-644763548 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #28836: [SPARK-31561][SQL] Add QUALIFY Clause

SparkQA removed a comment on pull request #28836: URL: https://github.com/apache/spark/pull/28836#issuecomment-644579385 **[Test build #124107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124107/testReport)** for PR 28836 at commit

[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

tgravescs commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r440858998 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] tgravescs commented on pull request #27843: [SPARK-31029] Avoid using global execution context in driver main thread for YarnSchedulerBackend

tgravescs commented on pull request #27843: URL: https://github.com/apache/spark/pull/27843#issuecomment-644786081 I didn't finish reviewing. Please update the description to describe the fix and add a comment by the line schedulerEndpointEC definition.

[GitHub] [spark] AmplabJenkins commented on pull request #28843: [SPARK-31710][SQL][FOLLOWUP] Replace CAST by TIMESTAMP_SECONDS in benchmarks

AmplabJenkins commented on pull request #28843: URL: https://github.com/apache/spark/pull/28843#issuecomment-644786372 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28843: [SPARK-31710][SQL][FOLLOWUP] Replace CAST by TIMESTAMP_SECONDS in benchmarks

AmplabJenkins removed a comment on pull request #28843: URL: https://github.com/apache/spark/pull/28843#issuecomment-644786372 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cchighman edited a comment on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path

cchighman edited a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-644793641 Thanks for your comments, @bart-samwel. I like your way of thinking, there are a lot of unique cases here. To provide more context behind the scenario I'm looking to

[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path

cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-644793641 Thanks for your comments, @bart-samwel. I like your way of thinking, there are a lot of unique cases here. Here's more context behind the scenario I'm looking to cover

[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide option to load files after a specified date when reading from a folder path

cchighman commented on a change in pull request #28841: URL: https://github.com/apache/spark/pull/28841#discussion_r440893695 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala ## @@ -356,26 +380,35 @@ object

<    1   2   3   4   5   6   7   >