[GitHub] [spark] ScrapCodes commented on pull request #29334: [SPARK-32495][2.4] Update jackson versions to a maintained release, to fix various security vulnerabilities.

ScrapCodes commented on pull request #29334: URL: https://github.com/apache/spark/pull/29334#issuecomment-678919877 Thank you @cowtowncoder, @srowen and @Fokko. Indeed, the Security vulnerabilities serve the purpose of generating the false alarm only and do not apply to spark, however if

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678918248 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678918248 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on a change in pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

maropu commented on a change in pull request #29421: URL: https://github.com/apache/spark/pull/29421#discussion_r475356961 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -182,7 +182,11 @@ class

[GitHub] [spark] SparkQA commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

SparkQA commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678917855 **[Test build #127830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127830/testReport)** for PR 29421 at commit

[GitHub] [spark] maropu commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

maropu commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678916433 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] maropu commented on a change in pull request #29485: [SPARK-32638][SQL] Corrects references when adding aliases in WidenSetOperationTypes

maropu commented on a change in pull request #29485: URL: https://github.com/apache/spark/pull/29485#discussion_r475352241 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ## @@ -328,27 +328,46 @@ object TypeCoercion { */

[GitHub] [spark] cloud-fan commented on a change in pull request #29485: [SPARK-32638][SQL] Corrects references when adding aliases in WidenSetOperationTypes

cloud-fan commented on a change in pull request #29485: URL: https://github.com/apache/spark/pull/29485#discussion_r475348817 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ## @@ -328,27 +328,46 @@ object TypeCoercion {

[GitHub] [spark] maropu commented on a change in pull request #29485: [SPARK-32638][SQL] Corrects references when adding aliases in WidenSetOperationTypes

maropu commented on a change in pull request #29485: URL: https://github.com/apache/spark/pull/29485#discussion_r475347837 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ## @@ -328,27 +328,46 @@ object TypeCoercion { */

[GitHub] [spark] viirya commented on a change in pull request #29485: [SPARK-32638][SQL] Corrects references when adding aliases in WidenSetOperationTypes

viirya commented on a change in pull request #29485: URL: https://github.com/apache/spark/pull/29485#discussion_r475347688 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ## @@ -328,27 +328,46 @@ object TypeCoercion { */

[GitHub] [spark] tanelk commented on pull request #29515: [WIP][SPARK-32688][SQL][TESTS] Add special values to LiteralGenerator for float and double

tanelk commented on pull request #29515: URL: https://github.com/apache/spark/pull/29515#issuecomment-678908920 There is a `org.apache.spark.sql.RandomDataGenerator`, that does pretty much the same thing as the `LiteralGenerator`. Perhaps they should be unified?

[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-678908616 I intend to update the PR based on comments, I'll try to swing around to it this evening. This is an

[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

cchighman commented on a change in pull request #28841: URL: https://github.com/apache/spark/pull/28841#discussion_r475347166 ## File path: docs/sql-data-sources-generic-options.md ## @@ -119,3 +119,48 @@ To load all files recursively, you can use: {% include_example

[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

AmplabJenkins commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-678908136 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] maropu commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

maropu commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678907495 Ah, I see. This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29485: [SPARK-32638][SQL] Corrects references when adding aliases in WidenSetOperationTypes

cloud-fan commented on a change in pull request #29485: URL: https://github.com/apache/spark/pull/29485#discussion_r475346278 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ## @@ -328,27 +328,46 @@ object TypeCoercion {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

AmplabJenkins removed a comment on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678907031 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] viirya commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

viirya commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678906923 @maropu I think #29406 was only merged to master, so we don't need to backport this. This is an automated

[GitHub] [spark] AmplabJenkins commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

AmplabJenkins commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678907031 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

SparkQA removed a comment on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678844753 **[Test build #127819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127819/testReport)** for PR 29526 at commit

[GitHub] [spark] SparkQA commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

SparkQA commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678906358 **[Test build #127819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127819/testReport)** for PR 29526 at commit

[GitHub] [spark] maropu commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

maropu commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678906274 Merged to master. @viirya Looks like conflicts with bnrahc3.0. Could you backport it? This is an automated

[GitHub] [spark] maropu edited a comment on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

maropu edited a comment on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678906274 Merged to master. @viirya Looks like conflicts with branch-3.0. Could you backport it? This is an

[GitHub] [spark] viirya commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

viirya commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678906138 Thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu closed pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

maropu closed pull request #29526: URL: https://github.com/apache/spark/pull/29526 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

AmplabJenkins removed a comment on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678905843 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

AmplabJenkins commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678905843 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

SparkQA removed a comment on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678845961 **[Test build #127820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127820/testReport)** for PR 29526 at commit

[GitHub] [spark] SparkQA commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

SparkQA commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678905088 **[Test build #127820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127820/testReport)** for PR 29526 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678903482 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

SparkQA commented on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-678903820 **[Test build #127829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127829/testReport)** for PR 29414 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678903475 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

SparkQA removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678859868 **[Test build #127823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127823/testReport)** for PR 29421 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678903475 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] srowen commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

srowen commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678903211 Oh yeah, to backport, you would need to check out branch-3.0, cherry-pick the commit, and the push straight to branch-3.0. It's not hard, just doesn't use the script (I don't

[GitHub] [spark] SparkQA commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

SparkQA commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678903206 **[Test build #127823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127823/testReport)** for PR 29421 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

AngersZh commented on a change in pull request #29414: URL: https://github.com/apache/spark/pull/29414#discussion_r475341428 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,224 @@ +-- Automatically generated by SQLQueryTestSuite

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

AmplabJenkins removed a comment on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-678902339 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

AmplabJenkins commented on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-678902339 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

AngersZh commented on a change in pull request #29414: URL: https://github.com/apache/spark/pull/29414#discussion_r475341428 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,224 @@ +-- Automatically generated by SQLQueryTestSuite

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

AmplabJenkins removed a comment on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678900712 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

AmplabJenkins removed a comment on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678900708 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

SparkQA removed a comment on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678870379 **[Test build #127824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127824/testReport)** for PR 29516 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

AmplabJenkins commented on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678900708 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] huaxingao commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

huaxingao commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678899420 I don't know how to merge this one. I got the following message: ``` Pull request 29501 is not mergeable in its current form. Continue? (experts only!) (y/n): ```

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29513: [SPARK-32646][SQL][3.0][test-hadoop2.7][test-hive1.2] ORC predicate pushdown should work with case-insensitive analysis

AmplabJenkins removed a comment on pull request #29513: URL: https://github.com/apache/spark/pull/29513#issuecomment-678897328 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29513: [SPARK-32646][SQL][3.0][test-hadoop2.7][test-hive1.2] ORC predicate pushdown should work with case-insensitive analysis

AmplabJenkins commented on pull request #29513: URL: https://github.com/apache/spark/pull/29513#issuecomment-678897328 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29513: [SPARK-32646][SQL][3.0][test-hadoop2.7][test-hive1.2] ORC predicate pushdown should work with case-insensitive analysis

SparkQA commented on pull request #29513: URL: https://github.com/apache/spark/pull/29513#issuecomment-678897095 **[Test build #127828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127828/testReport)** for PR 29513 at commit

[GitHub] [spark] viirya commented on pull request #29513: [SPARK-32646][SQL][3.0][test-hadoop2.7][test-hive1.2] ORC predicate pushdown should work with case-insensitive analysis

viirya commented on pull request #29513: URL: https://github.com/apache/spark/pull/29513#issuecomment-678896440 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29527: [SPARK-32664] fixes log level

AmplabJenkins removed a comment on pull request #29527: URL: https://github.com/apache/spark/pull/29527#issuecomment-678895387 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins commented on pull request #29527: [SPARK-32664] fixes log level

AmplabJenkins commented on pull request #29527: URL: https://github.com/apache/spark/pull/29527#issuecomment-678895672 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29527: [SPARK-32664] fixes log level

AmplabJenkins commented on pull request #29527: URL: https://github.com/apache/spark/pull/29527#issuecomment-678895387 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] srowen commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

srowen commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678895333 Go ahead yes This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dmoore62 opened a new pull request #29527: [SPARK-32664] fixes log level

dmoore62 opened a new pull request #29527: URL: https://github.com/apache/spark/pull/29527 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] huaxingao commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

huaxingao commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678893930 @srowen I will merge into 3.0? This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] huaxingao commented on pull request #29524: [SPARK-32092][ML][PySpark][3.0] Removed foldCol related code

huaxingao commented on pull request #29524: URL: https://github.com/apache/spark/pull/29524#issuecomment-678893290 Merged to 3.0. Thank you all! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] huaxingao closed pull request #29524: [SPARK-32092][ML][PySpark][3.0] Removed foldCol related code

huaxingao closed pull request #29524: URL: https://github.com/apache/spark/pull/29524 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

AmplabJenkins removed a comment on pull request #29228: URL: https://github.com/apache/spark/pull/29228#issuecomment-67647 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

AmplabJenkins commented on pull request #29509: URL: https://github.com/apache/spark/pull/29509#issuecomment-67621 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

AmplabJenkins removed a comment on pull request #29509: URL: https://github.com/apache/spark/pull/29509#issuecomment-67621 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

AmplabJenkins commented on pull request #29228: URL: https://github.com/apache/spark/pull/29228#issuecomment-67647 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

SparkQA commented on pull request #29228: URL: https://github.com/apache/spark/pull/29228#issuecomment-67372 **[Test build #127827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127827/testReport)** for PR 29228 at commit

[GitHub] [spark] SparkQA commented on pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

SparkQA commented on pull request #29509: URL: https://github.com/apache/spark/pull/29509#issuecomment-67349 **[Test build #127826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127826/testReport)** for PR 29509 at commit

[GitHub] [spark] huaxingao commented on pull request #29355: [SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function

huaxingao commented on pull request #29355: URL: https://github.com/apache/spark/pull/29355#issuecomment-678887751 Thanks a lot! @maropu This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] baohe-zhang commented on a change in pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

baohe-zhang commented on a change in pull request #29509: URL: https://github.com/apache/spark/pull/29509#discussion_r475325934 ## File path: core/src/test/scala/org/apache/spark/deploy/history/HybridStoreSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache

[GitHub] [spark] baohe-zhang commented on a change in pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

baohe-zhang commented on a change in pull request #29509: URL: https://github.com/apache/spark/pull/29509#discussion_r475325165 ## File path: core/src/test/scala/org/apache/spark/deploy/history/HybridStoreSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache

[GitHub] [spark] baohe-zhang commented on a change in pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

baohe-zhang commented on a change in pull request #29509: URL: https://github.com/apache/spark/pull/29509#discussion_r475325041 ## File path: core/src/test/scala/org/apache/spark/deploy/history/HybridStoreSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache

[GitHub] [spark] baohe-zhang commented on a change in pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

baohe-zhang commented on a change in pull request #29509: URL: https://github.com/apache/spark/pull/29509#discussion_r475325096 ## File path: core/src/test/scala/org/apache/spark/deploy/history/HybridStoreSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache

[GitHub] [spark] baohe-zhang commented on a change in pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

baohe-zhang commented on a change in pull request #29509: URL: https://github.com/apache/spark/pull/29509#discussion_r475325018 ## File path: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala ## @@ -1509,13 +1513,18 @@ class

[GitHub] [spark] huaxingao commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

huaxingao commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678885037 I think we need to put the fix in 3.0, because in the case of data is already cached, this fix makes 3.0.0 behave the same as 2.4. In 2.4 ``` cache norm in memory

[GitHub] [spark] cloud-fan commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

cloud-fan commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678884094 good catch! LGTM This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] baohe-zhang commented on a change in pull request #29509: [SPARK-31608][CORE][WEBUI][TEST] Add test suites for HybridStore and HistoryServerMemoryManager

baohe-zhang commented on a change in pull request #29509: URL: https://github.com/apache/spark/pull/29509#discussion_r475320679 ## File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ## @@ -1214,8 +1214,8 @@ private[history] class

[GitHub] [spark] LuciferYang commented on pull request #29000: [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode

LuciferYang commented on pull request #29000: URL: https://github.com/apache/spark/pull/29000#issuecomment-678872666 @Ngone51 Could you please review it again ？ This is an automated message from the Apache Git Service. To

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29452: [SPARK-32643][CORE][K8s] Consolidate state decommissioning in the TaskSchedulerImpl realm

agrawaldevesh commented on a change in pull request #29452: URL: https://github.com/apache/spark/pull/29452#discussion_r475310046 ## File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorDecommissionInfo.scala ## @@ -18,11 +18,22 @@ package

[GitHub] [spark] zhengruifeng commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

zhengruifeng commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678871882 this double caching did not exist in 2.4, and it was first introduced in 3.0.0, so I tend to put it into RC2. How doyou think about it? @huaxingao

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

AngersZh commented on a change in pull request #29526: URL: https://github.com/apache/spark/pull/29526#discussion_r475310377 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala ## @@ -176,9 +176,10 @@ object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29507: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query

AmplabJenkins removed a comment on pull request #29507: URL: https://github.com/apache/spark/pull/29507#issuecomment-678870683 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29507: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query

AmplabJenkins commented on pull request #29507: URL: https://github.com/apache/spark/pull/29507#issuecomment-678870683 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

SparkQA commented on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678870379 **[Test build #127824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127824/testReport)** for PR 29516 at commit

[GitHub] [spark] SparkQA commented on pull request #29507: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query

SparkQA commented on pull request #29507: URL: https://github.com/apache/spark/pull/29507#issuecomment-678870388 **[Test build #127825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127825/testReport)** for PR 29507 at commit

[GitHub] [spark] srowen commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

srowen commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678869607 Do we need it in 3.0? I'm not super against it but it's more of an improvement, optimization, not a bug fix

[GitHub] [spark] srowen commented on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

srowen commented on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678869497 BTW I think we may still have a real test failure here, I'm looking into it. This is an automated message from

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

AmplabJenkins removed a comment on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678869069 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

AmplabJenkins commented on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678869069 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] zhengruifeng commented on pull request #29516: [WIP][SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

zhengruifeng commented on pull request #29516: URL: https://github.com/apache/spark/pull/29516#issuecomment-678868865 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] Ngone51 commented on a change in pull request #29270: [SPARK-32466][TEST][SQL] Add PlanStabilitySuite to detect SparkPlan regression

Ngone51 commented on a change in pull request #29270: URL: https://github.com/apache/spark/pull/29270#discussion_r475307366 ## File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-modified/q10.sf100/explain.txt ## @@ -0,0 +1,286 @@ +== Physical Plan ==

[GitHub] [spark] LuciferYang commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13

LuciferYang commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-678867553 @srowen @cloud-fan @HyukjinKwon @dongjoon-hyun Thank you for your review~ This is an automated message from

[GitHub] [spark] zhengruifeng commented on pull request #29501: [SPARK-32676][3.0][ML] Fix double caching in KMeans/BiKMeans

zhengruifeng commented on pull request #29501: URL: https://github.com/apache/spark/pull/29501#issuecomment-678867157 @srowen @huaxingao Thanks for reviewing! would you mind to help backporting this to 3.0? I do not have a computer to do this right now

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29477: [SPARK-32661][K8S] Spark executors should request extra memory for off-heap allocations.

AmplabJenkins removed a comment on pull request #29477: URL: https://github.com/apache/spark/pull/29477#issuecomment-678860803 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #29477: [SPARK-32661][K8S] Spark executors should request extra memory for off-heap allocations.

SparkQA commented on pull request #29477: URL: https://github.com/apache/spark/pull/29477#issuecomment-678860789 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32446/

[GitHub] [spark] AmplabJenkins commented on pull request #29477: [SPARK-32661][K8S] Spark executors should request extra memory for off-heap allocations.

AmplabJenkins commented on pull request #29477: URL: https://github.com/apache/spark/pull/29477#issuecomment-678860798 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29477: [SPARK-32661][K8S] Spark executors should request extra memory for off-heap allocations.

AmplabJenkins removed a comment on pull request #29477: URL: https://github.com/apache/spark/pull/29477#issuecomment-678860798 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678860152 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

AmplabJenkins commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678860152 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29421: [SPARK-32388][SQL][test-hadoop2.7][test-hive1.2] TRANSFORM with schema-less mode should keep the same with hive

SparkQA commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-678859868 **[Test build #127823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127823/testReport)** for PR 29421 at commit

[GitHub] [spark] SparkQA commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

SparkQA commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678845961 **[Test build #127820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127820/testReport)** for PR 29526 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #29505: [SPARK-32648][SS] Remove unused DELETE_ACTION in FileStreamSinkLog

dongjoon-hyun commented on pull request #29505: URL: https://github.com/apache/spark/pull/29505#issuecomment-678846054 Thank you and welcome, @michal-wieleba . You are added to the Apache Spark contributor group and SPARK-32648 is assigned to you.

[GitHub] [spark] viirya commented on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

viirya commented on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678845494 Yeah, we don't run hive-1.2 test usually except we know the diff touches hive 1.2 code path. For these failed tests, they don't touch the code directly, but affect it

[GitHub] [spark] dongjoon-hyun closed pull request #29505: [SPARK-32648][SS] Remove unused DELETE_ACTION in FileStreamSinkLog

dongjoon-hyun closed pull request #29505: URL: https://github.com/apache/spark/pull/29505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] maropu edited a comment on pull request #29526: [SPARK-32352][SQL][FOLLOW-UP][test-hadoop2.7][test-hive1.2] Exclude partition columns from data columns

maropu edited a comment on pull request #29526: URL: https://github.com/apache/spark/pull/29526#issuecomment-678845205 Nice, thanks for the swift fixes, @viirya! Anyway, it seems we didn't notice this test failure for 10+ days, so we need to carefully check the branches w/hive-1.2...

  1   2   3   4   5   >