[GitHub] [spark] dongjoon-hyun closed pull request #29078: [SPARK-29292][STREAMING][SQL][BUILD] Get streaming, catalyst, sql compiling for Scala 2.13

2020-07-14 Thread GitBox
dongjoon-hyun closed pull request #29078: URL: https://github.com/apache/spark/pull/29078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29024: [WIP][SPARK-32001][SQL]Create JDBC authentication provider developer API

2020-07-14 Thread GitBox
gaborgsomogyi commented on a change in pull request #29024: URL: https://github.com/apache/spark/pull/29024#discussion_r454215950 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala ## @@ -23,12 +23,15 @@ import java.util.{Lo

[GitHub] [spark] SparkQA commented on pull request #29024: [WIP][SPARK-32001][SQL]Create JDBC authentication provider developer API

2020-07-14 Thread GitBox
SparkQA commented on pull request #29024: URL: https://github.com/apache/spark/pull/29024#issuecomment-658069123 **[Test build #125820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125820/testReport)** for PR 29024 at commit [`265b26e`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29100: [MINOR][R] Match collectAsArrowToR with non-streaming collectAsArrowToPython

2020-07-14 Thread GitBox
SparkQA commented on pull request #29100: URL: https://github.com/apache/spark/pull/29100#issuecomment-658069110 **[Test build #125819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125819/testReport)** for PR 29100 at commit [`8c6be21`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29100: [MINOR][R] Match collectAsArrowToR with non-streaming collectAsArrowToPython

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29100: URL: https://github.com/apache/spark/pull/29100#issuecomment-658069657 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29024: [WIP][SPARK-32001][SQL]Create JDBC authentication provider developer API

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29024: URL: https://github.com/apache/spark/pull/29024#issuecomment-658069808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29100: [MINOR][R] Match collectAsArrowToR with non-streaming collectAsArrowToPython

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29100: URL: https://github.com/apache/spark/pull/29100#issuecomment-658069657 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29024: [WIP][SPARK-32001][SQL]Create JDBC authentication provider developer API

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29024: URL: https://github.com/apache/spark/pull/29024#issuecomment-658069808 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
HyukjinKwon commented on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658070578 Let me close this for now. This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [spark] HyukjinKwon closed pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
HyukjinKwon closed pull request #29096: URL: https://github.com/apache/spark/pull/29096 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658073592 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658073592 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
SparkQA commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658073453 **[Test build #125817 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125817/testReport)** for PR 29099 at commit [`e986c65`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658061150 **[Test build #125817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125817/testReport)** for PR 29099 at commit [`e986c65`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658073607 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125

[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #29045: [SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables

2020-07-14 Thread GitBox
SaurabhChawla100 commented on a change in pull request #29045: URL: https://github.com/apache/spark/pull/29045#discussion_r454234413 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala ## @@ -116,47 +116,53 @@ object OrcUtils ext

[GitHub] [spark] gengliangwang commented on pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-14 Thread GitBox
gengliangwang commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658093795 This is still in progress. I will add more test cases and update the PR description. This is an automated

[GitHub] [spark] gengliangwang opened a new pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-14 Thread GitBox
gengliangwang opened a new pull request #29101: URL: https://github.com/apache/spark/pull/29101 ### What changes were proposed in this pull request? In https://github.com/apache/spark/pull/28733, CNF conversion is used to push down disjunctive predicates through join and part

[GitHub] [spark] AmplabJenkins commented on pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658094178 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] wangyum commented on a change in pull request #29088: [SPARK-32289][SQL] Some characters are garbled when opening csv files with Excel

2020-07-14 Thread GitBox
wangyum commented on a change in pull request #29088: URL: https://github.com/apache/spark/pull/29088#discussion_r454249166 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CsvOutputWriter.scala ## @@ -39,6 +39,10 @@ class CsvOutputWriter(

[GitHub] [spark] SparkQA commented on pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-14 Thread GitBox
SparkQA commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658096734 **[Test build #125821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125821/testReport)** for PR 29101 at commit [`25eb140`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29101: [WIP][SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-658094178 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
SparkQA commented on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658098403 **[Test build #125812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125812/testReport)** for PR 29096 at commit [`f86a96f`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658099020 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658099025 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125

[GitHub] [spark] HyukjinKwon opened a new pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
HyukjinKwon opened a new pull request #29102: URL: https://github.com/apache/spark/pull/29102 ### What changes were proposed in this pull request? Currently the Jenkins PIP packaging test fails as below: ``` Installing dist into virtual env Processing ./python/dist/pyspar

[GitHub] [spark] SparkQA removed a comment on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658021416 **[Test build #125812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125812/testReport)** for PR 29096 at commit [`f86a96f`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
SparkQA commented on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658101371 **[Test build #125813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125813/testReport)** for PR 29096 at commit [`f86a96f`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658031266 **[Test build #125813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125813/testReport)** for PR 29096 at commit [`f86a96f`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658101861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
HyukjinKwon commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658101951 retest this please This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] HyukjinKwon commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
HyukjinKwon commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658102047 Seems like this only happens in worker 4. This is an automated message from the Apache Git Service. To respo

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658099020 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658101861 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
SparkQA commented on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658099896 **[Test build #125822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125822/testReport)** for PR 28917 at commit [`b060daa`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29096: [WIP][TESTS] Enable test-dependencies.sh and Unidoc test in Jenkins jobs

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29096: URL: https://github.com/apache/spark/pull/29096#issuecomment-658101867 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125

[GitHub] [spark] HyukjinKwon removed a comment on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
HyukjinKwon removed a comment on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658102047 Seems like this only happens in worker 4. This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658100409 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658100409 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
SparkQA commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658102990 **[Test build #125823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125823/testReport)** for PR 29102 at commit [`fc648af`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
SparkQA commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658103043 **[Test build #125824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125824/testReport)** for PR 29099 at commit [`e986c65`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658103603 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658103577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658103603 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658103577 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AngersZhuuuu commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
AngersZh commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658104535 > Can we use `Cast` to turn catalyst value to string and pass to the script? Nice advise! Updated T

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658106181 **[Test build #125827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125827/testReport)** for PR 29085 at commit [`5bfa669`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29090: [WIP][SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

2020-07-14 Thread GitBox
SparkQA commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-658106176 **[Test build #125826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125826/testReport)** for PR 29090 at commit [`2ddbe0c`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
SparkQA commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658106124 **[Test build #125825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125825/testReport)** for PR 29102 at commit [`49a827f`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658106538 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658106650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29090: [WIP][SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-658106642 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658106650 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29090: [WIP][SPARK-32293] Fix inconsistency between Spark memory configs and JVM option

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29090: URL: https://github.com/apache/spark/pull/29090#issuecomment-658106642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658106538 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658112603 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
SparkQA commented on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658112583 **[Test build #125824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125824/testReport)** for PR 29099 at commit [`e986c65`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658103043 **[Test build #125824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125824/testReport)** for PR 29099 at commit [`e986c65`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658112603 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29099: [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29099: URL: https://github.com/apache/spark/pull/29099#issuecomment-658112613 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125

[GitHub] [spark] Fokko commented on pull request #28957: [SPARK-32138] Drop Python 2.7, 3.4 and 3.5

2020-07-14 Thread GitBox
Fokko commented on pull request #28957: URL: https://github.com/apache/spark/pull/28957#issuecomment-658115224 Cool stuff, thanks for the work @HyukjinKwon This is an automated message from the Apache Git Service. To res

[GitHub] [spark] ScrapCodes commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
ScrapCodes commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658115916 Hi Folks @dongjoon-hyun and @holdenk, do we plan to include this patch in 2.4.7 release? This is an

[GitHub] [spark] stijndehaes commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
stijndehaes commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658116799 @ScrapCodes the code in the 2.4.x is significantly different from the code here. But we can reuse the same idea as here. I guess it has to be a new PR -

[GitHub] [spark] ScrapCodes commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
ScrapCodes commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658117121 BTW, when do we receive a version changed from K8s? This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-656431096 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8

[GitHub] [spark] attilapiros commented on a change in pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
attilapiros commented on a change in pull request #29102: URL: https://github.com/apache/spark/pull/29102#discussion_r454277192 ## File path: dev/run-pip-tests ## @@ -96,6 +96,8 @@ for python in "${PYTHON_EXECS[@]}"; do cd "$FWDIR"/python # Delete the egg info file if

[GitHub] [spark] ScrapCodes commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
ScrapCodes commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658117892 > @ScrapCodes the code in the 2.4.x is significantly different from the code here. But we can reuse the same idea as here. I guess it has to be a new PR Currently the J

[GitHub] [spark] SparkQA commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
SparkQA commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658117883 **[Test build #125828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125828/testReport)** for PR 28423 at commit [`51dc743`](https://github.com

[GitHub] [spark] stijndehaes commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
stijndehaes commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658121676 > BTW, when do we receive a version changed from K8s? It happens when etcd compaction kicks in for example. On aws EKS I never saw this happening on EKS 1.14, but it h

[GitHub] [spark] SparkQA commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
SparkQA commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658122989 **[Test build #125828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125828/testReport)** for PR 28423 at commit [`51dc743`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658123134 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658123134 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658117883 **[Test build #125828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125828/testReport)** for PR 28423 at commit [`51dc743`](https://gi

[GitHub] [spark] SparkQA commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
SparkQA commented on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658128852 **[Test build #125814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125814/testReport)** for PR 28917 at commit [`ec0d8d0`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658031305 **[Test build #125814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125814/testReport)** for PR 28917 at commit [`ec0d8d0`](https://gi

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
HyukjinKwon commented on a change in pull request #29102: URL: https://github.com/apache/spark/pull/29102#discussion_r454291352 ## File path: dev/run-pip-tests ## @@ -96,6 +96,8 @@ for python in "${PYTHON_EXECS[@]}"; do cd "$FWDIR"/python # Delete the egg info file if

[GitHub] [spark] mayurdb commented on pull request #29097: [SPARK-32299] [SQL] Decide SMJ Join Orientation adaptively

2020-07-14 Thread GitBox
mayurdb commented on pull request #29097: URL: https://github.com/apache/spark/pull/29097#issuecomment-658130172 cc @maryannxue @cloud-fan This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658130153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658130102 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658130102 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-658130153 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] alfozan edited a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
alfozan edited a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658131729 > @alfozan Hi, alfozan, I know that in facebook using script transform a lot, in your case, do you will use script transform with serde? @AngersZh Yes, we imple

[GitHub] [spark] alfozan commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
alfozan commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658131729 > @alfozan Hi, alfozan, I know that in facebook using script transform a lot, in your case, do you will use script transform with serde? Yes, we implemented two native Ser

[GitHub] [spark] SparkQA commented on pull request #29102: [SPARK-32303][PYTHON][BUILD] Remove leftover from editable mode installation in PIP test

2020-07-14 Thread GitBox
SparkQA commented on pull request #29102: URL: https://github.com/apache/spark/pull/29102#issuecomment-658132447 **[Test build #125829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125829/testReport)** for PR 29102 at commit [`5879260`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658134139 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-658134139 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] alfozan commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
alfozan commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r454298255 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkScriptTransformationExec.scala ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apa

[GitHub] [spark] alfozan commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
alfozan commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r454298255 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkScriptTransformationExec.scala ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apa

[GitHub] [spark] cloud-fan commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
cloud-fan commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r454298739 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,200 @@ +/* + * Licensed to

[GitHub] [spark] cloud-fan commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
cloud-fan commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r454299734 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,200 @@ +/* + * Licensed to

[GitHub] [spark] cloud-fan commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-14 Thread GitBox
cloud-fan commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-658136574 What's the behavior of hive if the script transformation doesn't specify a serde? Does Hive pick a default serde, or it well defines the behavior of non-serde? -

[GitHub] [spark] SparkQA commented on pull request #29098: [SPARK-32300][PYTHON][2.4] toPandas should work from a Spark DataFrame with no partitions

2020-07-14 Thread GitBox
SparkQA commented on pull request #29098: URL: https://github.com/apache/spark/pull/29098#issuecomment-658139766 **[Test build #125815 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125815/testReport)** for PR 29098 at commit [`c3a7f7e`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29098: [SPARK-32300][PYTHON][2.4] toPandas should work from a Spark DataFrame with no partitions

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #29098: URL: https://github.com/apache/spark/pull/29098#issuecomment-658053443 **[Test build #125815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125815/testReport)** for PR 29098 at commit [`c3a7f7e`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29098: [SPARK-32300][PYTHON][2.4] toPandas should work from a Spark DataFrame with no partitions

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29098: URL: https://github.com/apache/spark/pull/29098#issuecomment-658140318 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29098: [SPARK-32300][PYTHON][2.4] toPandas should work from a Spark DataFrame with no partitions

2020-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #29098: URL: https://github.com/apache/spark/pull/29098#issuecomment-658140318 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on a change in pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command

2020-07-14 Thread GitBox
cloud-fan commented on a change in pull request #28840: URL: https://github.com/apache/spark/pull/28840#discussion_r454307435 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ## @@ -236,6 +236,45 @@ case class ShowFunctionsCommand(

[GitHub] [spark] LuciferYang opened a new pull request #29103: [SPARK-32305][BUILD]Automatically cleanup meta data generated by test cases not in target dir

2020-07-14 Thread GitBox
LuciferYang opened a new pull request #29103: URL: https://github.com/apache/spark/pull/29103 ### What changes were proposed in this pull request? Add additional configuration to `maven-clean-plugin` to ensure cleanup `metastore_db` and `spark-warehouse` directory when execute `mvn c

[GitHub] [spark] SparkQA commented on pull request #29024: [WIP][SPARK-32001][SQL]Create JDBC authentication provider developer API

2020-07-14 Thread GitBox
SparkQA commented on pull request #29024: URL: https://github.com/apache/spark/pull/29024#issuecomment-658146504 **[Test build #125820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125820/testReport)** for PR 29024 at commit [`265b26e`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29024: [WIP][SPARK-32001][SQL]Create JDBC authentication provider developer API

2020-07-14 Thread GitBox
SparkQA removed a comment on pull request #29024: URL: https://github.com/apache/spark/pull/29024#issuecomment-658069123 **[Test build #125820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125820/testReport)** for PR 29024 at commit [`265b26e`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29103: [SPARK-32305][BUILD]Automatically cleanup meta data generated by test cases not in target dir

2020-07-14 Thread GitBox
AmplabJenkins commented on pull request #29103: URL: https://github.com/apache/spark/pull/29103#issuecomment-658146644 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

<    1   2   3   4   5   6   7   8   9   10   >