[GitHub] [spark] BryanCutler commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
BryanCutler commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704712489 cc @HyukjinKwon @dongjoon-hyun , I tested this locally with Python 2 This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
dongjoon-hyun commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704712390 Thank you, @BryanCutler ! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704720549 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34097/

[GitHub] [spark] tanelk commented on a change in pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
tanelk commented on a change in pull request #29092: URL: https://github.com/apache/spark/pull/29092#discussion_r500786803 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -1854,3 +1855,37 @@ object OptimizeLimitZero

[GitHub] [spark] SparkQA commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
SparkQA commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704757171 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34099/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704749045 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
HyukjinKwon commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704756667 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] SparkQA removed a comment on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704627610 **[Test build #129481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129481/testReport)** for PR 29953 at commit

[GitHub] [spark] cloud-fan commented on pull request #29132: [SPARK-32331][SQL] Keep advanced statistics when pruning partitions

2020-10-07 Thread GitBox
cloud-fan commented on pull request #29132: URL: https://github.com/apache/spark/pull/29132#issuecomment-704727451 @maryannxue This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] AmplabJenkins commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704727645 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704727645 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
SparkQA commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704732330 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34098/

[GitHub] [spark] cloud-fan commented on a change in pull request #29074: [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection

2020-10-07 Thread GitBox
cloud-fan commented on a change in pull request #29074: URL: https://github.com/apache/spark/pull/29074#discussion_r500773402 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala ## @@ -0,0 +1,134 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA removed a comment on pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29092: URL: https://github.com/apache/spark/pull/29092#issuecomment-704655263 **[Test build #129487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129487/testReport)** for PR 29092 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704740237 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29092: URL: https://github.com/apache/spark/pull/29092#issuecomment-704740807 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704704721 **[Test build #129491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129491/testReport)** for PR 29961 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29092: URL: https://github.com/apache/spark/pull/29092#issuecomment-704740797 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704740176 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
SparkQA commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704740867 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34098/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704740267 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704696951 **[Test build #129490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129490/testReport)** for PR 29913 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704635457 **[Test build #129482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129482/testReport)** for PR 29843 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704740335 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704740179 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704733871 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704686282 **[Test build #129489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129489/testReport)** for PR 29953 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29092: URL: https://github.com/apache/spark/pull/29092#issuecomment-704740797 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704740276 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704740249 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704740888 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704740888 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] cloud-fan commented on a change in pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
cloud-fan commented on a change in pull request #29913: URL: https://github.com/apache/spark/pull/29913#discussion_r500794424 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala ## @@ -587,6 +589,20 @@ object

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
SparkQA commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704759188 **[Test build #129495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129495/testReport)** for PR 29843 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
dongjoon-hyun commented on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704717597 Merged to branch-3.0. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704717193 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun closed pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
dongjoon-hyun closed pull request #29953: URL: https://github.com/apache/spark/pull/29953 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704740896 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] tanelk commented on a change in pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
tanelk commented on a change in pull request #29092: URL: https://github.com/apache/spark/pull/29092#discussion_r500784343 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -1854,3 +1855,37 @@ object OptimizeLimitZero

[GitHub] [spark] SparkQA commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704751857 **[Test build #129494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129494/testReport)** for PR 29961 at commit

[GitHub] [spark] cloud-fan commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

2020-10-07 Thread GitBox
cloud-fan commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-704751988 > I think we just need a new configuration that works like SparkContext.addJar/addFile, using `comma separator and resolving URIs with the HDFS's default scheme consistently

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
SparkQA commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704780306 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34101/

[GitHub] [spark] SparkQA removed a comment on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704713094 **[Test build #129492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129492/testReport)** for PR 29962 at commit

[GitHub] [spark] cloud-fan closed pull request #29880: [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier

2020-10-07 Thread GitBox
cloud-fan closed pull request #29880: URL: https://github.com/apache/spark/pull/29880 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #29880: [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier

2020-10-07 Thread GitBox
cloud-fan commented on pull request #29880: URL: https://github.com/apache/spark/pull/29880#issuecomment-704726274 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
SparkQA commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704727043 **[Test build #129492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129492/testReport)** for PR 29962 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #29132: [SPARK-32331][SQL] Keep advanced statistics when pruning partitions

2020-10-07 Thread GitBox
HeartSaVioR commented on pull request #29132: URL: https://github.com/apache/spark/pull/29132#issuecomment-704726524 cc. @cloud-fan @wzhfy @gengliangwang This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] itholic opened a new pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOWUP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
itholic opened a new pull request #29963: URL: https://github.com/apache/spark/pull/29963 ### What changes were proposed in this pull request? This PR is a follow up of #29781 to fix broken link and typo.

[GitHub] [spark] dbtsai commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
dbtsai commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704753694 Test this again This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] SparkQA commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
SparkQA commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704713094 **[Test build #129492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129492/testReport)** for PR 29962 at commit

[GitHub] [spark] SparkQA commented on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
SparkQA commented on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704713453 **[Test build #129481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129481/testReport)** for PR 29953 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29074: [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29074: URL: https://github.com/apache/spark/pull/29074#issuecomment-704721295 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29074: [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29074: URL: https://github.com/apache/spark/pull/29074#issuecomment-704721295 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704733868 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704733868 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704740176 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
SparkQA commented on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704739982 **[Test build #129490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129490/testReport)** for PR 29913 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704740237 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704740326 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
SparkQA commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704739973 **[Test build #129482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129482/testReport)** for PR 29843 at commit

[GitHub] [spark] SparkQA commented on pull request #29092: [SPARK-32295][SQL] Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

2020-10-07 Thread GitBox
SparkQA commented on pull request #29092: URL: https://github.com/apache/spark/pull/29092#issuecomment-704739977 **[Test build #129487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129487/testReport)** for PR 29092 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704740267 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704739980 **[Test build #129491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129491/testReport)** for PR 29961 at commit

[GitHub] [spark] SparkQA commented on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
SparkQA commented on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704739976 **[Test build #129489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129489/testReport)** for PR 29953 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-07 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r500792651 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala ## @@ -136,8 +136,15 @@ trait WindowExecBase extends

[GitHub] [spark] HyukjinKwon commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
HyukjinKwon commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704755707 Merged to master. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HyukjinKwon closed pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
HyukjinKwon closed pull request #29963: URL: https://github.com/apache/spark/pull/29963 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
SparkQA commented on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704762769 **[Test build #129496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129496/testReport)** for PR 29913 at commit

[GitHub] [spark] SparkQA commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704774640 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34100/

[GitHub] [spark] peter-toth commented on pull request #29132: [SPARK-32331][SQL] Keep advanced statistics when pruning partitions

2020-10-07 Thread GitBox
peter-toth commented on pull request #29132: URL: https://github.com/apache/spark/pull/29132#issuecomment-704774309 Thanks @HeartSaVioR. I've updated the description. This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704714517 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704714517 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29797: [SPARK-32932][SQL] Do not use local shuffle reader at final stage on DataWritingCommand

2020-10-07 Thread GitBox
cloud-fan commented on a change in pull request #29797: URL: https://github.com/apache/spark/pull/29797#discussion_r500796701 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala ## @@ -102,6 +103,14 @@ case class

[GitHub] [spark] SparkQA commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
SparkQA commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704769759 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34099/

[GitHub] [spark] HyukjinKwon commented on pull request #29962: [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures

2020-10-07 Thread GitBox
HyukjinKwon commented on pull request #29962: URL: https://github.com/apache/spark/pull/29962#issuecomment-704770249 Merged to branch-3.0. Thanks @BryanCutler and @dongjoon-hyun. This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704769780 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704769780 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-07 Thread GitBox
SparkQA commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-704777424 **[Test build #129497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129497/testReport)** for PR 29843 at commit

[GitHub] [spark] SparkQA commented on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
SparkQA commented on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704717164 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34096/

[GitHub] [spark] AmplabJenkins commented on pull request #29913: [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29913: URL: https://github.com/apache/spark/pull/29913#issuecomment-704717193 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29074: [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection

2020-10-07 Thread GitBox
cloud-fan commented on a change in pull request #29074: URL: https://github.com/apache/spark/pull/29074#discussion_r500772981 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala ## @@ -176,19 +181,51 @@ case class

[GitHub] [spark] maropu commented on pull request #29953: [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

2020-10-07 Thread GitBox
maropu commented on pull request #29953: URL: https://github.com/apache/spark/pull/29953#issuecomment-704731386 Thanks, all! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29074: [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29074: URL: https://github.com/apache/spark/pull/29074#issuecomment-704643644 **[Test build #129483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129483/testReport)** for PR 29074 at commit

[GitHub] [spark] SparkQA commented on pull request #29074: [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection

2020-10-07 Thread GitBox
SparkQA commented on pull request #29074: URL: https://github.com/apache/spark/pull/29074#issuecomment-704720405 **[Test build #129483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129483/testReport)** for PR 29074 at commit

[GitHub] [spark] SparkQA commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
SparkQA commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704738140 **[Test build #129493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129493/testReport)** for PR 29963 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
SparkQA removed a comment on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704738140 **[Test build #129493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129493/testReport)** for PR 29963 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704749045 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
SparkQA commented on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704748700 **[Test build #129493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129493/testReport)** for PR 29963 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #29132: [SPARK-32331][SQL] Keep advanced statistics when pruning partitions

2020-10-07 Thread GitBox
HeartSaVioR commented on pull request #29132: URL: https://github.com/apache/spark/pull/29132#issuecomment-704728664 Just a general comment (as I'm not expert of this area and feeling unqualified): it'd be nice if we have the information on improvement

[GitHub] [spark] SparkQA commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704732996 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34097/

[GitHub] [spark] AmplabJenkins commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
AmplabJenkins commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704733015 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704733015 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HeartSaVioR commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
HeartSaVioR commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704750144 retest this, please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-07 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r500793413 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala ## @@ -151,10 +169,69 @@ final class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29963: [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs

2020-10-07 Thread GitBox
AmplabJenkins removed a comment on pull request #29963: URL: https://github.com/apache/spark/pull/29963#issuecomment-704769785 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #29961: [SPARK-33082][SQL] Remove hive-1.2 workaround code

2020-10-07 Thread GitBox
SparkQA commented on pull request #29961: URL: https://github.com/apache/spark/pull/29961#issuecomment-704786179 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34100/

[GitHub] [spark] gemelen commented on pull request #29286: [SPARK-21708][BUILD] Migrate build to sbt 1.x

2020-10-07 Thread GitBox
gemelen commented on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-704788735 Let's review this thing :) This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HyukjinKwon commented on pull request #29964: [SPARK-32047][SQL]Add JDBC connection provider disable possibility

2020-10-07 Thread GitBox
HyukjinKwon commented on pull request #29964: URL: https://github.com/apache/spark/pull/29964#issuecomment-704833552 BTW, quick question: do you plan to write a doc for JDBC security stuff? It should be good to have it.

[GitHub] [spark] stijndehaes commented on a change in pull request #29934: [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template

2020-10-07 Thread GitBox
stijndehaes commented on a change in pull request #29934: URL: https://github.com/apache/spark/pull/29934#discussion_r500921364 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/PodTemplateConfigMapStep.scala ## @@ -31,14

[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

2020-10-07 Thread GitBox
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-704866420 **[Test build #129504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129504/testReport)** for PR 29881 at commit

  1   2   3   4   5   6   7   8   >