[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3.3 since we support hadoop-3.3 as default now

AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r755673891 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 +

[GitHub] [spark] srowen commented on a change in pull request #34689: [SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3.3 since we support hadoop-3.3 as default now

srowen commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r755675407 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 + hadoop-3.3

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3.3 since we support hadoop-3.3 as default now

AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r755676187 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 +

[GitHub] [spark] SparkQA removed a comment on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

SparkQA removed a comment on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977218053 **[Test build #145551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145551/testReport)** for PR 34593 at commit

[GitHub] [spark] SparkQA commented on pull request #34642: [SPARK-37369][SQL] Avoid redundant ColumnarToRow transistion on InMemoryTableScan

SparkQA commented on pull request #34642: URL: https://github.com/apache/spark/pull/34642#issuecomment-977476970 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50028/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34642: [SPARK-37369][SQL] Avoid redundant ColumnarToRow transistion on InMemoryTableScan

AmplabJenkins removed a comment on pull request #34642: URL: https://github.com/apache/spark/pull/34642#issuecomment-977494976 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50028/

[GitHub] [spark] AmplabJenkins commented on pull request #34694: [MINOR][DOCS] Fix scaladoc for skewed partitions optimization

AmplabJenkins commented on pull request #34694: URL: https://github.com/apache/spark/pull/34694#issuecomment-977496507 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

AmplabJenkins removed a comment on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977497565 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50027/

[GitHub] [spark] SparkQA commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

SparkQA commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-977522744 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50030/ -- This is an automated message from the

[GitHub] [spark] cloud-fan commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

cloud-fan commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-977525153 @zhengruifeng can you highlight the differences between your PR and this one? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] cloud-fan commented on a change in pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

cloud-fan commented on a change in pull request #34668: URL: https://github.com/apache/spark/pull/34668#discussion_r755695393 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -73,6 +78,16 @@ grammar SqlBase; return false;

[GitHub] [spark] sarutak commented on a change in pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

sarutak commented on a change in pull request #34593: URL: https://github.com/apache/spark/pull/34593#discussion_r755696900 ## File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ## @@ -2197,13 +2197,23 @@ object functions { def round(e: Column): Column

[GitHub] [spark] SparkQA removed a comment on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

SparkQA removed a comment on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977395008 **[Test build #14 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14/testReport)** for PR 34685 at commit

[GitHub] [spark] SparkQA commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

SparkQA commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977527767 **[Test build #14 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14/testReport)** for PR 34685 at commit

[GitHub] [spark] sarutak commented on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon

sarutak commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-977529479 @LuciferYang Thank you @LuciferYang . It's better to note the command lines in `How was this patch tested?` in the description. -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

SparkQA commented on pull request #34692: URL: https://github.com/apache/spark/pull/34692#issuecomment-977529863 **[Test build #145557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145557/testReport)** for PR 34692 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

SparkQA removed a comment on pull request #34692: URL: https://github.com/apache/spark/pull/34692#issuecomment-977396780 **[Test build #145557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145557/testReport)** for PR 34692 at commit

[GitHub] [spark] sarutak edited a comment on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon

sarutak edited a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-977529479 Thank you @LuciferYang . It's better to note the command lines in `How was this patch tested?` in the description. -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-977530622 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50034/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

SparkQA commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977530333 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50032/ -- This is an automated message from the Apache

[GitHub] [spark] cloud-fan commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

cloud-fan commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r755702573 ## File path: sql/core/src/test/resources/sql-tests/results/timestampNTZ/timestamp.sql.out ## @@ -373,17 +374,19 @@ struct

[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977534404 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50031/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977534422 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50031/ --

[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework

cloud-fan commented on a change in pull request #34060: URL: https://github.com/apache/spark/pull/34060#discussion_r755707733 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -3469,9 +3469,14 @@ class AstBuilder extends

[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework

cloud-fan commented on a change in pull request #34060: URL: https://github.com/apache/spark/pull/34060#discussion_r755707836 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -1090,3 +1106,12 @@ case class DropIndex(

[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977542401 **[Test build #145559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145559/testReport)** for PR 34668 at commit

[GitHub] [spark] sadikovi commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

sadikovi commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r755709143 ## File path: sql/core/src/test/resources/sql-tests/results/timestampNTZ/timestamp.sql.out ## @@ -373,17 +374,19 @@ struct

[GitHub] [spark] sadikovi commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

sadikovi commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r755709555 ## File path: sql/core/src/test/resources/sql-tests/results/timestampNTZ/timestamp.sql.out ## @@ -373,17 +374,19 @@ struct

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #34695: [WIP][SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors

AngersZh edited a comment on pull request #34695: URL: https://github.com/apache/spark/pull/34695#issuecomment-977546790 @sarutak @patil Hi, I am trying to build this pr's UI table, but I found when I write such ``` $('#executorSummaryMetricsTitle').html("Summary Metrics for "

[GitHub] [spark] AngersZhuuuu commented on pull request #34695: [WIP][SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors

AngersZh commented on pull request #34695: URL: https://github.com/apache/spark/pull/34695#issuecomment-977546790 @sarutak Hi, I am trying to build this pr's UI table, but I found when I write such ``` $('#executorSummaryMetricsTitle').html("Summary Metrics for " + "" +

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-977548067 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50034/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

SparkQA commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977548087 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50032/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile and related CI test

SparkQA commented on pull request #34679: URL: https://github.com/apache/spark/pull/34679#issuecomment-977551358 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50033/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977462251 **[Test build #145559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145559/testReport)** for PR 34668 at commit

[GitHub] [spark] SparkQA commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

SparkQA commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977463243 **[Test build #145551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145551/testReport)** for PR 34593 at commit

[GitHub] [spark] srowen commented on a change in pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile

srowen commented on a change in pull request #34679: URL: https://github.com/apache/spark/pull/34679#discussion_r755673065 ## File path: pom.xml ## @@ -3353,11 +3353,6 @@ - Review comment: No, I just mean do not remove the profile, so that

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3.3 since we support hadoop-3.3 as default now

AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r755673891 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 +

[GitHub] [spark] srowen commented on a change in pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile

srowen commented on a change in pull request #34679: URL: https://github.com/apache/spark/pull/34679#discussion_r755675181 ## File path: pom.xml ## @@ -3353,11 +3353,6 @@ - Review comment: After you remove this? Won't maven say that the profile

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile

AngersZh commented on a change in pull request #34679: URL: https://github.com/apache/spark/pull/34679#discussion_r755675902 ## File path: pom.xml ## @@ -3353,11 +3353,6 @@ - Review comment: yea, won't failed with a profile doesn't exist, So I

[GitHub] [spark] SparkQA removed a comment on pull request #34386: [WIP] - Changes to PySpark doc homepage and User Guide

SparkQA removed a comment on pull request #34386: URL: https://github.com/apache/spark/pull/34386#issuecomment-977334751 **[Test build #145553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145553/testReport)** for PR 34386 at commit

[GitHub] [spark] SparkQA commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

SparkQA commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-977474971 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50030/ -- This is an automated message from the Apache

[GitHub] [spark] sperlingxx opened a new pull request #34694: [MINOR][DOCS] Fix scaladoc for skewed partitions optimization

sperlingxx opened a new pull request #34694: URL: https://github.com/apache/spark/pull/34694 What changes were proposed in this pull request? Replaces the incorrect reference of `PartialMapperPartitionSpec` with `PartialReducerPartitionSpec` for the comments of skew partitions methods.

[GitHub] [spark] srowen commented on a change in pull request #34689: [SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3.3 since we support hadoop-3.3 as default now

srowen commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r755681520 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 + hadoop-3.3

[GitHub] [spark] zhengruifeng commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

zhengruifeng commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-977484067 @wangyum @cloud-fan @tanelk I had just updated the `RankLimitExec` in https://github.com/apache/spark/pull/34367/files to apply `Utils.takeOrdered` instead of

[GitHub] [spark] AmplabJenkins commented on pull request #34642: [SPARK-37369][SQL] Avoid redundant ColumnarToRow transistion on InMemoryTableScan

AmplabJenkins commented on pull request #34642: URL: https://github.com/apache/spark/pull/34642#issuecomment-977494976 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50028/ --

[GitHub] [spark] SparkQA commented on pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile and related CI test

SparkQA commented on pull request #34679: URL: https://github.com/apache/spark/pull/34679#issuecomment-977497183 **[Test build #145561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145561/testReport)** for PR 34679 at commit

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-977497924 **[Test build #145562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145562/testReport)** for PR 34367 at commit

[GitHub] [spark] SparkQA commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

SparkQA commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977497181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

AmplabJenkins commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977497565 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50027/ --

[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977501814 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50031/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

AmplabJenkins commented on pull request #34692: URL: https://github.com/apache/spark/pull/34692#issuecomment-977511256 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50029/ --

[GitHub] [spark] SparkQA commented on pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

SparkQA commented on pull request #34692: URL: https://github.com/apache/spark/pull/34692#issuecomment-977511105 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50029/ -- This is an automated message from the

[GitHub] [spark] sarutak commented on a change in pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

sarutak commented on a change in pull request #34593: URL: https://github.com/apache/spark/pull/34593#discussion_r755696389 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ## @@ -413,6 +413,18 @@ final class Decimal extends Ordered[Decimal]

[GitHub] [spark] sarutak commented on a change in pull request #34683: [SPARK-37283][SQL][FOLLOWUP] Avoid trying to store a table which contains timestamp_ntz types in Hive compatible format

sarutak commented on a change in pull request #34683: URL: https://github.com/apache/spark/pull/34683#discussion_r755697630 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ## @@ -1411,6 +1411,7 @@ object HiveExternalCatalog {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

AmplabJenkins removed a comment on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-977533569 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50030/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

AmplabJenkins removed a comment on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977533566 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/14/

[GitHub] [spark] AmplabJenkins commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

AmplabJenkins commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977533566 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/14/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

AmplabJenkins commented on pull request #34692: URL: https://github.com/apache/spark/pull/34692#issuecomment-977533568 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145557/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

AmplabJenkins commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-977533569 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50030/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

AmplabJenkins removed a comment on pull request #34692: URL: https://github.com/apache/spark/pull/34692#issuecomment-976440746 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile and related CI test

SparkQA commented on pull request #34679: URL: https://github.com/apache/spark/pull/34679#issuecomment-977533473 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50033/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977534422 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50031/

[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework

cloud-fan commented on a change in pull request #34060: URL: https://github.com/apache/spark/pull/34060#discussion_r755708588 ## File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala ## @@ -143,25 +143,25 @@ class

[GitHub] [spark] AngersZhuuuu opened a new pull request #34695: [WIP][SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors

AngersZh opened a new pull request #34695: URL: https://github.com/apache/spark/pull/34695 ### What changes were proposed in this pull request? This pr continue the work of https://github.com/apache/spark/pull/29247 since origin author didn't reply for a long time. Will add as

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34688: [SPARK-32079][PYTHON] Remove namedtuple hack by replacing built-in pickle to cloudpickle

HyukjinKwon commented on a change in pull request #34688: URL: https://github.com/apache/spark/pull/34688#discussion_r755712526 ## File path: python/pyspark/serializers.py ## @@ -19,7 +19,7 @@ PySpark supports custom serializers for transferring data; this can improve

[GitHub] [spark] AngersZhuuuu commented on pull request #34695: [WIP][SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors

AngersZh commented on pull request #34695: URL: https://github.com/apache/spark/pull/34695#issuecomment-977547408 Also ping @pgandhi999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] LuciferYang commented on pull request #34454: [SPARK-37013][CORE][SQL][FOLLOWUP] Use the new error framework to throw error in `FormatString`

LuciferYang commented on pull request #34454: URL: https://github.com/apache/spark/pull/34454#issuecomment-977548498 thanks all ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-977462251 **[Test build #145559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145559/testReport)** for PR 34668 at commit

[GitHub] [spark] LuciferYang edited a comment on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon

LuciferYang edited a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-976148375 @sarutak you are right, f87467b adds code corresponding to sbt. There are 3 modules with `Extended{LevelDB,RocksDB}Test` annotation: `core`,`sql/core`, `yarn`

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-976940851 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50022/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-976940853 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145549/

[GitHub] [spark] AmplabJenkins commented on pull request #34693: [SPARK-37259][SQL] Support CTE queries with MSSQL JDBC

AmplabJenkins commented on pull request #34693: URL: https://github.com/apache/spark/pull/34693#issuecomment-976848358 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145548/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-976848362 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145550/ -- This

[GitHub] [spark] ChenMichael commented on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

ChenMichael commented on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976849576 In order for this problem to manifest, we have to do join planning between the time a InMemoryRelation is converted to an rdd and the time where the job executing this rdd

[GitHub] [spark] ChenMichael commented on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

ChenMichael commented on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976858837 Oh I realized if a plan containing InMemoryRelation is converted to a RDD, but the RDD is never submitted to DAG scheduler, then the stats would also be inaccurate. I think

[GitHub] [spark] AmplabJenkins commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-976940853 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145549/ -- This

[GitHub] [spark] viirya edited a comment on pull request #34642: [SPARK-37369][SQL] Avoid redundant ColumnarToRow transistion on InMemoryTableScan

viirya edited a comment on pull request #34642: URL: https://github.com/apache/spark/pull/34642#issuecomment-977016485 > I'm trying to understand the motivation. Is it because in-memory table can output rows efficiently? Parquet scan can also output rows but we try our best to output

[GitHub] [spark] ChenMichael edited a comment on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

ChenMichael edited a comment on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976849576 In order for this problem to manifest, we have to do join planning in between the time an InMemoryRelation is converted to a RDD and the time where the job executing

[GitHub] [spark] SparkQA removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-976492795 **[Test build #145549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145549/testReport)** for PR 34668 at commit

[GitHub] [spark] sumeetgajjar commented on a change in pull request #34693: [SPARK-37259][SQL] Support CTE queries with MSSQL JDBC

sumeetgajjar commented on a change in pull request #34693: URL: https://github.com/apache/spark/pull/34693#discussion_r755398500 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala ## @@ -356,4 +357,42 @@

[GitHub] [spark] sumeetgajjar commented on pull request #34693: [SPARK-37259][SQL] Support CTE queries with MSSQL JDBC

sumeetgajjar commented on pull request #34693: URL: https://github.com/apache/spark/pull/34693#issuecomment-976972982 > This change also seem to work with MSSQL's temp table syntax: > > ``` > val withClause = "(SELECT * INTO #TempTable FROM (SELECT * FROM tbl WHERE x > 10) t)"

[GitHub] [spark] AmplabJenkins commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

AmplabJenkins commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977297115 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50023/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

AmplabJenkins removed a comment on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977297115 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50023/

[GitHub] [spark] SparkQA commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

SparkQA commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977296577 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50023/ -- This is an automated message from the

[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

zero323 commented on a change in pull request #34513: URL: https://github.com/apache/spark/pull/34513#discussion_r755596954 ## File path: python/pyspark/mllib/stat/_statistics.py ## @@ -170,10 +190,29 @@ def corr(x, y=None, method=None): if not y: return

[GitHub] [spark] HyukjinKwon commented on pull request #34655: [SPARK-37380][PYTHON] Miscellaneous Python lint infra cleanup

HyukjinKwon commented on pull request #34655: URL: https://github.com/apache/spark/pull/34655#issuecomment-977297819 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon closed pull request #34655: [SPARK-37380][PYTHON] Miscellaneous Python lint infra cleanup

HyukjinKwon closed pull request #34655: URL: https://github.com/apache/spark/pull/34655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] SparkQA commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

SparkQA commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977320493 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50024/ -- This is an automated message from the

[GitHub] [spark] wangyum commented on a change in pull request #34504: [SPARK-37226][SQL] Filter push down through window if partitionSpec isEmpty

wangyum commented on a change in pull request #34504: URL: https://github.com/apache/spark/pull/34504#discussion_r755612866 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -1548,6 +1548,31 @@ object

[GitHub] [spark] AmplabJenkins commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

AmplabJenkins commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977347904 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145552/ -- This

[GitHub] [spark] SparkQA commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

SparkQA commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-977346973 **[Test build #145552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145552/testReport)** for PR 34593 at commit

[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

dchvn commented on a change in pull request #34513: URL: https://github.com/apache/spark/pull/34513#discussion_r755622882 ## File path: python/pyspark/mllib/stat/_statistics.py ## @@ -170,10 +190,29 @@ def corr(x, y=None, method=None): if not y: return

[GitHub] [spark] SparkQA commented on pull request #34386: [WIP] - Changes to PySpark doc homepage and User Guide

SparkQA commented on pull request #34386: URL: https://github.com/apache/spark/pull/34386#issuecomment-977385098 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50025/ -- This is an automated message from the Apache

[GitHub] [spark] pan3793 commented on a change in pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

pan3793 commented on a change in pull request #34692: URL: https://github.com/apache/spark/pull/34692#discussion_r755634427 ## File path: core/src/main/scala/org/apache/spark/util/SizeEstimator.scala ## @@ -33,10 +33,9 @@ import org.apache.spark.util.collection.OpenHashSet

[GitHub] [spark] srowen commented on a change in pull request #34692: [MINOR][DOCS] Update scaladoc of KnownSizeEstimation

srowen commented on a change in pull request #34692: URL: https://github.com/apache/spark/pull/34692#discussion_r755635322 ## File path: core/src/main/scala/org/apache/spark/util/SizeEstimator.scala ## @@ -33,10 +33,9 @@ import org.apache.spark.util.collection.OpenHashSet

[GitHub] [spark] SparkQA commented on pull request #34386: [WIP] - Changes to PySpark doc homepage and User Guide

SparkQA commented on pull request #34386: URL: https://github.com/apache/spark/pull/34386#issuecomment-977392113 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50025/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

SparkQA commented on pull request #34513: URL: https://github.com/apache/spark/pull/34513#issuecomment-977393918 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50026/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

AmplabJenkins removed a comment on pull request #34513: URL: https://github.com/apache/spark/pull/34513#issuecomment-977393605 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145554/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34386: [WIP] - Changes to PySpark doc homepage and User Guide

AmplabJenkins removed a comment on pull request #34386: URL: https://github.com/apache/spark/pull/34386#issuecomment-977393604 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50025/

[GitHub] [spark] SparkQA commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

SparkQA commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-977395008 **[Test build #14 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14/testReport)** for PR 34685 at commit

<    1   2   3   4   >