[GitHub] [spark] AmplabJenkins removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950554993 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950554990 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144571/

[GitHub] [spark] AmplabJenkins commented on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950554991 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144569/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950554994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950554991 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144569/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950554992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49046/

[GitHub] [spark] AmplabJenkins commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950554990 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144571/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950554992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49046/ --

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950554323 Yes, I have added this step to the PR description -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] LuciferYang commented on a change in pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
LuciferYang commented on a change in pull request #34368: URL: https://github.com/apache/spark/pull/34368#discussion_r735276108 ## File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ## @@ -394,8 +395,17 @@ private[spark] object ClosureCleaner extends

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950553649 No problem. Thank you for your swift answer. Could you revise the PR description according to the procedure? -- This is an automated message from the Apache Git

[GitHub] [spark] SparkQA removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
SparkQA removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950482146 **[Test build #144571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144571/testReport)** for PR 34372 at commit

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 @dongjoon-hyun sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description I used zulu17:

[GitHub] [spark] SparkQA removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950490362 **[Test build #144574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144574/testReport)** for PR 34368 at commit

[GitHub] [spark] HyukjinKwon edited a comment on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

2021-10-24 Thread GitBox
HyukjinKwon edited a comment on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-950552721 @tgravescs and @revans2 FYI. I am thinking about introducing an API like `DataFrame.mapInArrow` like

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950553295 **[Test build #144571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144571/testReport)** for PR 34372 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-950553265 The main reason of this API are: - pandas instances don't support nested instances, and it disallows vectorized operations on that (PySpark) - Easier Arrow integration

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 @dongjoon-hyun sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description I used zulu17

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 @dongjoon-hyun sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description I test with zulu17

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950553099 **[Test build #144574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144574/testReport)** for PR 34368 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950482167 **[Test build #144572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144572/testReport)** for PR 34368 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
SparkQA removed a comment on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950438237 **[Test build #144569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144569/testReport)** for PR 34356 at commit

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description -- This is an automated

[GitHub] [spark] HyukjinKwon commented on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-950552721 @tgravescs and @revans2 FYI. I am thinking about introducing an API like `DataFrame.mapInArrow` like

[GitHub] [spark] SparkQA commented on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
SparkQA commented on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950552263 **[Test build #144569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144569/testReport)** for PR 34356 at commit

[GitHub] [spark] cloud-fan commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolution rules into one

2021-10-24 Thread GitBox
cloud-fan commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-950552208 also cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] cloud-fan commented on a change in pull request #34358: [SPARK-37087][SQL] Merge three relation resolution rules into one

2021-10-24 Thread GitBox
cloud-fan commented on a change in pull request #34358: URL: https://github.com/apache/spark/pull/34358#discussion_r735274431 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1184,24 +1039,22 @@ class Analyzer(override val

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950551898 **[Test build #144572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144572/testReport)** for PR 34368 at commit

[GitHub] [spark] SparkQA commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
SparkQA commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950550704 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49048/ -- This is an automated message from the Apache

[GitHub] [spark] PengleiShi commented on a change in pull request #33914: [SPARK-32268][SQL] Dynamic bloom filter join pruning

2021-10-24 Thread GitBox
PengleiShi commented on a change in pull request #33914: URL: https://github.com/apache/spark/pull/33914#discussion_r733515433 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/DynamicBloomFilterPruning.scala ## @@ -0,0 +1,191 @@ +/* + *

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950549267 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49047/ -- This is an automated message from the Apache

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950548048 Does this code work for all JVMs, @LuciferYang ? When I tried this on Java 17, it seems to fail. ``` $ java -version openjdk version "17" 2021-09-14 LTS

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950547919 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49046/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
dongjoon-hyun commented on a change in pull request #34368: URL: https://github.com/apache/spark/pull/34368#discussion_r735268807 ## File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ## @@ -394,8 +395,17 @@ private[spark] object ClosureCleaner extends

[GitHub] [spark] tanelk commented on a change in pull request #33404: [SPARK-36194][SQL] Remove the aggregation from left semi/anti join if the same aggregation has already been done on left side

2021-10-24 Thread GitBox
tanelk commented on a change in pull request #33404: URL: https://github.com/apache/spark/pull/33404#discussion_r735264211 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RemoveRedundantAggregates.scala ## @@ -47,6 +47,10 @@ object

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950530139 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49045/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-939396991 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] SparkQA commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
SparkQA commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950529556 **[Test build #144577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144577/testReport)** for PR 34234 at commit

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950529376 **[Test build #144576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144576/testReport)** for PR 34367 at commit

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950528353 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49046/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950527920 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49043/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950527923 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49044/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950527922 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49042/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950527921 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144575/

[GitHub] [spark] AmplabJenkins commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950527922 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49042/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950527921 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144575/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950527923 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49044/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950527920 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49043/ --

[GitHub] [spark] zhengruifeng commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
zhengruifeng commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950527904 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA removed a comment on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
SparkQA removed a comment on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950511723 **[Test build #144575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144575/testReport)** for PR 34375 at commit

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950525915 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49042/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950524514 **[Test build #144575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144575/testReport)** for PR 34375 at commit

[GitHub] [spark] HyukjinKwon closed pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
HyukjinKwon closed pull request #34372: URL: https://github.com/apache/spark/pull/34372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950523923 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950523509 I cherry-picked https://github.com/apache/spark/commit/76a317aaafbf52e8e1d5687e9d72e3a22b0a322e back to the master branch. -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
dongjoon-hyun commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522783 No problem, @HyukjinKwon ~ We are okay because this is not released and we can fix it. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dongjoon-hyun commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
dongjoon-hyun commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522434 Ya, this commit technically reverted `[SPARK-37084][SQL] Set spark.sql.files.openCostInBytes to bytesConf` -- This is an automated message from the Apache Git Service.

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522356 sorry it was my bad. let me correct this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] HyukjinKwon edited a comment on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
HyukjinKwon edited a comment on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522016 This is a bit odds because the PR shows the diff only the one in description https://github.com/apache/spark/pull/34371/files, and I haven't had any problem in such

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522016 This is a bit odds because the PR shows the diff only the one in description https://github.com/apache/spark/pull/34371/files -- This is an automated message from the

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950521745 oops -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
dongjoon-hyun edited a comment on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950521492 @HyukjinKwon , the final commit looks still wrong to me. - https://github.com/apache/spark/commit/360897153755e76608a7c067d01f635fca2a5da8 Could you check

[GitHub] [spark] dongjoon-hyun commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

2021-10-24 Thread GitBox
dongjoon-hyun commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950521492 @HyukjinKwon , the final commit looks still wrong to me. - https://github.com/apache/spark/commit/360897153755e76608a7c067d01f635fca2a5da8 Could you check the

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950518495 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49043/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950517863 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49044/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950515608 Thank you for updating, @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] dchvn commented on pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

2021-10-24 Thread GitBox
dchvn commented on pull request #34235: URL: https://github.com/apache/spark/pull/34235#issuecomment-950515078 Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] mridulm commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
mridulm commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950514370 Ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] mridulm commented on a change in pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
mridulm commented on a change in pull request #34234: URL: https://github.com/apache/spark/pull/34234#discussion_r735245433 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -255,9 +255,37 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950512011 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144570/

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950512011 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144570/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950443311 **[Test build #144570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144570/testReport)** for PR 34367 at commit

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950511723 **[Test build #144575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144575/testReport)** for PR 34375 at commit

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950511716 **[Test build #144570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144570/testReport)** for PR 34367 at commit

[GitHub] [spark] dchvn opened a new pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

2021-10-24 Thread GitBox
dchvn opened a new pull request #34375: URL: https://github.com/apache/spark/pull/34375 ### What changes were proposed in this pull request? Inline type hints for python/pyspark/status.py ### Why are the changes needed? We can take advantage of static type checking

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950501750 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49042/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950500260 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49044/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950499077 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49043/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon closed pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

2021-10-24 Thread GitBox
HyukjinKwon closed pull request #34235: URL: https://github.com/apache/spark/pull/34235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

2021-10-24 Thread GitBox
HyukjinKwon commented on pull request #34235: URL: https://github.com/apache/spark/pull/34235#issuecomment-950496167 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

2021-10-24 Thread GitBox
dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735235386 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1008,6 +1008,16 @@ object SQLConf {

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

2021-10-24 Thread GitBox
dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735235184 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1010,8 +1010,8 @@ object SQLConf { val

[GitHub] [spark] dchvn commented on pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

2021-10-24 Thread GitBox
dchvn commented on pull request #34235: URL: https://github.com/apache/spark/pull/34235#issuecomment-950494670 Ping @ueshin @itholic @xinrong-databricks , Could you take a look? Many thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

2021-10-24 Thread GitBox
dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735234256 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala ## @@ -577,6 +579,61 @@ class

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950490362 **[Test build #144574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144574/testReport)** for PR 34368 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

2021-10-24 Thread GitBox
dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735231356 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala ## @@ -31,16 +31,18 @@ import

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

2021-10-24 Thread GitBox
dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735231356 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala ## @@ -31,16 +31,18 @@ import

[GitHub] [spark] weixiuli commented on pull request #34302: [SPARK-37028][UI] Add a 'kill' executor link in the Web UI.

2021-10-24 Thread GitBox
weixiuli commented on pull request #34302: URL: https://github.com/apache/spark/pull/34302#issuecomment-950486619 ping @Ngone51 Kindly review. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950141240 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

2021-10-24 Thread GitBox
SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950482167 **[Test build #144572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144572/testReport)** for PR 34368 at commit

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950482209 **[Test build #144573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144573/testReport)** for PR 34367 at commit

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

2021-10-24 Thread GitBox
SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950482146 **[Test build #144571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144571/testReport)** for PR 34372 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950481577 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49040/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950481576 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49041/

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950481576 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49041/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
AmplabJenkins commented on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950481577 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49040/ --

[GitHub] [spark] wankunde commented on a change in pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
wankunde commented on a change in pull request #34234: URL: https://github.com/apache/spark/pull/34234#discussion_r735226478 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -255,9 +255,24 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] [spark] wankunde commented on a change in pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

2021-10-24 Thread GitBox
wankunde commented on a change in pull request #34234: URL: https://github.com/apache/spark/pull/34234#discussion_r735225991 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -255,9 +255,24 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-10-24 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950476042 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49041/ -- This is an automated message from the

[GitHub] [spark] yoda-mon commented on a change in pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

2021-10-24 Thread GitBox
yoda-mon commented on a change in pull request #34356: URL: https://github.com/apache/spark/pull/34356#discussion_r735220881 ## File path: python/pyspark/sql/functions.py ## @@ -2131,6 +2131,24 @@ def weekofyear(col: "ColumnOrName") -> Column: return

  1   2   3   >