[GitHub] [spark] AmplabJenkins removed a comment on pull request #34713: [SPARK-37464][SQL] SCHEMA and DATABASE should simply be aliases of NAMESPACE

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34713: URL: https://github.com/apache/spark/pull/34713#issuecomment-979363057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50104/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979363056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
SparkQA commented on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979365714 **[Test build #145635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145635/testReport)** for PR 34712 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979365033 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145634/

[GitHub] [spark] sunchao commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader

2021-11-25 Thread GitBox
sunchao commented on pull request #34659: URL: https://github.com/apache/spark/pull/34659#issuecomment-979373931 Thanks @sadikovi , really appreciate your feedback! will address the comments soon! and then you can re-visit the PR. -- This is an automated message from the Apache Git

[GitHub] [spark] imback82 commented on pull request #34708: [SPARK-37311][SQL] Migrate ALTER NAMESPACE ... SET LOCATION to use V2 command by default

2021-11-25 Thread GitBox
imback82 commented on pull request #34708: URL: https://github.com/apache/spark/pull/34708#issuecomment-979373704 Thanks @cloud-fan for your help throughout the year! Happy Thanksgiving! -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
SparkQA commented on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979321199 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50105/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34713: [SPARK-37464][SQL] SCHEMA and DATABASE should simply be aliases of NAMESPACE

2021-11-25 Thread GitBox
SparkQA commented on pull request #34713: URL: https://github.com/apache/spark/pull/34713#issuecomment-979321481 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50104/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-979326802 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50103/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979326801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50102/ --

[GitHub] [spark] SparkQA commented on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
SparkQA commented on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979326234 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50106/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-979328359 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145629/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-979328359 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145629/

[GitHub] [spark] cloud-fan commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

2021-11-25 Thread GitBox
cloud-fan commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-979352361 @sathiyapk you are right, but I can also argue that the `round` function with a rounding mode can not do what I want by using the `floor`/`ceil` function with a scale

[GitHub] [spark] SparkQA commented on pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp

2021-11-25 Thread GitBox
SparkQA commented on pull request #34712: URL: https://github.com/apache/spark/pull/34712#issuecomment-979354195 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50106/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979524304 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145638/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979524304 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145638/

[GitHub] [spark] SparkQA commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
SparkQA commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979524887 **[Test build #145639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145639/testReport)** for PR 34705 at commit

[GitHub] [spark] github-actions[bot] closed pull request #32468: [SPARK-35335][SQL] Coalesce shuffle partition as much as possible for REPARTITION_BY_NONE

2021-11-25 Thread GitBox
github-actions[bot] closed pull request #32468: URL: https://github.com/apache/spark/pull/32468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] github-actions[bot] closed pull request #32475: [SPARK-34775][SQL] Push down limit through window when partitionSpec is not empty

2021-11-25 Thread GitBox
github-actions[bot] closed pull request #32475: URL: https://github.com/apache/spark/pull/32475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] github-actions[bot] commented on pull request #33323: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads

2021-11-25 Thread GitBox
github-actions[bot] commented on pull request #33323: URL: https://github.com/apache/spark/pull/33323#issuecomment-979526233 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] github-actions[bot] closed pull request #33725: [SPARK-36494] Add param bucketSpec when create LogicalRelation for the hive table in HiveMetastoreCatalog.covertToLogicalRelation

2021-11-25 Thread GitBox
github-actions[bot] closed pull request #33725: URL: https://github.com/apache/spark/pull/33725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757166139 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757166689 ## File path: dev/test-dependencies.sh ## @@ -35,7 +35,7 @@ HADOOP_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive \

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757166337 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 + hadoop-3

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757166768 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757166337 ## File path: hadoop-cloud/pom.xml ## @@ -201,7 +201,7 @@ enables store-specific committers. --> - hadoop-3.2 + hadoop-3

[GitHub] [spark] HyukjinKwon commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979534061 Guys, im gonna revert this - this fix is incomplete, and we should keep the name same as `hadoop3` instead of `hadoop3.3`. -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979544279 cc @dongjoon-hyun too FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] SparkQA commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
SparkQA commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979544809 **[Test build #145639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145639/testReport)** for PR 34705 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
SparkQA removed a comment on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979524887 **[Test build #145639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145639/testReport)** for PR 34705 at commit

[GitHub] [spark] SparkQA commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
SparkQA commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979558138 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50110/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
SparkQA commented on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979574632 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50109/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
dongjoon-hyun commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979578221 Thank you for pinging me, @HyukjinKwon . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins commented on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979578846 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50109/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979578847 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145639/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979578846 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50109/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979578847 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145639/

[GitHub] [spark] dongjoon-hyun commented on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon

2021-11-25 Thread GitBox
dongjoon-hyun commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-979580809 Sorry, but I'm -1 for this auto-disablement because this hides all the issues from the most of the developers and doesn't help the developers fix the root causes. That's

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-978896363 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145615/

[GitHub] [spark] sadikovi commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
sadikovi commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979584489 Tests failed due to ``` Error: Could not find hadoop3.2 in the list. Valid options are dict_keys(['hadoop2.7', 'hadoop3.3']) Error: Process completed with exit code

[GitHub] [spark] HyukjinKwon commented on pull request #34710: [SPARK-37461][YARN] YARN-CLIENT mode client.appId is always null

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34710: URL: https://github.com/apache/spark/pull/34710#issuecomment-979586828 @AngersZh can you fill "How was this patch tested?"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] SparkQA commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
SparkQA commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979587426 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50110/ -- This is an automated message from the

[GitHub] [spark] HyukjinKwon commented on pull request #34677: [SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34677: URL: https://github.com/apache/spark/pull/34677#issuecomment-979587707 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon closed pull request #34677: [SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark

2021-11-25 Thread GitBox
HyukjinKwon closed pull request #34677: URL: https://github.com/apache/spark/pull/34677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] srowen commented on a change in pull request #34710: [SPARK-37461][YARN] YARN-CLIENT mode client.appId is always null

2021-11-25 Thread GitBox
srowen commented on a change in pull request #34710: URL: https://github.com/apache/spark/pull/34710#discussion_r757183265 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -181,7 +180,7 @@ private[spark] class Client(

[GitHub] [spark] HyukjinKwon commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979592260 Thanks, @srowen. Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon closed pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
HyukjinKwon closed pull request #34705: URL: https://github.com/apache/spark/pull/34705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757187262 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757187015 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757187015 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] AngersZhuuuu commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979595176 > Guys, im gonna revert this - this fix is incomplete, and we should keep the name same as `hadoop3` instead of `hadoop3.3`. So what should I do next? raise a new pr

[GitHub] [spark] AmplabJenkins commented on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979596029 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50110/ --

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757187812 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] HyukjinKwon commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979600444 Yes, renaming profiles and release file names were discussed at https://mail-archives.apache.org/mod_mbox/spark-dev/202106.mbox/browser. I don't mind who works on that but

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34705: [SPARK-37457][PYTHON] Update cloudpickle to v2.0.0

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34705: URL: https://github.com/apache/spark/pull/34705#issuecomment-979596029 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50110/

[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

2021-11-25 Thread GitBox
SparkQA commented on pull request #34513: URL: https://github.com/apache/spark/pull/34513#issuecomment-979602119 **[Test build #145640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145640/testReport)** for PR 34513 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34710: [SPARK-37461][YARN] YARN-CLIENT mode client.appId is always null

2021-11-25 Thread GitBox
AngersZh commented on a change in pull request #34710: URL: https://github.com/apache/spark/pull/34710#discussion_r757188225 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -181,7 +180,7 @@ private[spark] class Client(

[GitHub] [spark] HyukjinKwon edited a comment on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon edited a comment on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979600444 Yes, renaming profiles and release file names were discussed at

[GitHub] [spark] HyukjinKwon edited a comment on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon edited a comment on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979600444 Yes, renaming profiles and release file names were discussed at

[GitHub] [spark] AngersZhuuuu commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979610978 > Yes, renaming profiles and release file names were discussed at

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757189605 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh edited a comment on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979610978 > Yes, renaming profiles and release file names were discussed at

[GitHub] [spark] cloud-fan commented on a change in pull request #34702: [SPARK-37455][SQL] Replace hash with sort aggregate if child is already sorted

2021-11-25 Thread GitBox
cloud-fan commented on a change in pull request #34702: URL: https://github.com/apache/spark/pull/34702#discussion_r757190792 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAgg.scala ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache

[GitHub] [spark] SparkQA commented on pull request #34713: [SPARK-37464][SQL] SCHEMA and DATABASE should simply be aliases of NAMESPACE

2021-11-25 Thread GitBox
SparkQA commented on pull request #34713: URL: https://github.com/apache/spark/pull/34713#issuecomment-979619854 **[Test build #145641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145641/testReport)** for PR 34713 at commit

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34689: URL: https://github.com/apache/spark/pull/34689#discussion_r757191223 ## File path: dev/create-release/release-build.sh ## @@ -322,18 +322,18 @@ if [[ "$1" == "package" ]]; then # 'python/pyspark/install.py' and

[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

2021-11-25 Thread GitBox
SparkQA commented on pull request #34513: URL: https://github.com/apache/spark/pull/34513#issuecomment-979637964 **[Test build #145640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145640/testReport)** for PR 34513 at commit

[GitHub] [spark] sunchao commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
sunchao commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979645285 > So what should I do next? raise a new pr to change all to hadoop3? or left it to @sunchao Feel free to take over @AngersZh . I can help on reviewing. -- This is

[GitHub] [spark] AngersZhuuuu commented on pull request #34689: [SPARK-37445][BUILD] Rename the maven profile hadoop-3.2 to hadoop-3 and change GA test name to hadoop3.3

2021-11-25 Thread GitBox
AngersZh commented on pull request #34689: URL: https://github.com/apache/spark/pull/34689#issuecomment-979645710 > > So what should I do next? raise a new pr to change all to hadoop3? or left it to @sunchao > > Feel free to take over @AngersZh . I can help on reviewing.

[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py

2021-11-25 Thread GitBox
SparkQA removed a comment on pull request #34513: URL: https://github.com/apache/spark/pull/34513#issuecomment-979602119 **[Test build #145640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145640/testReport)** for PR 34513 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979487314 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50108/

[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979487314 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50108/ --

[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979493392 **[Test build #145637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145637/testReport)** for PR 34596 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
SparkQA removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979448590 **[Test build #145637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145637/testReport)** for PR 34596 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979493515 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145637/

[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-25 Thread GitBox
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-979493515 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145637/ -- This

[GitHub] [spark] pgandhi999 commented on pull request #34695: [WIP][SPARK-32446][CORE] Add percentile distribution REST API & UI of peak memory metrics for all executors

2021-11-25 Thread GitBox
pgandhi999 commented on pull request #34695: URL: https://github.com/apache/spark/pull/34695#issuecomment-979494372 I shall definitely review the PR, thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] c21 commented on a change in pull request #34702: [SPARK-37455][SQL] Replace hash with sort aggregate if child is already sorted

2021-11-25 Thread GitBox
c21 commented on a change in pull request #34702: URL: https://github.com/apache/spark/pull/34702#discussion_r757151713 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ## @@ -423,6 +423,9 @@ object QueryExecution {

[GitHub] [spark] zero323 opened a new pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
zero323 opened a new pull request #34714: URL: https://github.com/apache/spark/pull/34714 ### What changes were proposed in this pull request? This PR replaces `\url` commands with `\href`, when alias is provided. ### Why are the changes needed? `\url`

[GitHub] [spark] SparkQA commented on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
SparkQA commented on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979510661 **[Test build #145638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145638/testReport)** for PR 34714 at commit

[GitHub] [spark] HyukjinKwon closed pull request #34688: [SPARK-32079][PYTHON] Remove namedtuple hack by replacing built-in pickle to cloudpickle

2021-11-25 Thread GitBox
HyukjinKwon closed pull request #34688: URL: https://github.com/apache/spark/pull/34688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34688: [SPARK-32079][PYTHON] Remove namedtuple hack by replacing built-in pickle to cloudpickle

2021-11-25 Thread GitBox
HyukjinKwon commented on pull request #34688: URL: https://github.com/apache/spark/pull/34688#issuecomment-979512505 Thanks, @viirya. Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757157746 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757157440 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] zero323 closed pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
zero323 closed pull request #34714: URL: https://github.com/apache/spark/pull/34714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] zero323 commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
zero323 commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757157904 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757157746 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] zero323 commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
zero323 commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757158748 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757158047 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] c21 commented on a change in pull request #34702: [SPARK-37455][SQL] Replace hash with sort aggregate if child is already sorted

2021-11-25 Thread GitBox
c21 commented on a change in pull request #34702: URL: https://github.com/apache/spark/pull/34702#discussion_r757158837 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAgg.scala ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
SparkQA removed a comment on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979510661 **[Test build #145638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145638/testReport)** for PR 34714 at commit

[GitHub] [spark] SparkQA commented on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
SparkQA commented on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979519498 **[Test build #145638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145638/testReport)** for PR 34714 at commit

[GitHub] [spark] zero323 commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
zero323 commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757161174 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
HyukjinKwon commented on a change in pull request #34714: URL: https://github.com/apache/spark/pull/34714#discussion_r757161338 ## File path: R/pkg/R/DataFrame.R ## @@ -890,7 +890,7 @@ setMethod("toJSON", #' save mode (it is 'error' by default) #' @param ...

[GitHub] [spark] SparkQA commented on pull request #34714: [MINOR][R][DOCS] Rewrite \url to \href, if name is provided

2021-11-25 Thread GitBox
SparkQA commented on pull request #34714: URL: https://github.com/apache/spark/pull/34714#issuecomment-979522114 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50109/ -- This is an automated message from the Apache

[GitHub] [spark] xinrong-databricks commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat

2021-11-25 Thread GitBox
xinrong-databricks commented on a change in pull request #34354: URL: https://github.com/apache/spark/pull/34354#discussion_r757162250 ## File path: python/pyspark/sql/functions.py ## @@ -3514,7 +3538,19 @@ def map_from_arrays(col1: "ColumnOrName", col2: "ColumnOrName") ->

[GitHub] [spark] SparkQA commented on pull request #33896: [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle

2021-11-25 Thread GitBox
SparkQA commented on pull request #33896: URL: https://github.com/apache/spark/pull/33896#issuecomment-979423302 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50107/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-11-25 Thread GitBox
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-979427252 **[Test build #145632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145632/testReport)** for PR 34367 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

2021-11-25 Thread GitBox
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-979427976 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145632/

[GitHub] [spark] SparkQA removed a comment on pull request #34713: [SPARK-37464][SQL] SCHEMA and DATABASE should simply be aliases of NAMESPACE

2021-11-25 Thread GitBox
SparkQA removed a comment on pull request #34713: URL: https://github.com/apache/spark/pull/34713#issuecomment-979287233 **[Test build #145633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145633/testReport)** for PR 34713 at commit

<    1   2   3   4   5   6   >