[GitHub] [spark] cloud-fan closed pull request #35308: [SPARK-38011][SQL] Remove duplicated and useless configuration in ParquetFileFormat

2022-01-26 Thread GitBox
cloud-fan closed pull request #35308: URL: https://github.com/apache/spark/pull/35308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] cloud-fan commented on pull request #35308: [SPARK-38011][SQL] Remove duplicated and useless configuration in ParquetFileFormat

2022-01-26 Thread GitBox
cloud-fan commented on pull request #35308: URL: https://github.com/apache/spark/pull/35308#issuecomment-1022939904 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] AmplabJenkins commented on pull request #35340: [WINDOW]improve structured streaming window of calculated

2022-01-26 Thread GitBox
AmplabJenkins commented on pull request #35340: URL: https://github.com/apache/spark/pull/35340#issuecomment-1022938396 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on pull request #32875: [SPARK-35703][SQL] Relax constraint for bucket join and remove HashClusteredDistribution

2022-01-26 Thread GitBox
cloud-fan commented on pull request #32875: URL: https://github.com/apache/spark/pull/32875#issuecomment-1022926161 @sunchao can we add back `HashClusteredDistribution` and use it for streaming join/aggregate? -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] HeartSaVioR edited a comment on pull request #32875: [SPARK-35703][SQL] Relax constraint for bucket join and remove HashClusteredDistribution

2022-01-26 Thread GitBox
HeartSaVioR edited a comment on pull request #32875: URL: https://github.com/apache/spark/pull/32875#issuecomment-1022894109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] LuciferYang commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support

2022-01-26 Thread GitBox
LuciferYang commented on pull request #35262: URL: https://github.com/apache/spark/pull/35262#issuecomment-1022919561 @parthchandra I think we should add some UTs similar to `String with Nulls Scan` because when I add ```

[GitHub] [spark] HyukjinKwon commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
HyukjinKwon commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r793302012 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -605,6 +606,66 @@ private[spark] object Utils extends Logging { } }

[GitHub] [spark] LuciferYang edited a comment on pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang edited a comment on pull request #35278: URL: https://github.com/apache/spark/pull/35278#issuecomment-1022899627 @zhongjingxiong Please check the failure of GA(`Build and test`) -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] LuciferYang commented on pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on pull request #35278: URL: https://github.com/apache/spark/pull/35278#issuecomment-1022899627 @zhongjingxiong Please pay attention to the failure of GA(`Build and test`) -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] zhongjingxiong commented on pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
zhongjingxiong commented on pull request #35278: URL: https://github.com/apache/spark/pull/35278#issuecomment-1022897967 @HyukjinKwon @LuciferYang @Yikf I used the zipfile under the common package, ZipArchiveEntry.getUnixMode() can expose the permissions of the file, so I use the

[GitHub] [spark] HeartSaVioR commented on pull request #32875: [SPARK-35703][SQL] Relax constraint for bucket join and remove HashClusteredDistribution

2022-01-26 Thread GitBox
HeartSaVioR commented on pull request #32875: URL: https://github.com/apache/spark/pull/32875#issuecomment-1022894109 @sunchao Sorry for the post-review. I didn't know this PR may affect streaming query and indicated later. I discussed with @cloud-fan about this change, and

[GitHub] [spark] Yikun edited a comment on pull request #34983: [SPARK-37713][K8S] Assign namespace to executor configmap

2022-01-26 Thread GitBox
Yikun edited a comment on pull request #34983: URL: https://github.com/apache/spark/pull/34983#issuecomment-1014248788 FYI, this PR breaks the case of using ConfigMap with namespace specified (driver side), see https://github.com/apache/spark/pull/35215 . You could using below cmd

[GitHub] [spark] GabeChurch commented on pull request #34983: [SPARK-37713][K8S] Assign namespace to executor configmap

2022-01-26 Thread GitBox
GabeChurch commented on pull request #34983: URL: https://github.com/apache/spark/pull/34983#issuecomment-1022874830 Noticing the same behavior as Yikun with this pull breaking ConfigMap usage in non "default" namespace(s). -- This is an automated message from the Apache Git Service.

[GitHub] [spark] itholic commented on a change in pull request #35252: [SPARK-37154][PYTHON] Inline hints for pyspark.rdd

2022-01-26 Thread GitBox
itholic commented on a change in pull request #35252: URL: https://github.com/apache/spark/pull/35252#discussion_r793259415 ## File path: python/pyspark/_typing.pyi ## @@ -17,17 +17,27 @@ # under the License. from typing import Callable, Iterable, Sized, TypeVar, Union

[GitHub] [spark] itholic commented on a change in pull request #35252: [SPARK-37154][PYTHON] Inline hints for pyspark.rdd

2022-01-26 Thread GitBox
itholic commented on a change in pull request #35252: URL: https://github.com/apache/spark/pull/35252#discussion_r793259415 ## File path: python/pyspark/_typing.pyi ## @@ -17,17 +17,27 @@ # under the License. from typing import Callable, Iterable, Sized, TypeVar, Union

[GitHub] [spark] itholic commented on a change in pull request #35252: [SPARK-37154][PYTHON] Inline hints for pyspark.rdd

2022-01-26 Thread GitBox
itholic commented on a change in pull request #35252: URL: https://github.com/apache/spark/pull/35252#discussion_r793259415 ## File path: python/pyspark/_typing.pyi ## @@ -17,17 +17,27 @@ # under the License. from typing import Callable, Iterable, Sized, TypeVar, Union

[GitHub] [spark] otterc commented on a change in pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
otterc commented on a change in pull request #35325: URL: https://github.com/apache/spark/pull/35325#discussion_r793258742 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java ## @@ -1161,8 +1161,8 @@ public

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793256119 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -309,6 +393,116 @@ DESC StudentInfo; |age| int| NULL|

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793255964 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -309,6 +393,116 @@ DESC StudentInfo; |age| int| NULL|

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793255420 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -101,6 +145,27 @@ ALTER TABLE table_identifier { ALTER | CHANGE } [ COLUMN ] col_spec

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793253463 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -75,6 +75,50 @@ ALTER TABLE table_identifier ADD COLUMNS ( col_spec [ , ... ] )

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793252862 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -75,6 +75,50 @@ ALTER TABLE table_identifier ADD COLUMNS ( col_spec [ , ... ] )

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793249421 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -75,6 +75,50 @@ ALTER TABLE table_identifier ADD COLUMNS ( col_spec [ , ... ] )

[GitHub] [spark] huaxingao commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

2022-01-26 Thread GitBox
huaxingao commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r793248260 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -75,6 +75,50 @@ ALTER TABLE table_identifier ADD COLUMNS ( col_spec [ , ... ] )

[GitHub] [spark] HyukjinKwon closed pull request #35339: [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules

2022-01-26 Thread GitBox
HyukjinKwon closed pull request #35339: URL: https://github.com/apache/spark/pull/35339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #35339: [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules

2022-01-26 Thread GitBox
HyukjinKwon commented on pull request #35339: URL: https://github.com/apache/spark/pull/35339#issuecomment-1022840884 Thanks all! Mima passed. Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] iRakson commented on pull request #35337: [SPARK-37840][SQL] Dynamic Update of UDF

2022-01-26 Thread GitBox
iRakson commented on pull request #35337: URL: https://github.com/apache/spark/pull/35337#issuecomment-1022838418 @HyukjinKwon @dongjoon-hyun Kindly take a look at this PR. UT failure is not related to changes made in this PR. More UTs will be added covering all other scenarios.

[GitHub] [spark] nyingping opened a new pull request #35340: [WINDOW]improve structured streaming window of calculated

2022-01-26 Thread GitBox
nyingping opened a new pull request #35340: URL: https://github.com/apache/spark/pull/35340 **What changes were proposed in this pull request?** Remove the `CaseWhen`,Modified the calculation method of the obtained window **Why are the changes needed?** More simpler

[GitHub] [spark] melin commented on pull request #35337: [SPARK-37840][SQL] Dynamic Update of UDF

2022-01-26 Thread GitBox
melin commented on pull request #35337: URL: https://github.com/apache/spark/pull/35337#issuecomment-1022829887 Update jar by CREATE OR REPLACE command, it is not very convenient, it is best to insist on jar file modification time update, take effect directly -- This is an automated

[GitHub] [spark] Yikun commented on a change in pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

2022-01-26 Thread GitBox
Yikun commented on a change in pull request #34212: URL: https://github.com/apache/spark/pull/34212#discussion_r793225136 ## File path: python/pyspark/pandas/series.py ## @@ -4483,6 +4487,181 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] Yikun commented on a change in pull request #34212: [SPARK-36402][PYTHON] Implement Series.combine

2022-01-26 Thread GitBox
Yikun commented on a change in pull request #34212: URL: https://github.com/apache/spark/pull/34212#discussion_r793219057 ## File path: python/pyspark/pandas/series.py ## @@ -4483,6 +4487,181 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] HyukjinKwon commented on pull request #35339: [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules

2022-01-26 Thread GitBox
HyukjinKwon commented on pull request #35339: URL: https://github.com/apache/spark/pull/35339#issuecomment-1022808335 cc @HeartSaVioR @cloud-fan @gengliangwang @dongjoon-hyun FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon opened a new pull request #35339: [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules

2022-01-26 Thread GitBox
HyukjinKwon opened a new pull request #35339: URL: https://github.com/apache/spark/pull/35339 ### What changes were proposed in this pull request? We don't currently run binary compatibility check in below modules: ``` [info] spark-parent: mimaPreviousArtifacts not set, not

[GitHub] [spark] Yikf commented on pull request #35337: [SPARK-37840][SQL] Dynamic Update of UDF

2022-01-26 Thread GitBox
Yikf commented on pull request #35337: URL: https://github.com/apache/spark/pull/35337#issuecomment-1022808022 > Support Permanent Function? The information from suite describe that permanent should be supportive -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] LuciferYang commented on a change in pull request #35335: [SPARK-38036][SQL][TESTS] Refactor `VersionsSuite` to `HiveClientSuite` and make it a subclass of `HiveVersionSuite`

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35335: URL: https://github.com/apache/spark/pull/35335#discussion_r793220735 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala ## @@ -0,0 +1,1072 @@ +/* + * Licensed to the Apache

[GitHub] [spark] LuciferYang commented on a change in pull request #35335: [SPARK-38036][SQL][TESTS] Refactor `VersionsSuite` to `HiveClientSuite` and make it a subclass of `HiveVersionSuite`

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35335: URL: https://github.com/apache/spark/pull/35335#discussion_r793219679 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ## @@ -1,1159 +0,0 @@ -/* - * Licensed to the Apache

[GitHub] [spark] LuciferYang commented on a change in pull request #35335: [SPARK-38036][SQL][TESTS] Refactor `VersionsSuite` to `HiveClientSuite` and make it a subclass of `HiveVersionSuite`

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35335: URL: https://github.com/apache/spark/pull/35335#discussion_r793219289 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ## @@ -1,1159 +0,0 @@ -/* - * Licensed to the Apache

[GitHub] [spark] Kimahriman commented on pull request #35085: [SPARK-37618][CORE] Remove shuffle blocks using the shuffle service for released executors

2022-01-26 Thread GitBox
Kimahriman commented on pull request #35085: URL: https://github.com/apache/spark/pull/35085#issuecomment-1022803122 I went with the make subdirs 770 and make files world readable approach, seemed a lot less hacky than relying on ACLs. Will update the description with the approach --

[GitHub] [spark] LuciferYang commented on a change in pull request #35335: [SPARK-38036][SQL][TESTS] Refactor `VersionsSuite` to `HiveClientSuite` and make it a subclass of `HiveVersionSuite`

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35335: URL: https://github.com/apache/spark/pull/35335#discussion_r793218821 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ## @@ -1,1159 +0,0 @@ -/* - * Licensed to the Apache

[GitHub] [spark] Yikf commented on pull request #35308: [SPARK-38011][SQL] Remove duplicated and useless configuration in ParquetFileFormat

2022-01-26 Thread GitBox
Yikf commented on pull request #35308: URL: https://github.com/apache/spark/pull/35308#issuecomment-1022795343 @cloud-fan Could you please take a look when you have a time? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Yikf commented on pull request #35308: [SPARK-38011][SQL] Remove duplicated and useless configuration in ParquetFileFormat

2022-01-26 Thread GitBox
Yikf commented on pull request #35308: URL: https://github.com/apache/spark/pull/35308#issuecomment-1022795191 > A similar situation exists in `ParquetScan#createReaderFactory` Thanks a log for your reminder, updated -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] melin commented on pull request #35337: [SPARK-37840][SQL] Dynamic Update of UDF

2022-01-26 Thread GitBox
melin commented on pull request #35337: URL: https://github.com/apache/spark/pull/35337#issuecomment-1022791504 Support Permanent Function? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] mridulm commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
mridulm commented on pull request #35325: URL: https://github.com/apache/spark/pull/35325#issuecomment-1022786290 @otterc Can you remove the WIP tag, given the PR should be complete now ? Thx -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] mridulm commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #35325: URL: https://github.com/apache/spark/pull/35325#discussion_r793202348 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -534,35 +515,33 @@ public

[GitHub] [spark] itholic edited a comment on pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

2022-01-26 Thread GitBox
itholic edited a comment on pull request #35191: URL: https://github.com/apache/spark/pull/35191#issuecomment-1022785523 @pralabhkumar Sorry for being delayed. I've been busy for couple of days. Will take a closer look soon  -- This is an automated message from the Apache Git Service.

[GitHub] [spark] itholic commented on pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

2022-01-26 Thread GitBox
itholic commented on pull request #35191: URL: https://github.com/apache/spark/pull/35191#issuecomment-1022785523 @pralabhkumar Sorry for being delayed. I've been busy couple of days. Will take a closer look soon  -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] dchvn closed pull request #34178: [SPARK-36296][SQL] Refactor seventh set of 20 in QueryExecutionErrors to use error classes

2022-01-26 Thread GitBox
dchvn closed pull request #34178: URL: https://github.com/apache/spark/pull/34178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #35289: [SPARK-37397][PYTHON] Inline annotations for pyspark.ml.base

2022-01-26 Thread GitBox
HyukjinKwon commented on pull request #35289: URL: https://github.com/apache/spark/pull/35289#issuecomment-1022757181 cc @WeichenXu123 FYI if you find some time to review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] zero323 commented on pull request #35289: [SPARK-37397][PYTHON] Inline annotations for pyspark.ml.base

2022-01-26 Thread GitBox
zero323 commented on pull request #35289: URL: https://github.com/apache/spark/pull/35289#issuecomment-1022755067 cc @itholic @ueshin @xinrong-databricks FYI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon commented on pull request #35333: [SPARK-38035][SQL] Add docker tests for build-in JDBC dialect

2022-01-26 Thread GitBox
HyukjinKwon commented on pull request #35333: URL: https://github.com/apache/spark/pull/35333#issuecomment-1022753153 cc @sarutak fyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] beliefer commented on pull request #35041: [SPARK-37691][SQL] Support ANSI Aggregation Function: `percentile_disc`

2022-01-26 Thread GitBox
beliefer commented on pull request #35041: URL: https://github.com/apache/spark/pull/35041#issuecomment-1022751663 ping @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] beliefer commented on pull request #35333: [SPARK-38035][SQL] Add docker tests for build-in JDBC dialect

2022-01-26 Thread GitBox
beliefer commented on pull request #35333: URL: https://github.com/apache/spark/pull/35333#issuecomment-1022735880 ping @huaxingao cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] github-actions[bot] commented on pull request #34178: [SPARK-36296][SQL] Refactor seventh set of 20 in QueryExecutionErrors to use error classes

2022-01-26 Thread GitBox
github-actions[bot] commented on pull request #34178: URL: https://github.com/apache/spark/pull/34178#issuecomment-1022720525 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] xkrogen commented on a change in pull request #35332: [SPARK-38030][SQL] Canonicalization of cast should remove nullability of target dataType

2022-01-26 Thread GitBox
xkrogen commented on a change in pull request #35332: URL: https://github.com/apache/spark/pull/35332#discussion_r793120223 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CanonicalizeSuite.scala ## @@ -177,4 +177,18 @@ class

[GitHub] [spark] shardulm94 commented on a change in pull request #35332: [SPARK-38030][SQL] Canonicalization of cast should remove nullability of target dataType

2022-01-26 Thread GitBox
shardulm94 commented on a change in pull request #35332: URL: https://github.com/apache/spark/pull/35332#discussion_r793102644 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala ## @@ -310,13 +310,16 @@ abstract class CastBase

[GitHub] [spark] dongjoon-hyun commented on pull request #35338: [SPARK-37934][BUILD][3.2] Upgrade Jetty version to 9.4.44

2022-01-26 Thread GitBox
dongjoon-hyun commented on pull request #35338: URL: https://github.com/apache/spark/pull/35338#issuecomment-1022659030 @JackBuggins . You can reopen this after discussing about the requirement on this PR. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dongjoon-hyun removed a comment on pull request #35338: [SPARK-37934][BUILD][3.2] Upgrade Jetty version to 9.4.44

2022-01-26 Thread GitBox
dongjoon-hyun removed a comment on pull request #35338: URL: https://github.com/apache/spark/pull/35338#issuecomment-1022657954 BTW, @JackBuggins . Could you enable GitHub Action in your forked Spark repository? - https://github.com/apache/spark/pull/35338/checks?check_run_id=4957367779

[GitHub] [spark] dongjoon-hyun closed pull request #35338: [SPARK-37934][BUILD][3.2] Upgrade Jetty version to 9.4.44

2022-01-26 Thread GitBox
dongjoon-hyun closed pull request #35338: URL: https://github.com/apache/spark/pull/35338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] dongjoon-hyun commented on pull request #35338: [SPARK-37934][BUILD][3.2] Upgrade Jetty version to 9.4.44

2022-01-26 Thread GitBox
dongjoon-hyun commented on pull request #35338: URL: https://github.com/apache/spark/pull/35338#issuecomment-1022657954 BTW, @JackBuggins . Could you enable GitHub Action in your forked Spark repository? - https://github.com/apache/spark/pull/35338/checks?check_run_id=4957367779 --

[GitHub] [spark] dongjoon-hyun commented on pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

2022-01-26 Thread GitBox
dongjoon-hyun commented on pull request #35230: URL: https://github.com/apache/spark/pull/35230#issuecomment-1022656387 In general, the dependency change on the release branch is -1 by default if there is no significant needs, @JackBuggins . > None to my knowledge @HyukjinKwon, but

[GitHub] [spark] c21 commented on pull request #35273: [SPARK-37983][SQL] Back out agg build time metrics from sort aggregate

2022-01-26 Thread GitBox
c21 commented on pull request #35273: URL: https://github.com/apache/spark/pull/35273#issuecomment-1022616989 @cloud-fan - the PR is ready for review again, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r793016468 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] JackBuggins opened a new pull request #35338: [SPARK-37934][BUILD] Upgrade Jetty version to 9.4.44 (3.2 Stream)

2022-01-26 Thread GitBox
JackBuggins opened a new pull request #35338: URL: https://github.com/apache/spark/pull/35338 ### What changes were proposed in this pull request? Mirroring https://github.com/apache/spark/pull/35230 but applying this to the 3.2 stream; the only difference is that

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r793016468 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r793006586 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] otterc commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
otterc commented on a change in pull request #35325: URL: https://github.com/apache/spark/pull/35325#discussion_r792983163 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -1008,19 +995,28 @@

[GitHub] [spark] otterc commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
otterc commented on a change in pull request #35325: URL: https://github.com/apache/spark/pull/35325#discussion_r792982698 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java ## @@ -1287,6 +1287,52 @@ void

[GitHub] [spark] xkrogen commented on a change in pull request #35332: [SPARK-38030][SQL] Canonicalization of cast should remove nullability of target dataType

2022-01-26 Thread GitBox
xkrogen commented on a change in pull request #35332: URL: https://github.com/apache/spark/pull/35332#discussion_r792961446 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CanonicalizeSuite.scala ## @@ -177,4 +177,18 @@ class

[GitHub] [spark] iRakson opened a new pull request #35337: [SPARK-37840][SQL] Dynamic Update of UDF

2022-01-26 Thread GitBox
iRakson opened a new pull request #35337: URL: https://github.com/apache/spark/pull/35337 ### What changes were proposed in this pull request? Currently, If we need to change definitions of our UDFs, we need to restart thriftserver. In production environment this is not always

[GitHub] [spark] parthchandra edited a comment on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support

2022-01-26 Thread GitBox
parthchandra edited a comment on pull request #35262: URL: https://github.com/apache/spark/pull/35262#issuecomment-1022462107 > > Updated the JDK 8 benchmark results as well. > > After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the

[GitHub] [spark] parthchandra commented on pull request #35262: [SPARK-37974][SQL] Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support

2022-01-26 Thread GitBox
parthchandra commented on pull request #35262: URL: https://github.com/apache/spark/pull/35262#issuecomment-1022462107 > > Updated the JDK 8 benchmark results as well. > > After comparing the new bench data, I find that the data corresponding to `Parquet Data Page V2` in the two

[GitHub] [spark] senthh opened a new pull request #35336: [SPARK-37936][SQL] Use error classes in the parsing errors of intervals

2022-01-26 Thread GitBox
senthh opened a new pull request #35336: URL: https://github.com/apache/spark/pull/35336 n the PR, We propose to throw ParseException from below methodswith the error classes: moreThanOneFromToUnitInIntervalLiteralError invalidIntervalLiteralError invalidIntervalFormError

[GitHub] [spark] c21 commented on pull request #35324: [SPARK-37896][SQL][FOLLOWUP] Fix NPE in ConstantColumnVector.close()

2022-01-26 Thread GitBox
c21 commented on pull request #35324: URL: https://github.com/apache/spark/pull/35324#issuecomment-1022421010 Thank you @cloud-fan and @Yaohua628 for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] MaxGekk commented on a change in pull request #35302: [SPARK-38001][SQL] Replace the error classes related to unsupported features by `UNSUPPORTED_FEATURE`

2022-01-26 Thread GitBox
MaxGekk commented on a change in pull request #35302: URL: https://github.com/apache/spark/pull/35302#discussion_r792819398 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala ## @@ -93,8 +93,8 @@ object QueryCompilationErrors {

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792766502 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792766502 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792766502 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792766502 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm commented on pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on pull request #34122: URL: https://github.com/apache/spark/pull/34122#issuecomment-1022330045 Can you take a look at the build failures @venkata91 ? Not sure if it requires to be updated to master. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792766502 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -114,15 +114,6 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792761973 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -135,6 +144,7 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r792761973 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -135,6 +144,7 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] mridulm edited a comment on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
mridulm edited a comment on pull request #35325: URL: https://github.com/apache/spark/pull/35325#issuecomment-1022312145 @pan3793 Functionally, @otterc's patch looks good (except for testing enhancements, renames, etc). Can you also confirm if this works for you, given that your env has

[GitHub] [spark] mridulm commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
mridulm commented on pull request #35325: URL: https://github.com/apache/spark/pull/35325#issuecomment-1022312145 @pan3793 Functionally, @otterc's patch looks good (except for testing enhancements, renames, etc). Can you also confirm if this works for you ? Thanks ! -- This is an

[GitHub] [spark] mridulm commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

2022-01-26 Thread GitBox
mridulm commented on a change in pull request #35325: URL: https://github.com/apache/spark/pull/35325#discussion_r792754171 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -1008,19 +995,28 @@

[GitHub] [spark] imback82 commented on pull request #35328: [SPARK-37937][SQL] Use error classes in the parsing errors of lateral join

2022-01-26 Thread GitBox
imback82 commented on pull request #35328: URL: https://github.com/apache/spark/pull/35328#issuecomment-1022307238 > @imback82 Could you fix the JIRA id in PR's title. Seems SPARK-37858 is not correct. Updated, thanks! -- This is an automated message from the Apache Git Service.

[GitHub] [spark] JackBuggins commented on pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

2022-01-26 Thread GitBox
JackBuggins commented on pull request #35230: URL: https://github.com/apache/spark/pull/35230#issuecomment-1022282155 None to my knowledge @HyukjinKwon, but wrt https://github.com/eclipse/jetty.project/issues/6973, my thinking is it would offer peace of mind -- This is an automated

[GitHub] [spark] Yikun edited a comment on pull request #35299: [SPARK-37916][SPARK-37713][K8S] Revert "Assign namespace to executor configmap"

2022-01-26 Thread GitBox
Yikun edited a comment on pull request #35299: URL: https://github.com/apache/spark/pull/35299#issuecomment-1019849743 FYI @dcoliversun I contacted with @dcoliversun, in the offline discussion, we have confirmed there are two problems are introduced in PR

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792679337 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -595,7 +593,7 @@ private[spark] object Utils extends Logging { if

[GitHub] [spark] HyukjinKwon commented on a change in pull request #35246: [SPARK-37929][SQL] Support cascade mode for `dropNamespace` API

2022-01-26 Thread GitBox
HyukjinKwon commented on a change in pull request #35246: URL: https://github.com/apache/spark/pull/35246#discussion_r792675110 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java ## @@ -136,15 +137,20 @@ void

[GitHub] [spark] HyukjinKwon commented on a change in pull request #35246: [SPARK-37929][SQL] Support cascade mode for `dropNamespace` API

2022-01-26 Thread GitBox
HyukjinKwon commented on a change in pull request #35246: URL: https://github.com/apache/spark/pull/35246#discussion_r792675110 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java ## @@ -136,15 +137,20 @@ void

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792673852 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -605,6 +603,22 @@ private[spark] object Utils extends Logging { } }

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792670450 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -605,6 +603,29 @@ private[spark] object Utils extends Logging { } }

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792668929 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -3196,8 +3217,8 @@ private[spark] object Utils extends Logging {

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792668929 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -3196,8 +3217,8 @@ private[spark] object Utils extends Logging {

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792667649 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -55,14 +53,14 @@ import org.apache.hadoop.conf.Configuration import

[GitHub] [spark] LuciferYang commented on a change in pull request #35278: [SPARK-37677][CORE] Use the shell command to decompress the ZIP file

2022-01-26 Thread GitBox
LuciferYang commented on a change in pull request #35278: URL: https://github.com/apache/spark/pull/35278#discussion_r792666914 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -32,7 +32,6 @@ import java.util.{Locale, Properties, Random, UUID} import

[GitHub] [spark] cloud-fan commented on a change in pull request #35246: [SPARK-37929][SQL] Support cascade mode for `dropNamespace` API

2022-01-26 Thread GitBox
cloud-fan commented on a change in pull request #35246: URL: https://github.com/apache/spark/pull/35246#discussion_r792639178 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java ## @@ -136,15 +137,20 @@ void alterNamespace(

[GitHub] [spark] srowen commented on pull request #35307: [SPARK-38008][CORE] Fix the method description of refill

2022-01-26 Thread GitBox
srowen commented on pull request #35307: URL: https://github.com/apache/spark/pull/35307#issuecomment-1022198541 Any other typos? you can run a spell check. No need for a JIRA for something like this. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon commented on a change in pull request #35246: [SPARK-37929][SQL] Support cascade mode for `dropNamespace` API

2022-01-26 Thread GitBox
HyukjinKwon commented on a change in pull request #35246: URL: https://github.com/apache/spark/pull/35246#discussion_r792623987 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java ## @@ -136,15 +137,20 @@ void

  1   2   >