Re: [PR] [SPARK-46507][PS][TESTS] Split `IndexesSlowTests` into multiple tests [spark]

zhengruifeng commented on PR #44491: URL: https://github.com/apache/spark/pull/44491#issuecomment-1869333957 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46507][PS][TESTS] Split `IndexesSlowTests` into multiple tests [spark]

zhengruifeng closed pull request #44491: [SPARK-46507][PS][TESTS] Split `IndexesSlowTests` into multiple tests URL: https://github.com/apache/spark/pull/44491 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] [SPARK-46510][CORE] Spark shell log filter should be applied to all AbstractAppender [spark]

AngersZh opened a new pull request, #44496: URL: https://github.com/apache/spark/pull/44496 ### What changes were proposed in this pull request? Current code `SparkShellLoggingFilter` only applied to `ConsoleAppender`, if we define `AsyncAppender` and ref to console appender it won't

Re: [PR] [SPARK-46444][SQL] V2SessionCatalog#createTable should not load the table [spark]

cloud-fan closed pull request #44377: [SPARK-46444][SQL] V2SessionCatalog#createTable should not load the table URL: https://github.com/apache/spark/pull/44377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-46444][SQL] V2SessionCatalog#createTable should not load the table [spark]

cloud-fan commented on PR #44377: URL: https://github.com/apache/spark/pull/44377#issuecomment-1869321189 thanks for the review, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-46502][SQL] Support timestamp types in UnwrapCastInBinaryComparison [spark]

viirya commented on PR #44480: URL: https://github.com/apache/spark/pull/44480#issuecomment-1869303202 > Apache Spark explicitly requires Timezone information for these kind of conversions. > >

[PR] [SPARK-46509][CORE] Replace `reverse.find` with `findLast` [spark]

LuciferYang opened a new pull request, #44495: URL: https://github.com/apache/spark/pull/44495 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

Re: [PR] [SPARK-46508][BUILD] Upgrade Jackson to 2.16.1 [spark]

LuciferYang commented on PR #44494: URL: https://github.com/apache/spark/pull/44494#issuecomment-1869293650 Test first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] [SPARK-46508][BUILD] Upgrade Jackson to 2.16.1 [spark]

LuciferYang opened a new pull request, #44494: URL: https://github.com/apache/spark/pull/44494 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[PR] [SPARK-46478][SQL][3.5] Revert SPARK-43049 to use oracle varchar(255) for string [spark]

yaooqinn opened a new pull request, #44493: URL: https://github.com/apache/spark/pull/44493 ### What changes were proposed in this pull request? Revert SPARK-43049 to use Oracle Varchar (255) for string for performance consideration ### Why are the changes needed?

[PR] [SPARK-46504][PS][TESTS][FOLLOWUPS] Make `test_insert` more stable by sorting before comparison [spark]

zhengruifeng opened a new pull request, #44492: URL: https://github.com/apache/spark/pull/44492 ### What changes were proposed in this pull request? Make `test_insert` more stable by sorting before comparison ### Why are the changes needed? this test may fail in some env

Re: [PR] [MINOR][DOCS] Fix rst link in Python API docs for .sql() [spark]

HyukjinKwon commented on PR #44488: URL: https://github.com/apache/spark/pull/44488#issuecomment-1869277788 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [MINOR][DOCS] Fix rst link in Python API docs for .sql() [spark]

HyukjinKwon closed pull request #44488: [MINOR][DOCS] Fix rst link in Python API docs for .sql() URL: https://github.com/apache/spark/pull/44488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-45597][PYTHON][SQL][FOLLOW-UP] Minor deduplicate datasource checking logic [spark]

HyukjinKwon closed pull request #44489: [SPARK-45597][PYTHON][SQL][FOLLOW-UP] Minor deduplicate datasource checking logic URL: https://github.com/apache/spark/pull/44489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-45597][PYTHON][SQL][FOLLOW-UP] Minor deduplicate datasource checking logic [spark]

HyukjinKwon commented on PR #44489: URL: https://github.com/apache/spark/pull/44489#issuecomment-1869277534 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46504][PS][TESTS] Factor slow tests out of `IndexesTests` [spark]

zhengruifeng commented on PR #44485: URL: https://github.com/apache/spark/pull/44485#issuecomment-1869263244 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46504][PS][TESTS] Factor slow tests out of `IndexesTests` [spark]

zhengruifeng closed pull request #44485: [SPARK-46504][PS][TESTS] Factor slow tests out of `IndexesTests` URL: https://github.com/apache/spark/pull/44485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-46507][PS][TESTS] Split `IndexesSlowTests` into multiple tests [spark]

zhengruifeng commented on PR #44491: URL: https://github.com/apache/spark/pull/44491#issuecomment-1869236695 ci: https://github.com/zhengruifeng/spark/actions/runs/7326412348/job/19952061265 -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] [SPARK-46507][PS][TESTS] Split `IndexesSlowTests` into multiple tests [spark]

zhengruifeng opened a new pull request, #44491: URL: https://github.com/apache/spark/pull/44491 ### What changes were proposed in this pull request? Split `IndexesSlowTests` into multiple tests ### Why are the changes needed? for testing parallelism ### Does this PR

[PR] [SPARK-46506][PYTHON][DOCS] Refine docstring of `array_intersect/array_union/array_except` [spark]

LuciferYang opened a new pull request, #44490: URL: https://github.com/apache/spark/pull/44490 ### What changes were proposed in this pull request? This pr refine docstring of `array_intersect/array_union/array_except` and add some new examples. ### Why are the changes needed?

Re: [PR] [SPARK-46505][CONNECT] Make bytes threshold configurable in `ProtoUtils.abbreviate` [spark]

zhengruifeng commented on PR #44486: URL: https://github.com/apache/spark/pull/44486#issuecomment-1869213610 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46505][CONNECT] Make bytes threshold configurable in `ProtoUtils.abbreviate` [spark]

zhengruifeng closed pull request #44486: [SPARK-46505][CONNECT] Make bytes threshold configurable in `ProtoUtils.abbreviate` URL: https://github.com/apache/spark/pull/44486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [MINOR][DOCS] Fix rst link in Python API docs for .sql() [spark]

HyukjinKwon commented on PR #44488: URL: https://github.com/apache/spark/pull/44488#issuecomment-1869210958 Build: https://github.com/nchammas/spark/actions/runs/7325659058/job/19950500574 (for my own ref) -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [SPARK-45597][PYTHON][SQL][FOLLOw-UP] Refactor User defined source checking logic to `DataSourceManager` [spark]

HyukjinKwon commented on PR #44489: URL: https://github.com/apache/spark/pull/44489#issuecomment-1869210593 cc @allisonwang-db and @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] [SPARK-45597][PYTHON][SQL][FOLLOw-UP] Refactor User defined source checking logic to `DataSourceManager` [spark]

HyukjinKwon opened a new pull request, #44489: URL: https://github.com/apache/spark/pull/44489 ### What changes were proposed in this pull request? This PR proposes to factor the checking logic for user defined source to `DataSourceManager`. ### Why are the changes needed?

Re: [PR] [MINOR][DOCS] Fix rst link in Python API docs for .sql() [spark]

nchammas commented on PR #44488: URL: https://github.com/apache/spark/pull/44488#issuecomment-1869208844 cc @MaxGekk since `git blame` suggests you wrote the docs I am editing in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] [MINOR][DOCS] Fix rst link in Python API docs for .sql() [spark]

nchammas opened a new pull request, #44488: URL: https://github.com/apache/spark/pull/44488 ### What changes were proposed in this pull request? This PR fixes the rst markup for a link in the documentation for `pyspark.sql.SparkSession.sql` and `pyspark.pandas.sql`. ### Why

Re: [PR] [SPARK-45600][SQL][PYTHON][FOLLOW-UP] Make Python data source registration session level [spark]

HyukjinKwon closed pull request #44487: [SPARK-45600][SQL][PYTHON][FOLLOW-UP] Make Python data source registration session level URL: https://github.com/apache/spark/pull/44487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-45600][SQL][PYTHON][FOLLOW-UP] Make Python data source registration session level [spark]

HyukjinKwon commented on PR #44487: URL: https://github.com/apache/spark/pull/44487#issuecomment-1869207188 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

panbingkun closed pull request #44293: [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` URL: https://github.com/apache/spark/pull/44293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

panbingkun commented on PR #44293: URL: https://github.com/apache/spark/pull/44293#issuecomment-1869207055 > To @panbingkun , the above description should be in the PR description. Please add them into the PR description to be complete next time. > > In addition, this PR is wrong as

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

panbingkun commented on code in PR #44293: URL: https://github.com/apache/spark/pull/44293#discussion_r1436200013 ## dev/.rat-excludes: ## @@ -4,8 +4,6 @@ cache .gitattributes .project .classpath -.mima-excludes Review Comment: After this PR

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

panbingkun commented on code in PR #44293: URL: https://github.com/apache/spark/pull/44293#discussion_r1436200250 ## dev/.rat-excludes: ## @@ -4,8 +4,6 @@ cache .gitattributes .project .classpath -.mima-excludes -.generated-mima-excludes Review Comment: After this PR

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

panbingkun commented on code in PR #44293: URL: https://github.com/apache/spark/pull/44293#discussion_r1436200013 ## dev/.rat-excludes: ## @@ -4,8 +4,6 @@ cache .gitattributes .project .classpath -.mima-excludes Review Comment: After this PR

Re: [PR] [SPARK-45600][SQL][PYTHON][FOLLOW-UP] Make Python data source registration session level [spark]

HyukjinKwon commented on PR #44487: URL: https://github.com/apache/spark/pull/44487#issuecomment-1869198411 Build: https://github.com/HyukjinKwon/spark/actions/runs/7325792358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-46487][SQL] Push down part of filter through aggregate with nondeterministic field [spark]

zml1206 commented on code in PR #44460: URL: https://github.com/apache/spark/pull/44460#discussion_r1436194640 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala: ## @@ -1706,14 +1706,18 @@ object PushPredicateThroughNonJoin extends

[PR] [SPARK-46505][CONNECT] Make bytes threshold configurable in `ProtoUtils.abbreviate` [spark]

zhengruifeng opened a new pull request, #44486: URL: https://github.com/apache/spark/pull/44486 ### What changes were proposed in this pull request? Make bytes threshold configurable in `ProtoUtils.abbreviate` ### Why are the changes needed? the bytes threshold should be also

[PR] [SPARK-46504][PS][TESTS] Factor slow tests out of `IndexesTests` [spark]

zhengruifeng opened a new pull request, #44485: URL: https://github.com/apache/spark/pull/44485 ### What changes were proposed in this pull request? Factor slow tests out of `IndexesTests` ### Why are the changes needed? for testing parallelism ### Does this PR

Re: [PR] [MINOR][DOCS] Python docs also require Pandoc [spark]

HyukjinKwon closed pull request #44484: [MINOR][DOCS] Python docs also require Pandoc URL: https://github.com/apache/spark/pull/44484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [MINOR][DOCS] Python docs also require Pandoc [spark]

HyukjinKwon commented on PR #44484: URL: https://github.com/apache/spark/pull/44484#issuecomment-1869183939 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Do not convert array type string retrieved from jdbc driver [spark]

HyukjinKwon commented on PR #44459: URL: https://github.com/apache/spark/pull/44459#issuecomment-1869181617 Thanks for the PR. Mind creating a JIRA please? (see also https://spark.apache.org/contributing.html). -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] [SPARK-46487][SQL] Push down part of filter through aggregate with nondeterministic field [spark]

HyukjinKwon commented on code in PR #44460: URL: https://github.com/apache/spark/pull/44460#discussion_r1436181963 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala: ## @@ -1706,14 +1706,18 @@ object PushPredicateThroughNonJoin extends

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

panbingkun commented on PR #44293: URL: https://github.com/apache/spark/pull/44293#issuecomment-1869178216 > To @panbingkun , the above description should be in the PR description. Please add them into the PR description to be complete next time. > > In addition, this PR is wrong as

Re: [PR] [MINOR][DOCS] Python docs also require Pandoc [spark]

nchammas commented on PR #44484: URL: https://github.com/apache/spark/pull/44484#issuecomment-1869177234 cc @HyukjinKwon for this quick PR on building the Python docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[PR] [MINOR][DOCS] Python docs also require Pandoc [spark]

nchammas opened a new pull request, #44484: URL: https://github.com/apache/spark/pull/44484 ### What changes were proposed in this pull request? Clarify in the README for building docs that the Python API docs also require Pandoc, not just R. ### Why are the changes needed?

Re: [PR] [SPARK-40876][SQL] Widening type promotions in Parquet readers [spark]

LuciferYang commented on PR #44368: URL: https://github.com/apache/spark/pull/44368#issuecomment-1869172603 > @LuciferYang thanks for catching it! Does it block PR merging? We may need to wait for a few days as it's the holiday season. If you can fix it then it's even better. We can revert

Re: [PR] [SPARK-40876][SQL][TESTS][FOLLOWUP] Fix failed test in `ParquetTypeWideningSuite` when `SPARK_ANSI_SQL_MODE` is set to true [spark]

LuciferYang commented on PR #44481: URL: https://github.com/apache/spark/pull/44481#issuecomment-1869172083 Thanks @dongjoon-hyun and @cloud-fan ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-46497][SQL][TESTS] Re-enable the test cases that were ignored in SPARK-45309 [spark]

LuciferYang commented on PR #44474: URL: https://github.com/apache/spark/pull/44474#issuecomment-1869171804 > This PR is valid because all changed versions of `spark.sql.hive.metastore.version` is prior to 2.0.0 which is removed by the following JIRA. > > * [[SPARK-45328][SQL]

Re: [PR] [SPARK-46498][CORE] Remove an unused local variables from `o.a.spark.util.Utils#getConfiguredLocalDirs` [spark]

LuciferYang commented on PR #44475: URL: https://github.com/apache/spark/pull/44475#issuecomment-1869170503 > Sorry but let's not remove these `Utils` methods although they are `private[spark]`. There is no maintenance overhead for these util methods and these might be used by a new

Re: [PR] [SPARK-46498][CORE] Remove an unused local variables from `o.a.spark.util.Utils#getConfiguredLocalDirs` [spark]

LuciferYang commented on code in PR #44475: URL: https://github.com/apache/spark/pull/44475#discussion_r1436174934 ## core/src/main/scala/org/apache/spark/util/Utils.scala: ## @@ -769,7 +769,6 @@ private[spark] object Utils * logic of locating the local directories

Re: [PR] [SPARK-46502][SQL] Support timestamp types in UnwrapCastInBinaryComparison [spark]

HyukjinKwon commented on PR #44480: URL: https://github.com/apache/spark/pull/44480#issuecomment-1869170225 cc @gengliangwang and @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Monthname function [spark]

HyukjinKwon commented on code in PR #44483: URL: https://github.com/apache/spark/pull/44483#discussion_r1436174767 ## R/pkg/R/functions.R: ## @@ -1091,6 +1091,20 @@ setMethod("dayofyear", column(jc) }) +#' @details +#' \code{monthname}: Extracts the

Re: [PR] [SPARK-46503][PS][TESTS] Move `test_default_index` to `pyspark.pandas.tests.indexes.*` [spark]

zhengruifeng commented on PR #44482: URL: https://github.com/apache/spark/pull/44482#issuecomment-1869166863 thanks @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46503][PS][TESTS] Move `test_default_index` to `pyspark.pandas.tests.indexes.*` [spark]

HyukjinKwon closed pull request #44482: [SPARK-46503][PS][TESTS] Move `test_default_index` to `pyspark.pandas.tests.indexes.*` URL: https://github.com/apache/spark/pull/44482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-46503][PS][TESTS] Move `test_default_index` to `pyspark.pandas.tests.indexes.*` [spark]

HyukjinKwon commented on PR #44482: URL: https://github.com/apache/spark/pull/44482#issuecomment-1869166549 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46501][INFRA] List the python packages with the correct versions [spark]

zhengruifeng commented on PR #44479: URL: https://github.com/apache/spark/pull/44479#issuecomment-1869166455 thanks @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-43752][SQL] Support default column value on DataSource V2 [spark]

github-actions[bot] closed pull request #42802: [SPARK-43752][SQL] Support default column value on DataSource V2 URL: https://github.com/apache/spark/pull/42802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-45122][DOCS] Automate updating versions.json [spark]

github-actions[bot] commented on PR #42881: URL: https://github.com/apache/spark/pull/42881#issuecomment-1869163721 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-45152][ML] Add includeLowest Param to Bucketizer [spark]

github-actions[bot] commented on PR #42924: URL: https://github.com/apache/spark/pull/42924#issuecomment-1869163713 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-46498][CORE] Clean up unused functions and local variables in `o.a.spark.util.Utils` [spark]

dongjoon-hyun commented on code in PR #44475: URL: https://github.com/apache/spark/pull/44475#discussion_r1436167156 ## core/src/main/scala/org/apache/spark/util/Utils.scala: ## @@ -769,7 +769,6 @@ private[spark] object Utils * logic of locating the local directories

Re: [PR] [SPARK-40876][SQL][TESTS][FOLLOWUP] Fix failed test in `ParquetTypeWideningSuite` when `SPARK_ANSI_SQL_MODE` is set to true [spark]

dongjoon-hyun closed pull request #44481: [SPARK-40876][SQL][TESTS][FOLLOWUP] Fix failed test in `ParquetTypeWideningSuite` when `SPARK_ANSI_SQL_MODE` is set to true URL: https://github.com/apache/spark/pull/44481 -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Test Ivy 2.5.2 [spark]

dongjoon-hyun commented on PR #44477: URL: https://github.com/apache/spark/pull/44477#issuecomment-1869151005 Thank you for the investigation, @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-46501][INFRA] List the python packages with the correct versions [spark]

dongjoon-hyun closed pull request #44479: [SPARK-46501][INFRA] List the python packages with the correct versions URL: https://github.com/apache/spark/pull/44479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-46497][SQL][TESTS] Re-enable the test cases that were ignored in SPARK-45309 [spark]

dongjoon-hyun commented on PR #44474: URL: https://github.com/apache/spark/pull/44474#issuecomment-1869149174 Merged to master for Apache Spark 4.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-46497][SQL][TESTS] Re-enable the test cases that were ignored in SPARK-45309 [spark]

dongjoon-hyun closed pull request #44474: [SPARK-46497][SQL][TESTS] Re-enable the test cases that were ignored in SPARK-45309 URL: https://github.com/apache/spark/pull/44474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes` [spark]

dongjoon-hyun commented on PR #44293: URL: https://github.com/apache/spark/pull/44293#issuecomment-1869147941 To @panbingkun , the above description should be in the PR description. Please add them into the PR description to be complete next time. In addition, this PR is wrong as

Re: [PR] [SPARK-46475][BUILD] Upgrade RoaringBitmap to 1.0.1 [spark]

dongjoon-hyun commented on PR #44439: URL: https://github.com/apache/spark/pull/44439#issuecomment-1869147133 Merged to master for Apache Spark 4. Thank you, @panbingkun . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-46475][BUILD] Upgrade RoaringBitmap to 1.0.1 [spark]

dongjoon-hyun closed pull request #44439: [SPARK-46475][BUILD] Upgrade RoaringBitmap to 1.0.1 URL: https://github.com/apache/spark/pull/44439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-46499][BUILD] Bump sbt-eclipse 6.2.0 [spark]

dongjoon-hyun commented on PR #44476: URL: https://github.com/apache/spark/pull/44476#issuecomment-1869146745 Merged to master for Apache Spark 4. Thank you, @pan3793 and @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [SPARK-46499][BUILD] Bump sbt-eclipse 6.2.0 [spark]

dongjoon-hyun closed pull request #44476: [SPARK-46499][BUILD] Bump sbt-eclipse 6.2.0 URL: https://github.com/apache/spark/pull/44476 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46502][SQL] Support timestamp types in UnwrapCastInBinaryComparison [spark]

viirya commented on code in PR #44480: URL: https://github.com/apache/spark/pull/44480#discussion_r1436028917 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala: ## @@ -144,6 +144,15 @@ object UnwrapCastInBinaryComparison

Re: [PR] [SPARK-42332][SQL][DOCS] Changing the require to a SparkException in ComplexTypeMergingExpression [spark]

MaxGekk commented on code in PR #44336: URL: https://github.com/apache/spark/pull/44336#discussion_r1436132674 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala: ## @@ -1300,14 +1301,16 @@ trait ComplexTypeMergingExpression extends

Re: [PR] [SPARK-46502][SQL] Support timestamp types in UnwrapCastInBinaryComparison [spark]

viirya commented on code in PR #44480: URL: https://github.com/apache/spark/pull/44480#discussion_r1436028917 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala: ## @@ -144,6 +144,15 @@ object UnwrapCastInBinaryComparison

Re: [PR] [SPARK-41954][CORE] Add isDecommissioned in ExecutorDeadException [spark]

warrenzhu25 commented on PR #43475: URL: https://github.com/apache/spark/pull/43475#issuecomment-1869054134 @dongjoon-hyun @mridulm Help take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing [spark]

MaxGekk commented on PR #44463: URL: https://github.com/apache/spark/pull/44463#issuecomment-1869050659 @stefankandic Congratulations for your first contribution to Apache Spark! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing [spark]

MaxGekk closed pull request #44463: [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing URL: https://github.com/apache/spark/pull/44463 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing [spark]

MaxGekk commented on PR #44463: URL: https://github.com/apache/spark/pull/44463#issuecomment-1869049907 +1, LGTM. Merging to master. Thank you, @stefankandic and @dbatomic for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [SPARK-24497][SQL] Support recursive SQL [spark]

milastdbx commented on PR #40744: URL: https://github.com/apache/spark/pull/40744#issuecomment-1869048522 Yes, thank you. Milan On Thu, Dec 21, 2023 at 12:29 PM Peter Toth ***@***.***> wrote: > @milastdbx do you think you can take over

[PR] Monthname function [spark]

stefankandic opened a new pull request, #44483: URL: https://github.com/apache/spark/pull/44483 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on code in PR #44477: URL: https://github.com/apache/spark/pull/44477#discussion_r1436116952 ## sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala: ## @@ -220,6 +221,7 @@ class HiveExternalCatalogVersionsSuite extends

Re: [PR] [SPARK-46366][SQL] Use WITH expression in BETWEEN to avoid duplicate expressions [spark]

cloud-fan commented on code in PR #44299: URL: https://github.com/apache/spark/pull/44299#discussion_r1436104814 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala: ## @@ -96,17 +96,22 @@ object StreamingJoinHelper extends

Re: [PR] [SPARK-46481] Execute immediate VariableReference foldable [spark]

cloud-fan closed pull request #44450: [SPARK-46481] Execute immediate VariableReference foldable URL: https://github.com/apache/spark/pull/44450 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-46481] Execute immediate VariableReference foldable [spark]

cloud-fan commented on PR #44450: URL: https://github.com/apache/spark/pull/44450#issuecomment-1869007051 the failed tests are unrelated, merging to master, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on code in PR #44477: URL: https://github.com/apache/spark/pull/44477#discussion_r1436096023 ## sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala: ## @@ -220,6 +221,7 @@ class HiveExternalCatalogVersionsSuite extends

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on code in PR #44477: URL: https://github.com/apache/spark/pull/44477#discussion_r1436095877 ## sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala: ## @@ -211,6 +213,10 @@ class HiveExternalCatalogVersionsSuite extends

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on code in PR #44477: URL: https://github.com/apache/spark/pull/44477#discussion_r1436095877 ## sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala: ## @@ -211,6 +213,10 @@ class HiveExternalCatalogVersionsSuite extends

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on code in PR #44477: URL: https://github.com/apache/spark/pull/44477#discussion_r1436089675 ## sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala: ## @@ -211,6 +213,10 @@ class HiveExternalCatalogVersionsSuite extends

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on code in PR #44477: URL: https://github.com/apache/spark/pull/44477#discussion_r1436089675 ## sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala: ## @@ -211,6 +213,10 @@ class HiveExternalCatalogVersionsSuite extends

Re: [PR] [SPARK-40876][SQL][TESTS][FOLLOWUP] Fix failed test in `ParquetTypeWideningSuite` when `SPARK_ANSI_SQL_MODE` is set to true [spark]

LuciferYang commented on code in PR #44481: URL: https://github.com/apache/spark/pull/44481#discussion_r1436075804 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala: ## @@ -166,11 +166,17 @@ class

Re: [PR] [SPARK-40876][SQL][TESTS][FOLLOWUP] Skip the test inputs in `ParquetTypeWideningSuite` that violate the data type conversion rules under ANSI mode [spark]

LuciferYang commented on code in PR #44481: URL: https://github.com/apache/spark/pull/44481#discussion_r1436074533 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala: ## @@ -166,11 +166,17 @@ class

Re: [PR] [MINOR][INFRA] Comments in GitHub scripts should start with # [spark]

zhengruifeng commented on PR #44473: URL: https://github.com/apache/spark/pull/44473#issuecomment-1868941863 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [MINOR][INFRA] Comments in GitHub scripts should start with # [spark]

zhengruifeng closed pull request #44473: [MINOR][INFRA] Comments in GitHub scripts should start with # URL: https://github.com/apache/spark/pull/44473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] [SPARK-46503][PS][TESTS] Move `test_default_index` to `pyspark.pandas.tests.indexes.*` [spark]

zhengruifeng opened a new pull request, #44482: URL: https://github.com/apache/spark/pull/44482 ### What changes were proposed in this pull request? Move `test_default_index` to `pyspark.pandas.tests.indexes.*` ### Why are the changes needed? test code clean up

Re: [PR] [SPARK-40876][SQL][TESTS][FOLLOWUP] Skip the test inputs in `ParquetTypeWideningSuite` that violate the data type conversion rules under ANSI mode [spark]

cloud-fan commented on code in PR #44481: URL: https://github.com/apache/spark/pull/44481#discussion_r1436055343 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala: ## @@ -166,11 +166,17 @@ class ParquetTypeWideningSuite

Re: [PR] [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing [spark]

stefankandic commented on code in PR #44463: URL: https://github.com/apache/spark/pull/44463#discussion_r1436055004 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala: ## @@ -619,6 +616,29 @@ trait SparkDateTimeUtils { case NonFatal(_)

Re: [PR] Test Ivy 2.5.2 [spark]

LuciferYang commented on PR #44477: URL: https://github.com/apache/spark/pull/44477#issuecomment-1868917258 When master use ivy 2.5.2 and the test targets use ivy 2.5.1: ``` [info] : java.lang.RuntimeException: problem during retrieve of

Re: [PR] [SPARK-40876][SQL][TESTS][FOLLOWUP] Skip the test inputs in `ParquetTypeWideningSuite` that violate the data type conversion rules under ANSI mode [spark]

LuciferYang commented on code in PR #44481: URL: https://github.com/apache/spark/pull/44481#discussion_r1436052374 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala: ## @@ -166,11 +166,17 @@ class

Re: [PR] [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing [spark]

MaxGekk commented on code in PR #44463: URL: https://github.com/apache/spark/pull/44463#discussion_r1436048800 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala: ## @@ -619,6 +616,29 @@ trait SparkDateTimeUtils { case NonFatal(_) =>

[PR] [SPARK-40876][SQL][TESTS] Skip the test inputs in `ParquetTypeWideningSuite` that violate the data type conversion rules under ANSI mode [spark]

LuciferYang opened a new pull request, #44481: URL: https://github.com/apache/spark/pull/44481 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

Re: [PR] [SPARK-46500][PS][TESTS] Reorganize `FrameParityPivotTests` [spark]

zhengruifeng commented on PR #44478: URL: https://github.com/apache/spark/pull/44478#issuecomment-1868893762 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

  1   2   >