This is an automated email from the ASF dual-hosted git repository.
dependabot[bot] pushed a change to branch
dependabot/maven/io.netty-netty-transport-native-epoll-4.2.13.Final
in repository https://gitbox.apache.org/repos/asf/spark.git
discard 2d6c2a94b21c Bump io.netty:netty-transport-native-epoll
add a2dd75708c38 [SPARK-56772][SQL][TESTS-ONLY] Improve `Window` testing
add 0fc754675ed9 [SPARK-56519][PYTHON] Isolate communication part from
python udf worker
add 5ed9636e5b49 [SPARK-56546][SQL] Block-chunked segment-tree window
frame for non-invertible sliding aggregates
add bb72aef62854 [MINOR][INFRA] Ignore AGENTS.md and CONTRIBUTING.md in
determine_modules_for_files
add a342461677e2 [SPARK-56770][DOCS] Update PROJ version to 9.8.1 in the
documentation for geospatial types
add d905e7378b6d [SPARK-56482][SQL][FOLLOWUP] Simplify UnionExec codegen
and narrow partition-index gate
add 1f4650674a66 [SPARK-34591][ML] Add decision tree pruning as a parameter
add 2c9cd4fb33bd [SPARK-56768][PYTHON][INFRA] Share SBT compile artifact
across pyspark CI jobs
add 4be5711835fe Revert "[SPARK-34591][ML] Add decision tree pruning as a
parameter"
add 484bb5b2cd71 [SPARK-56312][PYTHON] Refactor SQL_COGROUPED_MAP_ARROW_UDF
add e4c6bc75ca24 [SPARK-56765][INFRA] Fix mypy attr-defined errors with
PyArrow 24+
add bae066b57c0f [SPARK-56682][GEO][CONNECT][PYTHON][SQL] Extend the
ST_AsBinary function to take optional endianness
add c6d289d5423a [SPARK-56771][SQL] Enable geospatial support by default
add b294189a1098 [SPARK-56794][SQL] Move `WindowSubstitution` Pipe Sql
resolution logic to `AstBuilder`
add ce3144518b60 [SPARK-56792][SQL] Support pan and zoom for SQL plan
visualization
add 6650e3f911f4 [SPARK-56755][SQL] Fix SHOW CREATE TABLE for v2 table
partitioned by bucket transform
add 8d5f3fd7200d [SPARK-56806][BUILD] Fix `jetty-proxy` artifact ID in
`core/pom.xml`'s `includeArtifactIds`
add 61672076ffd0 [SPARK-56692][INFRA] Check release date of upgrade of pom
dependency
add 6cc6e8ad4a2a [SPARK-56750] default path config
add 35f02b163018 [SPARK-40373][PS] Implement ps.show_versions
add 922d2ac4a77e [SPARK-40340][SPARK-40343][PYTHON] Implement Rolling.sem
and Expanding.sem in pandas API on Spark
add 7041d1eb72e0 [SPARK-40336][PS] Implement DataFrameGroupBy.cov
add dc1e903748e0 [SPARK-56797][BUILD] Upgrade `netty-tcnative` to
2.0.77.Final
add 79e16fe59350 Bump io.netty:netty-transport-native-epoll
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (2d6c2a94b21c)
\
N -- N -- N
refs/heads/dependabot/maven/io.netty-netty-transport-native-epoll-4.2.13.Final
(79e16fe59350)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../apache-spark-scala-lint-before-commit.mdc | 11 +
.github/workflows/build_and_test.yml | 92 ++-
AGENTS.md | 2 +
.../src/main/resources/error/error-conditions.json | 6 +
core/pom.xml | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 12 +-
dev/run-tests.py | 15 +-
dev/sparktestsupport/modules.py | 2 +
dev/sparktestsupport/utils.py | 106 ++-
docs/sql-ref-geospatial-types.md | 2 +-
pom.xml | 2 +-
.../reference/pyspark.pandas/general_functions.rst | 7 +
.../source/reference/pyspark.pandas/groupby.rst | 1 +
python/mypy.ini | 6 +
python/pyspark/errors/error-conditions.json | 6 +
python/pyspark/pandas/__init__.py | 1 +
python/pyspark/pandas/groupby.py | 210 +++++
python/pyspark/pandas/missing/groupby.py | 1 -
python/pyspark/pandas/namespace.py | 129 +++
.../test_parity_cov.py} | 6 +-
python/pyspark/pandas/tests/groupby/test_cov.py | 124 +++
python/pyspark/pandas/tests/test_namespace.py | 41 +
.../pandas/tests/window/test_expanding_adv.py | 4 +
.../pandas/tests/window/test_rolling_adv.py | 4 +
python/pyspark/pandas/window.py | 127 +++
python/pyspark/sql/connect/functions/builtin.py | 8 +-
python/pyspark/sql/conversion.py | 14 +
python/pyspark/sql/functions/builtin.py | 34 +-
python/pyspark/sql/tests/test_functions.py | 16 +-
python/pyspark/taskcontext.py | 6 +-
python/pyspark/worker.py | 254 +++---
python/pyspark/worker_message.py | 272 +++++++
python/pyspark/worker_util.py | 145 ++--
.../spark/sql/catalyst/parser/SqlBaseParser.g4 | 4 +
.../scala/org/apache/spark/sql/functions.scala | 18 +
.../org/apache/spark/sql/catalyst/util/Geo.java | 1 -
.../apache/spark/sql/catalyst/util/Geography.java | 5 -
.../apache/spark/sql/catalyst/util/Geometry.java | 5 -
.../apache/spark/sql/catalyst/util/STUtils.java | 21 +-
.../spark/sql/catalyst/analysis/Analyzer.scala | 16 +-
.../sql/catalyst/analysis/FunctionResolution.scala | 16 +
.../analysis/resolver/FunctionResolver.scala | 41 +-
.../analysis/resolver/FunctionResolverUtils.scala | 19 +-
.../resolver/HigherOrderFunctionResolver.scala | 2 +-
.../analysis/resolver/ResolutionValidator.scala | 6 +
.../sql/catalyst/analysis/resolver/Resolver.scala | 12 +
.../sql/catalyst/catalog/SessionCatalog.scala | 100 ++-
.../catalyst/expressions/st/stExpressions.scala | 39 +-
.../sql/catalyst/parser/AbstractSqlParser.scala | 13 +
.../spark/sql/catalyst/parser/AstBuilder.scala | 40 +-
.../plans/logical/basicLogicalOperators.scala | 3 +-
.../sql/connector/catalog/CatalogManager.scala | 111 ++-
.../spark/sql/connector/catalog/PathElement.scala | 141 ++++
.../spark/sql/errors/QueryExecutionErrors.scala | 18 +
.../org/apache/spark/sql/internal/SQLConf.scala | 82 +-
.../sql/catalyst/CatalystTypeConvertersSuite.scala | 28 +-
.../sql/catalyst/parser/PlanParserSuite.scala | 6 +-
.../sql/catalyst/util/GeographyExecutionSuite.java | 6 +-
.../sql/catalyst/util/GeometryExecutionSuite.java | 6 +-
.../spark/sql/catalyst/util/StUtilsSuite.java | 7 +-
.../benchmarks/WindowBenchmark-jdk21-results.txt | 172 ++++
.../benchmarks/WindowBenchmark-jdk25-results.txt | 172 ++++
sql/core/benchmarks/WindowBenchmark-results.txt | 172 ++++
.../sql/execution/ui/static/spark-sql-viz.css | 35 +-
.../spark/sql/execution/ui/static/spark-sql-viz.js | 139 +++-
.../spark/sql/execution/SparkSqlParser.scala | 10 +-
.../sql/execution/basicPhysicalOperators.scala | 159 ++--
.../LeafRunnableCommandResolver.scala} | 12 +-
.../sql/execution/command/SetPathCommand.scala | 81 +-
.../spark/sql/execution/command/functions.scala | 10 +-
.../datasources/v2/ShowCreateTableExec.scala | 5 +-
.../spark/sql/execution/ui/ExecutionPage.scala | 12 +
.../sql/execution/window/AggregateProcessor.scala | 24 +-
.../window/SegmentTreeWindowFunctionFrame.scala | 294 +++++++
.../execution/window/WindowEvaluatorFactory.scala | 8 +-
.../window/WindowEvaluatorFactoryBase.scala | 127 ++-
.../spark/sql/execution/window/WindowExec.scala | 11 +-
.../sql/execution/window/WindowSegmentTree.scala | 614 ++++++++++++++
.../sql/internal/BaseSessionStateBuilder.scala | 7 +-
.../sql-functions/sql-expression-schema.md | 6 +-
.../analyzer-results/named-window.sql.out | 619 ++++++++++++++
.../analyzer-results/pipe-operators.sql.out | 4 +-
.../analyzer-results/st-functions.sql.out | 183 ++++-
.../resources/sql-tests/inputs/named-window.sql | 207 +++++
.../resources/sql-tests/inputs/st-functions.sql | 27 +
.../sql-tests/results/named-window.sql.out | 822 +++++++++++++++++++
.../sql-tests/results/pipe-operators.sql.out | 4 +-
.../sql-tests/results/st-functions.sql.out | 208 ++++-
.../org/apache/spark/sql/STExpressionsSuite.scala | 52 +-
.../org/apache/spark/sql/STFunctionsSuite.scala | 35 +-
.../scala/org/apache/spark/sql/SetPathSuite.scala | 278 ++++++-
.../spark/sql/execution/SparkSqlParserSuite.scala | 5 +-
.../spark/sql/execution/UnionCodegenSuite.scala | 51 ++
.../sql/execution/benchmark/WindowBenchmark.scala | 372 +++++++++
.../command/v2/ShowCreateTableSuite.scala | 23 +-
.../window/SegmentTreeWindowFunctionSuite.scala | 896 +++++++++++++++++++++
.../window/SegmentTreeWindowMetricsSuite.scala | 213 +++++
.../window/SegmentTreeWindowTestHelper.scala | 176 ++++
.../window/WindowSegmentTreeAllowlistSuite.scala | 187 +++++
.../window/WindowSegmentTreeMemorySuite.scala | 351 ++++++++
.../window/WindowSegmentTreePropertySuite.scala | 475 +++++++++++
.../execution/window/WindowSegmentTreeSuite.scala | 521 ++++++++++++
.../spark/sql/hive/HiveSessionStateBuilder.scala | 2 +
103 files changed, 9611 insertions(+), 604 deletions(-)
create mode 100644 .cursor/rules/apache-spark-scala-lint-before-commit.mdc
copy python/pyspark/pandas/tests/connect/{frame/test_parity_asof.py =>
groupby/test_parity_cov.py} (91%)
create mode 100644 python/pyspark/pandas/tests/groupby/test_cov.py
create mode 100644 python/pyspark/worker_message.py
create mode 100644
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/PathElement.scala
create mode 100644 sql/core/benchmarks/WindowBenchmark-jdk21-results.txt
create mode 100644 sql/core/benchmarks/WindowBenchmark-jdk25-results.txt
create mode 100644 sql/core/benchmarks/WindowBenchmark-results.txt
copy
sql/core/src/main/scala/org/apache/spark/sql/execution/{datasources/LogicalRelationResolver.scala
=> command/LeafRunnableCommandResolver.scala} (71%)
create mode 100644
sql/core/src/main/scala/org/apache/spark/sql/execution/window/SegmentTreeWindowFunctionFrame.scala
create mode 100644
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowSegmentTree.scala
create mode 100644
sql/core/src/test/resources/sql-tests/analyzer-results/named-window.sql.out
create mode 100644
sql/core/src/test/resources/sql-tests/inputs/named-window.sql
create mode 100644
sql/core/src/test/resources/sql-tests/results/named-window.sql.out
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/WindowBenchmark.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/SegmentTreeWindowFunctionSuite.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/SegmentTreeWindowMetricsSuite.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/SegmentTreeWindowTestHelper.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeAllowlistSuite.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeMemorySuite.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreePropertySuite.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
