This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7a34b906f5 Regenerate Spark 4.3.0 release notes (#716)
7a34b906f5 is described below
commit 7a34b906f5ab16ee86e69f842d4d9a6a7cc1f468
Author: Jungtaek Lim <[email protected]>
AuthorDate: Wed Nov 4 11:42:18 2026 +0900
Regenerate Spark 4.3.0 release notes (#716)
Rewrites the Spark 4.3.0 release-notes post, reconstructed from the
`v4.2.0..v4.3.0` commit range and public Apache JIRA metadata. The
existing frontmatter is preserved; only the post body changes.
Highlights covered: stateful Real-Time Mode (streaming aggregation,
`dropDuplicates`, `transformWithState`) built on concurrent stage
scheduling and pipelined shuffles, SCD Type 2 in Auto CDC, faster
queries on small data, Spark Connect REST endpoints and operation IDs,
`JSON_TABLE` and `VARIANT` transformation functions, SQL `ASOF JOIN`,
and nanosecond-precision timestamps (preview).
The rendered `site/releases/spark-release-4-3-0.html` is regenerated
with `bundle exec jekyll build`, and contributor credits are cleaned up
(bot accounts and duplicate identities removed).
---
releases/_posts/2026-10-30-spark-release-4-3-0.md | 300 ++++++++++++++-
site/releases/spark-release-4-3-0.html | 422 +++++++++++++++++++++-
2 files changed, 716 insertions(+), 6 deletions(-)
diff --git a/releases/_posts/2026-10-30-spark-release-4-3-0.md
b/releases/_posts/2026-10-30-spark-release-4-3-0.md
index 2082ea71d9..b52af82388 100644
--- a/releases/_posts/2026-10-30-spark-release-4-3-0.md
+++ b/releases/_posts/2026-10-30-spark-release-4-3-0.md
@@ -11,8 +11,302 @@ meta:
_wpas_done_all: '1'
---
-Apache Spark 4.3.0 is a new feature release. It introduces new functionality
and improvements. We encourage users to try it and provide feedback.
+Apache Spark 4.3.0 is the fourth release of the Spark 4.x line. With
significant contributions from the open-source community, this release
addressed over 1,100 Jira tickets with contributions from more than 210
individuals.
-You can find the list of resolved issues and detailed changes in the [JIRA
release
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12356944).
+This release brings stateful processing to Structured Streaming Real-Time
Mode, adds history-preserving SCD Type 2 Auto CDC to Spark Declarative
Pipelines, and makes Spark faster to iterate with on small data. It also marks
the start of a quarterly feature-release cadence.
-We would like to acknowledge all community members for contributing patches
and features to this release.
+Real-Time Mode now supports stateful queries: streaming aggregation,
`dropDuplicates`, and `transformWithState` run with end-to-end latency on the
order of 100 milliseconds using the familiar Structured Streaming APIs.
Underneath, Spark gains native concurrent stage scheduling and pipelined
shuffles, so producer and consumer stages can run at the same time.
+
+Auto CDC adds Slowly Changing Dimension (SCD) Type 2 through Python, SQL, and
Spark Connect, so pipelines can track how records change over time instead of
overwriting the latest value.
+
+Work toward faster queries in local laptop mode reduces fixed execution
overhead on small data, with opt-in shuffle-free single-task execution for
small queries and opt-in Arrow-native Dataset caching.
+
+Spark Connect adds REST endpoints for inspecting sessions and operations,
operation IDs for end-to-end request attribution, and a more complete JDBC
driver. SQL adds `JSON_TABLE` and functions for transforming `VARIANT` values,
ANSI `UNNEST`, SQL `ASOF JOIN`, and a preview of nanosecond-precision
timestamps.
+
+To download Apache Spark 4.3.0, please visit the
[downloads](https://spark.apache.org/downloads.html) page. You can consult JIRA
for the [detailed
changes](https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%204.3.0).
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+### Highlights
+- **[[SPARK-58635]](https://issues.apache.org/jira/browse/SPARK-58635)**,
**[[SPARK-57228]](https://issues.apache.org/jira/browse/SPARK-57228)**,
**[[SPARK-58508]](https://issues.apache.org/jira/browse/SPARK-58508)** Stateful
Real-Time Mode — streaming aggregation, `dropDuplicates`, and
`transformWithState` with end-to-end latency on the order of 100 milliseconds,
built on native concurrent stage scheduling and pipelined shuffles
+- **[[SPARK-58321]](https://issues.apache.org/jira/browse/SPARK-58321)**,
**[[SPARK-58247]](https://issues.apache.org/jira/browse/SPARK-58247)**,
**[[SPARK-58261]](https://issues.apache.org/jira/browse/SPARK-58261)** SCD Type
2 in Auto CDC — preserve the history of every record through Python, SQL, and
Spark Connect
+- **[[SPARK-57851]](https://issues.apache.org/jira/browse/SPARK-57851)**,
**[[SPARK-57268]](https://issues.apache.org/jira/browse/SPARK-57268)** Faster
queries on small data — opt-in shuffle-free single-task execution for small
queries and Arrow-native Dataset caching for faster iteration on small data
+- **[[SPARK-57941]](https://issues.apache.org/jira/browse/SPARK-57941)**,
**[[SPARK-58547]](https://issues.apache.org/jira/browse/SPARK-58547)** Spark
Connect REST endpoints to list and inspect sessions and operations, with
operation IDs for end-to-end request attribution
+- **[[SPARK-58366]](https://issues.apache.org/jira/browse/SPARK-58366)**,
**[[SPARK-57804]](https://issues.apache.org/jira/browse/SPARK-57804)**
`JSON_TABLE` and `VARIANT` transformation functions (`variant_set`,
`variant_insert`, `variant_delete`, ...) to process nested documents natively
in SQL
+- **[[SPARK-58092]](https://issues.apache.org/jira/browse/SPARK-58092)** SQL
`ASOF JOIN` for point-in-time matching of events to the information available
at the time
+- **[[SPARK-56876]](https://issues.apache.org/jira/browse/SPARK-56876)**
Nanosecond-precision timestamps `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`
(preview)
+
+---
+
+### SQL Foundation
+- [[SPARK-58366]](https://issues.apache.org/jira/browse/SPARK-58366) Support
the `JSON_TABLE` table-valued function
+- [[SPARK-58419]](https://issues.apache.org/jira/browse/SPARK-58419) Support
ANSI SQL `UNNEST` in the `FROM` clause
+- [[SPARK-57421]](https://issues.apache.org/jira/browse/SPARK-57421) Support
`@`-syntax version time travel on table names
+- [[SPARK-58012]](https://issues.apache.org/jira/browse/SPARK-58012) Support
`@`-syntax timestamp time travel on table names
+- [[SPARK-52719]](https://issues.apache.org/jira/browse/SPARK-52719) Support
scalar UDFs in table-valued function arguments
+- [[SPARK-54831]](https://issues.apache.org/jira/browse/SPARK-54831) Support
`COMMENT ON COLUMN` for table columns
+- [[SPARK-58065]](https://issues.apache.org/jira/browse/SPARK-58065) Support
column lists for `INSERT ... REPLACE WHERE`
+- [[SPARK-57681]](https://issues.apache.org/jira/browse/SPARK-57681) Support
dynamic table options for `UPDATE`
+- [[SPARK-58007]](https://issues.apache.org/jira/browse/SPARK-58007) Support
dynamic table options for `MERGE`
+- [[SPARK-58008]](https://issues.apache.org/jira/browse/SPARK-58008) Support
dynamic table options for `DELETE`
+- [[SPARK-57993]](https://issues.apache.org/jira/browse/SPARK-57993) Support
an advisory partition size in the `REBALANCE` hint
+- [[SPARK-53840]](https://issues.apache.org/jira/browse/SPARK-53840) Add `AS
JSON` output for `SHOW TABLES` and `SHOW TABLE EXTENDED`
+- [[SPARK-54710]](https://issues.apache.org/jira/browse/SPARK-54710) Add `AS
JSON` output for `SHOW PARTITIONS`
+- [[SPARK-58260]](https://issues.apache.org/jira/browse/SPARK-58260) Add a
config to render Hive DDL in `SHOW CREATE TABLE` for Hive tables
+- [[SPARK-58052]](https://issues.apache.org/jira/browse/SPARK-58052) Throw
overflow errors for `ceil` and `floor` in ANSI mode
+- [[SPARK-57775]](https://issues.apache.org/jira/browse/SPARK-57775) Return
one row for `GROUP BY GROUPING SETS (())` over empty input
+- [[SPARK-57194]](https://issues.apache.org/jira/browse/SPARK-57194) Add the
`preOperatorOptimizationRules` extension point to the Optimizer
+- [[SPARK-56147]](https://issues.apache.org/jira/browse/SPARK-56147) Use a
parser-based SQL statement splitter in the `spark-sql` CLI
+- [[SPARK-57518]](https://issues.apache.org/jira/browse/SPARK-57518) Make
Thrift server JDBC metadata operations DataSource V2 catalog-aware (opt-in via
`spark.sql.thriftServer.catalogMetadata.enabled`)
+- **SQL support for ASOF join**
([[SPARK-58092]](https://issues.apache.org/jira/browse/SPARK-58092))
+ - [[SPARK-58122]](https://issues.apache.org/jira/browse/SPARK-58122) Add SQL
`ASOF JOIN ... MATCH_CONDITION` syntax, enabled with
`spark.sql.join.asofJoin.enabled` (off by default)
+ - [[SPARK-58124]](https://issues.apache.org/jira/browse/SPARK-58124)
Sort-merge physical execution for SQL `ASOF JOIN`
+- **Add BIN BY relation operator for aligning range-typed rows to fixed bin
boundaries**
([[SPARK-57133]](https://issues.apache.org/jira/browse/SPARK-57133))
+ - [[SPARK-57440]](https://issues.apache.org/jira/browse/SPARK-57440) `BIN
BY` is available behind `spark.sql.binByRelationOperator.enabled` (off by
default)
+ - [[SPARK-58003]](https://issues.apache.org/jira/browse/SPARK-58003)
Physical execution for `BIN BY`
+ - [[SPARK-58064]](https://issues.apache.org/jira/browse/SPARK-58064) Filter
pushdown for `BIN BY`
+ - [[SPARK-58138]](https://issues.apache.org/jira/browse/SPARK-58138)
Preserve child output partitioning through `BIN BY`
+
+#### Built-in Functions
+- [[SPARK-56820]](https://issues.apache.org/jira/browse/SPARK-56820) Add the
`counter_diff` window function for converting cumulative counters to deltas
+- [[SPARK-57253]](https://issues.apache.org/jira/browse/SPARK-57253) Add the
`jaro_winkler_similarity` function
+- [[SPARK-58399]](https://issues.apache.org/jira/browse/SPARK-58399) Add the
`collect_union` aggregate function
+- [[SPARK-58295]](https://issues.apache.org/jira/browse/SPARK-58295) Add the
`to_base32` and `from_base32` functions
+
+#### VARIANT manipulation
+- [[SPARK-57804]](https://issues.apache.org/jira/browse/SPARK-57804) Add the
`variant_set` expression, plus `try_variant_set`
([[SPARK-58221]](https://issues.apache.org/jira/browse/SPARK-58221))
+- [[SPARK-57641]](https://issues.apache.org/jira/browse/SPARK-57641) Add the
`variant_insert` expression, plus `try_variant_insert`
([[SPARK-58035]](https://issues.apache.org/jira/browse/SPARK-58035))
+- [[SPARK-57417]](https://issues.apache.org/jira/browse/SPARK-57417) Add the
`variant_delete` expression
+- [[SPARK-58061]](https://issues.apache.org/jira/browse/SPARK-58061) Add the
`variant_array_append` expression, plus `try_variant_array_append`
([[SPARK-58222]](https://issues.apache.org/jira/browse/SPARK-58222))
+- [[SPARK-58062]](https://issues.apache.org/jira/browse/SPARK-58062) Add the
`variant_strip_nulls` expression
+
+#### TIME type
+- [[SPARK-57551]](https://issues.apache.org/jira/browse/SPARK-57551) Extend
`TIME` precision to nanoseconds (`TIME(p)` with `p` up to 9)
+- [[SPARK-57558]](https://issues.apache.org/jira/browse/SPARK-57558) Add the
`LOCALTIME` datetime value function
+- [[SPARK-57618]](https://issues.apache.org/jira/browse/SPARK-57618) Support
casting between `TIME(p)` and `TIMESTAMP_NTZ(q)` / `TIMESTAMP_LTZ(q)`
([[SPARK-57660]](https://issues.apache.org/jira/browse/SPARK-57660))
+- [[SPARK-57572]](https://issues.apache.org/jira/browse/SPARK-57572) Infer
`TIME` during CSV and JSON schema inference, and in XML
([[SPARK-57634]](https://issues.apache.org/jira/browse/SPARK-57634))
+- [[SPARK-57555]](https://issues.apache.org/jira/browse/SPARK-57555) Support
`TIME` in the built-in JDBC data source and the Protobuf connector
([[SPARK-57573]](https://issues.apache.org/jira/browse/SPARK-57573))
+- [[SPARK-57571]](https://issues.apache.org/jira/browse/SPARK-57571) `TIME`
predicate pushdown to ORC, and Parquet/ORC aggregate pushdown
([[SPARK-57568]](https://issues.apache.org/jira/browse/SPARK-57568))
+- [[SPARK-57557]](https://issues.apache.org/jira/browse/SPARK-57557) Support
`TIME` in quantile aggregates, `approx_top_k`
([[SPARK-57848]](https://issues.apache.org/jira/browse/SPARK-57848)), and
`to_char`/`to_varchar`
([[SPARK-57575]](https://issues.apache.org/jira/browse/SPARK-57575))
+
+#### Nanosecond-precision timestamps (preview)
+- [[SPARK-56876]](https://issues.apache.org/jira/browse/SPARK-56876) Add the
`TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` types with precision `p` from 7
through 9, as a preview enabled with `spark.sql.timestampNanosTypes.enabled`
(off by default)
+- [[SPARK-56965]](https://issues.apache.org/jira/browse/SPARK-56965) SQL
parser support for `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)`
+- [[SPARK-57293]](https://issues.apache.org/jira/browse/SPARK-57293) Casts
between nanosecond- and microsecond-precision timestamps, and to and from
`DATE`, `TIME`, and strings
+- [[SPARK-57454]](https://issues.apache.org/jira/browse/SPARK-57454) Type
coercion and widening rules for nanosecond-precision timestamps
+- [[SPARK-57526]](https://issues.apache.org/jira/browse/SPARK-57526) Add the
`timestamp_nanos` and `unix_nanos`
([[SPARK-57527]](https://issues.apache.org/jira/browse/SPARK-57527)) functions,
and the precision argument in `current_timestamp(p)`
([[SPARK-57837]](https://issues.apache.org/jira/browse/SPARK-57837))
+- [[SPARK-57816]](https://issues.apache.org/jira/browse/SPARK-57816) Datetime
functions (`date_format`, `date_trunc`, `EXTRACT`, `hour`/`minute`/`second`,
`unix_*`) over nanosecond-precision timestamps
+- [[SPARK-57501]](https://issues.apache.org/jira/browse/SPARK-57501) ANSI
interval arithmetic and timestamp subtraction for nanosecond-precision
timestamps
+- [[SPARK-57102]](https://issues.apache.org/jira/browse/SPARK-57102) Parquet
support, including the vectorized reader
([[SPARK-57828]](https://issues.apache.org/jira/browse/SPARK-57828)) and
predicate pushdown
([[SPARK-57822]](https://issues.apache.org/jira/browse/SPARK-57822))
+- [[SPARK-57455]](https://issues.apache.org/jira/browse/SPARK-57455) ORC,
JSON, CSV, XML, Avro, and JDBC data source support for nanosecond-precision
timestamps
+- [[SPARK-57160]](https://issues.apache.org/jira/browse/SPARK-57160) Arrow and
Spark Connect support for nanosecond-precision timestamp types and literals
+
+#### Notable fixes
+- [[SPARK-60304]](https://issues.apache.org/jira/browse/SPARK-60304) Fix
subplan merging losing a shared lower filter or a reused filter attribute
([[SPARK-60084]](https://issues.apache.org/jira/browse/SPARK-60084)), which
could produce wrong results
+- [[SPARK-57956]](https://issues.apache.org/jira/browse/SPARK-57956) Fix AQE
incorrectly eliminating a global `LIMIT` over a non-materialized query stage
+- [[SPARK-60508]](https://issues.apache.org/jira/browse/SPARK-60508) Fix wrong
results from `semanticEquals` of higher-order functions capturing lambda
variables
+- [[SPARK-58384]](https://issues.apache.org/jira/browse/SPARK-58384) Preserve
null semantics in `OptimizeJoinCondition`
+- [[SPARK-58481]](https://issues.apache.org/jira/browse/SPARK-58481) Fix `IN
(subquery)` returning wrong results when the subquery output contains NULLs
+- [[SPARK-59434]](https://issues.apache.org/jira/browse/SPARK-59434) Do not
push non-deterministic predicates into CTE definitions
+- [[SPARK-58959]](https://issues.apache.org/jira/browse/SPARK-58959) Fix
`PIVOT` with duplicate values corrupting the aggregation buffer
+- [[SPARK-58816]](https://issues.apache.org/jira/browse/SPARK-58816) Fix
`INSERT` with a column list to resolve structs inside arrays and maps
positionally
+- [[SPARK-57727]](https://issues.apache.org/jira/browse/SPARK-57727) Fix
incorrect query results from constraint inference substituting attributes with
non-binary-stable collations
+- [[SPARK-58204]](https://issues.apache.org/jira/browse/SPARK-58204) Fix
expressions that held unprotected mutable state (regex, `translate`,
`format_number`, JSON/CSV/XML
([[SPARK-58205]](https://issues.apache.org/jira/browse/SPARK-58205)),
`ApplyFunctionExpression`
([[SPARK-58578]](https://issues.apache.org/jira/browse/SPARK-58578)), and Hive
`GenericUDF`
([[SPARK-58792]](https://issues.apache.org/jira/browse/SPARK-58792))), which
could corrupt results under concurrent evaluation
+
+### Query API
+- [[SPARK-55886]](https://issues.apache.org/jira/browse/SPARK-55886) Add
`DataFrame.zip` for merging column-projected DataFrames
+- [[SPARK-57284]](https://issues.apache.org/jira/browse/SPARK-57284) Add Scala
and Python bindings for vector functions
+- [[SPARK-46776]](https://issues.apache.org/jira/browse/SPARK-46776) Support
`pa.ChunkedArray` columns in `createDataFrame` from pandas
+- [[SPARK-57190]](https://issues.apache.org/jira/browse/SPARK-57190) Fix the
API inconsistency of the 4-argument `regexp_replace`
+- [[SPARK-57860]](https://issues.apache.org/jira/browse/SPARK-57860) Make the
storage level of MLlib intermediate datasets configurable for KMeans and ALS
+- [[SPARK-34591]](https://issues.apache.org/jira/browse/SPARK-34591) Add
decision tree pruning as a parameter
+
+#### Pandas API on Spark
+- [[SPARK-40336]](https://issues.apache.org/jira/browse/SPARK-40336) Implement
`DataFrameGroupBy.cov`
+- [[SPARK-40337]](https://issues.apache.org/jira/browse/SPARK-40337) Implement
`SeriesGroupBy.describe`
+- [[SPARK-40340]](https://issues.apache.org/jira/browse/SPARK-40340) Implement
`Rolling.sem` and `Expanding.sem`
+- [[SPARK-40341]](https://issues.apache.org/jira/browse/SPARK-40341) Implement
`Rolling.median` and `Expanding.median`
+- [[SPARK-40373]](https://issues.apache.org/jira/browse/SPARK-40373) Implement
`ps.show_versions`
+- [[SPARK-46158]](https://issues.apache.org/jira/browse/SPARK-46158) Support
axis columns in `DataFrame.xs`
+
+### Connectors
+
+#### Data Source V2 framework
+- [[SPARK-57644]](https://issues.apache.org/jira/browse/SPARK-57644) Support
generated column values on V2 table writes
+- [[SPARK-55347]](https://issues.apache.org/jira/browse/SPARK-55347) Pass
generated column expressions to DSv2 connectors
+- [[SPARK-58523]](https://issues.apache.org/jira/browse/SPARK-58523) Add a
Catalyst runtime filtering interface for DSv2 scans
+- [[SPARK-55617]](https://issues.apache.org/jira/browse/SPARK-55617) Push down
`variant_get` filters to DSv2 sources
+- [[SPARK-56660]](https://issues.apache.org/jira/browse/SPARK-56660) Decompose
struct equality into field predicates for data source pushdown
+
+#### Reading from archives (opt-in)
+- [[SPARK-57135]](https://issues.apache.org/jira/browse/SPARK-57135) Read CSV
files inside tar archives, with schema inference
([[SPARK-57321]](https://issues.apache.org/jira/browse/SPARK-57321))
+- [[SPARK-57419]](https://issues.apache.org/jira/browse/SPARK-57419) Read and
infer JSON schema from tar archives
+- [[SPARK-57479]](https://issues.apache.org/jira/browse/SPARK-57479) Read and
infer XML schema from tar archives
+- [[SPARK-57478]](https://issues.apache.org/jira/browse/SPARK-57478) Read text
files from tar archives
+- [[SPARK-57481]](https://issues.apache.org/jira/browse/SPARK-57481) Read and
infer Avro schema from tar archives
+- [[SPARK-57590]](https://issues.apache.org/jira/browse/SPARK-57590) Read and
infer Parquet schema from archives
+- [[SPARK-57591]](https://issues.apache.org/jira/browse/SPARK-57591) Read and
infer ORC schema from archives
+- [[SPARK-57705]](https://issues.apache.org/jira/browse/SPARK-57705) Read CSV,
JSON, text, and XML files from zip archives
+- [[SPARK-58246]](https://issues.apache.org/jira/browse/SPARK-58246) Read
files from 7z archives
+- [[SPARK-58382]](https://issues.apache.org/jira/browse/SPARK-58382) Support
archives in the `binaryFile` source via the `wholeFile` option
+- [[SPARK-58609]](https://issues.apache.org/jira/browse/SPARK-58609) Add the
`archivePathFilter` option to select entries inside archives
+
+#### File Sources
+- [[SPARK-57354]](https://issues.apache.org/jira/browse/SPARK-57354) Add the
`ignoredPathSegmentRegex` data source option and config
+- [[SPARK-57678]](https://issues.apache.org/jira/browse/SPARK-57678) Add the
`explodeEmbeddedArray` JSON option for streaming scans of embedded arrays
+- [[SPARK-57802]](https://issues.apache.org/jira/browse/SPARK-57802) Make XML
schema inference incremental, aligning with CSV
+
+#### JDBC, Hive, and Kafka
+- [[SPARK-52825]](https://issues.apache.org/jira/browse/SPARK-52825) Register
existing JDBC dialects for additional URL prefixes
+- [[SPARK-59330]](https://issues.apache.org/jira/browse/SPARK-59330) Add an
option to defer Hive `InputFormat` and `OutputFormat` class initialization
+- [[SPARK-59328]](https://issues.apache.org/jira/browse/SPARK-59328) Add an
option to reject specified Kafka options in the Kafka data source
+
+#### Notable fixes
+- [[SPARK-58112]](https://issues.apache.org/jira/browse/SPARK-58112) Keep
nondeterministic filters out of DSv2 Catalyst and V2 filter pushdown
([[SPARK-58207]](https://issues.apache.org/jira/browse/SPARK-58207)), so Spark
evaluates them after the scan
+- [[SPARK-59108]](https://issues.apache.org/jira/browse/SPARK-59108) Fix Avro
positional field matching under column pruning
+
+### UDF (User Defined Functions)
+- [[SPARK-55206]](https://issues.apache.org/jira/browse/SPARK-55206)
Experimental transpilation of simple Python UDFs to Catalyst expressions (e.g.
`len()` to `length`
([[SPARK-55214]](https://issues.apache.org/jira/browse/SPARK-55214))), enabled
with `spark.sql.experimental.optimizer.transpilePyUDFs`
+- [[SPARK-56413]](https://issues.apache.org/jira/browse/SPARK-56413) Add a
gRPC UDF execution protocol and logical/physical planning nodes
([[SPARK-56661]](https://issues.apache.org/jira/browse/SPARK-56661)) as
groundwork for language-agnostic UDFs
+- [[SPARK-57271]](https://issues.apache.org/jira/browse/SPARK-57271)
Optionally print local variables in Python UDF tracebacks
+- [[SPARK-57593]](https://issues.apache.org/jira/browse/SPARK-57593) Bound
pickled Python UDF input batches by size, with instrumentation
+- [[SPARK-58601]](https://issues.apache.org/jira/browse/SPARK-58601) Require
`mapInPandas` functions to return a strict iterator
+
+### Streaming
+- [[SPARK-58724]](https://issues.apache.org/jira/browse/SPARK-58724)
Incremental state cleanup for streaming `dropDuplicates`
+
+#### Real-Time Mode
+- [[SPARK-58508]](https://issues.apache.org/jira/browse/SPARK-58508) Support
stateful queries in Real-Time Mode, including `dropDuplicates`, by running them
on pipelined shuffles
+- [[SPARK-58635]](https://issues.apache.org/jira/browse/SPARK-58635) Support
streaming aggregation in Real-Time Mode
+- [[SPARK-57228]](https://issues.apache.org/jira/browse/SPARK-57228) Support
Scala/Java `transformWithState` in Real-Time Mode
+- [[SPARK-57281]](https://issues.apache.org/jira/browse/SPARK-57281) Remove
the `@Experimental` annotation from Real-Time Mode
+- [[SPARK-58131]](https://issues.apache.org/jira/browse/SPARK-58131) Support
async progress tracking in Real-Time Mode
+- [[SPARK-58185]](https://issues.apache.org/jira/browse/SPARK-58185) Add
`PipelinedShuffleDependency`, letting consumers read while producers are still
running, and route each shuffle to a shuffle manager by dependency type
+- [[SPARK-58263]](https://issues.apache.org/jira/browse/SPARK-58263) Natively
schedule pipelined stage groups concurrently in the DAGScheduler, with
group-atomic failure and fail-fast admission when task slots are insufficient
([[SPARK-58398]](https://issues.apache.org/jira/browse/SPARK-58398)), decoupled
from the `MapOutputTracker`
([[SPARK-58454]](https://issues.apache.org/jira/browse/SPARK-58454))
+- [[SPARK-57141]](https://issues.apache.org/jira/browse/SPARK-57141) Add
`StreamingShuffleManager`, the built-in pipelined shuffle implementation, with
a Netty-based wire protocol
([[SPARK-56674]](https://issues.apache.org/jira/browse/SPARK-56674))
+
+#### Notable fixes
+- [[SPARK-59315]](https://issues.apache.org/jira/browse/SPARK-59315) Fix the
correctness of `dropDuplicates` with a watermark after query restart
+
+### Spark Connect Framework
+- [[SPARK-57941]](https://issues.apache.org/jira/browse/SPARK-57941) Expose
Spark Connect sessions and operations via a REST API
+- [[SPARK-57021]](https://issues.apache.org/jira/browse/SPARK-57021) Add a
`SQLContext` wrapper for Spark Connect
+- [[SPARK-57787]](https://issues.apache.org/jira/browse/SPARK-57787) Reuse a
persistent local Spark Connect server across runs for faster local PySpark
startup (opt-in via `SPARK_LOCAL_CONNECT_REUSE=1`)
+- [[SPARK-58159]](https://issues.apache.org/jira/browse/SPARK-58159) Support
the `with` statement for Spark Connect sessions
+- [[SPARK-58547]](https://issues.apache.org/jira/browse/SPARK-58547) Expose
operation IDs for end-to-end request attribution
+- [[SPARK-57247]](https://issues.apache.org/jira/browse/SPARK-57247) Support
`DataFrame.zip` in Spark Connect
+- [[SPARK-58251]](https://issues.apache.org/jira/browse/SPARK-58251) Size
estimates for nearly all Spark ML models (feature transformers, tree,
clustering, ALS, LDA, FPGrowth, and tuning models), with fixes for pipeline
overcounting
([[SPARK-58281]](https://issues.apache.org/jira/browse/SPARK-58281)), for
better Spark Connect ML model cache management
+- [[SPARK-44462]](https://issues.apache.org/jira/browse/SPARK-44462) Fix the
session passed to `foreachBatch` in Spark Connect
+
+#### JDBC driver
+- [[SPARK-57380]](https://issues.apache.org/jira/browse/SPARK-57380) Support
complex types (`ARRAY`, `MAP`, `STRUCT`) in the Connect JDBC driver
+- [[SPARK-57627]](https://issues.apache.org/jira/browse/SPARK-57627) Support
primary key, foreign key, and SQL type metadata in `DatabaseMetaData`
+- [[SPARK-57274]](https://issues.apache.org/jira/browse/SPARK-57274) Support
fetch/type accessors and `getMoreResults` for `SparkConnectStatement`
+- [[SPARK-57933]](https://issues.apache.org/jira/browse/SPARK-57933) Map Spark
Connect client failures to `SQLException`, and connection errors by SQLSTATE
class `08` ([[SPARK-58619]](https://issues.apache.org/jira/browse/SPARK-58619))
+
+### Performance and stability
+- **Add bulk read+convert paths for type-converting Parquet vector updaters**
([[SPARK-56800]](https://issues.apache.org/jira/browse/SPARK-56800))
+ - [[SPARK-56791]](https://issues.apache.org/jira/browse/SPARK-56791) Bulk
read-and-convert paths in the vectorized Parquet reader for widening (INT32 to
Long/Double, FLOAT to Double), INT64 `DECIMAL` narrowing, and `DATE` to
`TIMESTAMP_NTZ` conversions
+- [[SPARK-56893]](https://issues.apache.org/jira/browse/SPARK-56893) Faster
vectorized Parquet decoding: dictionary decoding fast paths, bulk
`DELTA_BINARY_PACKED`
([[SPARK-56892]](https://issues.apache.org/jira/browse/SPARK-56892)) and
timestamp/date
([[SPARK-56896]](https://issues.apache.org/jira/browse/SPARK-56896)) reads, and
lower-allocation RLE decoding
([[SPARK-56895]](https://issues.apache.org/jira/browse/SPARK-56895))
+- [[SPARK-56894]](https://issues.apache.org/jira/browse/SPARK-56894) Add a
vectorized Parquet `BYTE_STREAM_SPLIT` reader
+
+#### Faster queries on small data
+- [[SPARK-57851]](https://issues.apache.org/jira/browse/SPARK-57851) Opt-in
shuffle-free single-task execution for eligible small, single-partition query
fragments (`spark.sql.optimizer.singleTaskExecution.enabled`), covering scans,
aggregations, distinct, windows, sorts, and limits
+- [[SPARK-57268]](https://issues.apache.org/jira/browse/SPARK-57268) Opt-in
Arrow-native in-memory Dataset caching with compression and column statistics
(`spark.sql.cache.serializer=org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer`)
+
+#### Query Optimizer and Execution
+- [[SPARK-58210]](https://issues.apache.org/jira/browse/SPARK-58210) Enable
`ReplaceHashWithSortAgg` and combining adjacent aggregations
([[SPARK-43317]](https://issues.apache.org/jira/browse/SPARK-43317)) by default
+- [[SPARK-56805]](https://issues.apache.org/jira/browse/SPARK-56805) Enable
whole-stage codegen for `Union` by default
+- [[SPARK-37019]](https://issues.apache.org/jira/browse/SPARK-37019) Add
codegen support for array higher-order functions
+- [[SPARK-40259]](https://issues.apache.org/jira/browse/SPARK-40259) Merge
equivalent DataSource V2 scans in subplan merging
+- [[SPARK-56887]](https://issues.apache.org/jira/browse/SPARK-56887) Add a
dedicated sort-merge physical operator for AS-OF joins
+- [[SPARK-56903]](https://issues.apache.org/jira/browse/SPARK-56903) Opt-in
(`spark.sql.shuffle.spreadNullJoinKeys.enabled`) spreading of NULL outer join
keys, and left anti join keys
([[SPARK-57282]](https://issues.apache.org/jira/browse/SPARK-57282)), across
shuffle partitions to avoid skew
+- [[SPARK-57370]](https://issues.apache.org/jira/browse/SPARK-57370) Add a JDK
(`javac`) compiler backend for whole-stage codegen
+- [[SPARK-58066]](https://issues.apache.org/jira/browse/SPARK-58066) Hoist
streamed-side predicates in outer, anti, and existence joins
+- [[SPARK-58084]](https://issues.apache.org/jira/browse/SPARK-58084) Support
converting sort merge join to shuffled hash join in the AQE physical plan
+- [[SPARK-58099]](https://issues.apache.org/jira/browse/SPARK-58099) Remove
the redundant local sort left below the shuffle added by skew join optimization
+- [[SPARK-58233]](https://issues.apache.org/jira/browse/SPARK-58233) Push down
local sorts to reduce redundant sorts within a stage
+- [[SPARK-58405]](https://issues.apache.org/jira/browse/SPARK-58405) Pull up
project aliases through windows to eliminate redundant shuffles and sorts
+- [[SPARK-54593]](https://issues.apache.org/jira/browse/SPARK-54593) Enable
dynamic partition pruning for materialized filtering sides
+- [[SPARK-58265]](https://issues.apache.org/jira/browse/SPARK-58265) Reuse
projected broadcast values for dynamic partition pruning
+- [[SPARK-58272]](https://issues.apache.org/jira/browse/SPARK-58272) Enable
runtime Bloom filters for materialized cached inputs
+- [[SPARK-58089]](https://issues.apache.org/jira/browse/SPARK-58089) Push
variant extractions through Aggregate, Sort, and Join
+- [[SPARK-57022]](https://issues.apache.org/jira/browse/SPARK-57022) Nested
column pruning for `transform` over arrays of structs and through
array-returning functions
([[SPARK-57176]](https://issues.apache.org/jira/browse/SPARK-57176))
+- [[SPARK-57043]](https://issues.apache.org/jira/browse/SPARK-57043) Collapse
grouped `SUM(COUNT(1))` rollups
+- [[SPARK-57220]](https://issues.apache.org/jira/browse/SPARK-57220) Extend
the segment-tree window frame to shrinking frames and to `first`/`last`
([[SPARK-57424]](https://issues.apache.org/jira/browse/SPARK-57424))
+- [[SPARK-57937]](https://issues.apache.org/jira/browse/SPARK-57937) Improve
decimal division performance by dividing at the result scale
+- [[SPARK-56769]](https://issues.apache.org/jira/browse/SPARK-56769) Add a
fast path for `date_trunc` at WEEK/MONTH/QUARTER/YEAR granularity
+- [[SPARK-57905]](https://issues.apache.org/jira/browse/SPARK-57905) Reduce
the size of generated whole-stage codegen, e.g. for null-guarded row writes and
`regexp_extract`
([[SPARK-57258]](https://issues.apache.org/jira/browse/SPARK-57258))
+- [[SPARK-57688]](https://issues.apache.org/jira/browse/SPARK-57688) Add
`spark.sql.execution.bypassPartialAggregation` to skip partial aggregation
+
+#### Python Performance
+- [[SPARK-56642]](https://issues.apache.org/jira/browse/SPARK-56642) Add
pipelined JVM-Python UDF data transfer, with an optional Unix Domain Socket mode
+- [[SPARK-58230]](https://issues.apache.org/jira/browse/SPARK-58230) Batch
pandas-on-Spark column projections
+- [[SPARK-58249]](https://issues.apache.org/jira/browse/SPARK-58249) Use
native Spark functions for NumPy math ufuncs, bitwise shifts
([[SPARK-58464]](https://issues.apache.org/jira/browse/SPARK-58464)), `rint`
([[SPARK-58469]](https://issues.apache.org/jira/browse/SPARK-58469)), `trunc`
([[SPARK-58470]](https://issues.apache.org/jira/browse/SPARK-58470)), and
inverse hyperbolic functions
([[SPARK-58234]](https://issues.apache.org/jira/browse/SPARK-58234)) in pandas
API on Spark
+
+#### Stability
+- [[SPARK-57723]](https://issues.apache.org/jira/browse/SPARK-57723) Enable
`spark.rdd.compress` by default, as in PySpark
+- [[SPARK-58107]](https://issues.apache.org/jira/browse/SPARK-58107) Detect
and repair non-deterministic local checkpoints via RDD-block content checksums
+- [[SPARK-57491]](https://issues.apache.org/jira/browse/SPARK-57491) Prevent
and detect stale push-based shuffle data from duplicate map task attempts
+- [[SPARK-58848]](https://issues.apache.org/jira/browse/SPARK-58848) Prevent
unbounded shuffle reader connection waits
+- [[SPARK-58004]](https://issues.apache.org/jira/browse/SPARK-58004) Add a
hard fan-in bound to AQE shuffle partition coalescing
+
+#### Security
+- [[SPARK-58054]](https://issues.apache.org/jira/browse/SPARK-58054) Enable
`spark.ui.contentSecurityPolicy.enabled` by default
+- [[SPARK-57904]](https://issues.apache.org/jira/browse/SPARK-57904) Change
the default of `spark.ui.xXssProtection` to `0`
+- [[SPARK-59319]](https://issues.apache.org/jira/browse/SPARK-59319) Add an
opt-in restricted SQL execution mode
+- [[SPARK-57448]](https://issues.apache.org/jira/browse/SPARK-57448) Support
`spark.sql.reflect.allowList` to restrict the methods invoked by
`reflect`/`java_method`
+- [[SPARK-57441]](https://issues.apache.org/jira/browse/SPARK-57441) Disallow
setting JVM system properties in Spark Thrift Server sessions, including via
`set:hiveconf`
([[SPARK-57480]](https://issues.apache.org/jira/browse/SPARK-57480))
+- [[SPARK-54293]](https://issues.apache.org/jira/browse/SPARK-54293) Make the
`ThriftHttpCLIService` SNI host check configurable
+- [[SPARK-58370]](https://issues.apache.org/jira/browse/SPARK-58370) Check
table write privileges when the write target is also read in the same statement
+- [[SPARK-59329]](https://issues.apache.org/jira/browse/SPARK-59329) Add an
allowlist of URI schemes for the Avro `schemaUrl` option
+- [[SPARK-57453]](https://issues.apache.org/jira/browse/SPARK-57453) Support
`spark.master.rest.allowedAppResourcePatterns` to restrict application
resources submitted through the Master REST server, and limit its request body
size ([[SPARK-58049]](https://issues.apache.org/jira/browse/SPARK-58049))
+- [[SPARK-58056]](https://issues.apache.org/jira/browse/SPARK-58056) Make
`NettyStreamManager` serve only files under the registered directory
+- [[SPARK-58153]](https://issues.apache.org/jira/browse/SPARK-58153) Reject
JAR entries escaping the extraction directory in `RPackageUtils`
+- [[SPARK-59508]](https://issues.apache.org/jira/browse/SPARK-59508) Harden
Python client safety with a restricted unpickle serializer
+- [[SPARK-59320]](https://issues.apache.org/jira/browse/SPARK-59320)
Optionally require an expiration claim in `JWSFilter`
+
+### Declarative Pipelines
+- [[SPARK-58321]](https://issues.apache.org/jira/browse/SPARK-58321) Support
SCD Type 2 in Auto CDC end to end: each key's history is kept as versioned rows
bounded by `__START_AT` and `__END_AT` (the target table must be in a catalog
supporting DSv2 row-level operations, e.g. Apache Iceberg)
+- [[SPARK-58247]](https://issues.apache.org/jira/browse/SPARK-58247) Support
SCD Type 2 in the Auto CDC Python and Spark Connect APIs
(`stored_as_scd_type=2`, `track_history_column_list`)
+- [[SPARK-58261]](https://issues.apache.org/jira/browse/SPARK-58261) Support
SCD Type 2 syntax (`STORED AS SCD TYPE 2`, `TRACK HISTORY ON`) in SQL `AUTO CDC`
+- [[SPARK-57402]](https://issues.apache.org/jira/browse/SPARK-57402) Add SQL
`AUTO CDC` flows (`CREATE FLOW ... AS AUTO CDC INTO`), with clauses accepted in
any order ([[SPARK-58271]](https://issues.apache.org/jira/browse/SPARK-58271))
+- [[SPARK-58319]](https://issues.apache.org/jira/browse/SPARK-58319) Derive
the SCD Type 2 target table schema, and validate reserved columns
([[SPARK-57251]](https://issues.apache.org/jira/browse/SPARK-57251)),
track-history columns
([[SPARK-58313]](https://issues.apache.org/jira/browse/SPARK-58313)), and
sequencing types
([[SPARK-58391]](https://issues.apache.org/jira/browse/SPARK-58391)) when an
Auto CDC flow is defined
+- [[SPARK-58418]](https://issues.apache.org/jira/browse/SPARK-58418) Reconcile
SCD Type 2 micro-batches that are narrower than an evolved target schema
+- [[SPARK-57594]](https://issues.apache.org/jira/browse/SPARK-57594) Support
`spark_confs` in `create_auto_cdc_flow`
+
+### Infrastructure
+
+#### Build and Python Upgrades
+- [[SPARK-56889]](https://issues.apache.org/jira/browse/SPARK-56889) Drop
Python 3.10 support
+- [[SPARK-56928]](https://issues.apache.org/jira/browse/SPARK-56928) Increase
the minimum NumPy version to 1.23.2
+- [[SPARK-57877]](https://issues.apache.org/jira/browse/SPARK-57877) Support
the `DirectByteBuffer` cleaner on Java 26
+
+#### Observability
+- [[SPARK-55876]](https://issues.apache.org/jira/browse/SPARK-55876) Add a
query statistics summary bar to the SQL tab
+- [[SPARK-57987]](https://issues.apache.org/jira/browse/SPARK-57987) Expose
modified configs and node descriptions in the SQL REST API
+- [[SPARK-58468]](https://issues.apache.org/jira/browse/SPARK-58468) Support
on-demand loading for single event logs in the Spark History Server
+
+### Deployment
+
+#### Kubernetes
+- [[SPARK-57351]](https://issues.apache.org/jira/browse/SPARK-57351) Enable
`spark.kubernetes.executor.useDriverPodIP` by default
+- [[SPARK-57493]](https://issues.apache.org/jira/browse/SPARK-57493) Set
`allowPrivilegeEscalation` to false for driver and executor containers by
default
+- [[SPARK-57546]](https://issues.apache.org/jira/browse/SPARK-57546) Spark
Docker images now use `eclipse-temurin` on Ubuntu 26.04 with Python 3.14 by
default
+- [[SPARK-57700]](https://issues.apache.org/jira/browse/SPARK-57700) Support
opt-in `jemalloc` in Spark Docker images
+- [[SPARK-58113]](https://issues.apache.org/jira/browse/SPARK-58113) Support
`publishNotReadyAddresses` in the driver service
+- [[SPARK-58926]](https://issues.apache.org/jira/browse/SPARK-58926) Set the
driver pod's `SPARK_USER` from `--proxy-user`
+- [[SPARK-59798]](https://issues.apache.org/jira/browse/SPARK-59798) Support
`spark.executor.extraLibraryPath` for K8s executors
+
+#### Other
+- [[SPARK-38954]](https://issues.apache.org/jira/browse/SPARK-38954) Support
delegation token renewal and distribution without Kerberos
+- [[SPARK-57387]](https://issues.apache.org/jira/browse/SPARK-57387) Make the
executor JVM option `-XX:OnOutOfMemoryError` configurable on YARN
+- [[SPARK-58192]](https://issues.apache.org/jira/browse/SPARK-58192) Support
fractional `spark.task.cpus`
+
+---
+
+### Credits
+
+Last but not least, this release would not have been possible without the
following contributors:
+Aaron Congo, Abdelrahman Gamal, AbhinavBattu, Adam Binford, adith-os, Adithya
Ajith, Akira Ajisaka, Akshat Shenoi, Albert Sugranyes, Alek Jarmov, Aleksandar
Trajkovic, Aleksandr Efimov, Andreas Chatzistergiou, Andreas Neumann, Andy
Grove, Andy Lam, Angerszhuuuu, Anish Mahto, Anshul Baliga, anshulbhide, Anton
Okolnychyi, Anupam Yadav, Anurag Kumar Dwivedi, Anurag Mantripragada,
art000109, Ashutosh Jindal, Attila Zsolt Piros, Avery Qi, Ayush Bilala, Ben
Hollis, Bobby Wang, Bojana Zecevic, [...]
diff --git a/site/releases/spark-release-4-3-0.html
b/site/releases/spark-release-4-3-0.html
index 2b2c938126..ce4a47ce31 100644
--- a/site/releases/spark-release-4-3-0.html
+++ b/site/releases/spark-release-4-3-0.html
@@ -156,11 +156,427 @@
<h1>Spark Release 4.3.0</h1>
- <p>Apache Spark 4.3.0 is a new feature release. It introduces new
functionality and improvements. We encourage users to try it and provide
feedback.</p>
+ <p>Apache Spark 4.3.0 is the fourth release of the Spark 4.x line. With
significant contributions from the open-source community, this release
addressed over 1,100 Jira tickets with contributions from more than 210
individuals.</p>
-<p>You can find the list of resolved issues and detailed changes in the <a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12356944">JIRA
release notes</a>.</p>
+<p>This release brings stateful processing to Structured Streaming Real-Time
Mode, adds history-preserving SCD Type 2 Auto CDC to Spark Declarative
Pipelines, and makes Spark faster to iterate with on small data. It also marks
the start of a quarterly feature-release cadence.</p>
-<p>We would like to acknowledge all community members for contributing patches
and features to this release.</p>
+<p>Real-Time Mode now supports stateful queries: streaming aggregation, <code
class="language-plaintext highlighter-rouge">dropDuplicates</code>, and <code
class="language-plaintext highlighter-rouge">transformWithState</code> run with
end-to-end latency on the order of 100 milliseconds using the familiar
Structured Streaming APIs. Underneath, Spark gains native concurrent stage
scheduling and pipelined shuffles, so producer and consumer stages can run at
the same time.</p>
+
+<p>Auto CDC adds Slowly Changing Dimension (SCD) Type 2 through Python, SQL,
and Spark Connect, so pipelines can track how records change over time instead
of overwriting the latest value.</p>
+
+<p>Work toward faster queries in local laptop mode reduces fixed execution
overhead on small data, with opt-in shuffle-free single-task execution for
small queries and opt-in Arrow-native Dataset caching.</p>
+
+<p>Spark Connect adds REST endpoints for inspecting sessions and operations,
operation IDs for end-to-end request attribution, and a more complete JDBC
driver. SQL adds <code class="language-plaintext
highlighter-rouge">JSON_TABLE</code> and functions for transforming <code
class="language-plaintext highlighter-rouge">VARIANT</code> values, ANSI <code
class="language-plaintext highlighter-rouge">UNNEST</code>, SQL <code
class="language-plaintext highlighter-rouge">ASOF JOIN</code>, and a [...]
+
+<p>To download Apache Spark 4.3.0, please visit the <a
href="https://spark.apache.org/downloads.html">downloads</a> page. You can
consult JIRA for the <a
href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%204.3.0">detailed
changes</a>.</p>
+
+<ul id="markdown-toc">
+ <li><a href="#highlights" id="markdown-toc-highlights">Highlights</a></li>
+ <li><a href="#sql-foundation" id="markdown-toc-sql-foundation">SQL
Foundation</a> <ul>
+ <li><a href="#built-in-functions"
id="markdown-toc-built-in-functions">Built-in Functions</a></li>
+ <li><a href="#variant-manipulation"
id="markdown-toc-variant-manipulation">VARIANT manipulation</a></li>
+ <li><a href="#time-type" id="markdown-toc-time-type">TIME type</a></li>
+ <li><a href="#nanosecond-precision-timestamps-preview"
id="markdown-toc-nanosecond-precision-timestamps-preview">Nanosecond-precision
timestamps (preview)</a></li>
+ <li><a href="#notable-fixes" id="markdown-toc-notable-fixes">Notable
fixes</a></li>
+ </ul>
+ </li>
+ <li><a href="#query-api" id="markdown-toc-query-api">Query API</a> <ul>
+ <li><a href="#pandas-api-on-spark"
id="markdown-toc-pandas-api-on-spark">Pandas API on Spark</a></li>
+ </ul>
+ </li>
+ <li><a href="#connectors" id="markdown-toc-connectors">Connectors</a> <ul>
+ <li><a href="#data-source-v2-framework"
id="markdown-toc-data-source-v2-framework">Data Source V2 framework</a></li>
+ <li><a href="#reading-from-archives-opt-in"
id="markdown-toc-reading-from-archives-opt-in">Reading from archives
(opt-in)</a></li>
+ <li><a href="#file-sources" id="markdown-toc-file-sources">File
Sources</a></li>
+ <li><a href="#jdbc-hive-and-kafka"
id="markdown-toc-jdbc-hive-and-kafka">JDBC, Hive, and Kafka</a></li>
+ <li><a href="#notable-fixes-1" id="markdown-toc-notable-fixes-1">Notable
fixes</a></li>
+ </ul>
+ </li>
+ <li><a href="#udf-user-defined-functions"
id="markdown-toc-udf-user-defined-functions">UDF (User Defined
Functions)</a></li>
+ <li><a href="#streaming" id="markdown-toc-streaming">Streaming</a> <ul>
+ <li><a href="#real-time-mode" id="markdown-toc-real-time-mode">Real-Time
Mode</a></li>
+ <li><a href="#notable-fixes-2" id="markdown-toc-notable-fixes-2">Notable
fixes</a></li>
+ </ul>
+ </li>
+ <li><a href="#spark-connect-framework"
id="markdown-toc-spark-connect-framework">Spark Connect Framework</a> <ul>
+ <li><a href="#jdbc-driver" id="markdown-toc-jdbc-driver">JDBC
driver</a></li>
+ </ul>
+ </li>
+ <li><a href="#performance-and-stability"
id="markdown-toc-performance-and-stability">Performance and stability</a>
<ul>
+ <li><a href="#faster-queries-on-small-data"
id="markdown-toc-faster-queries-on-small-data">Faster queries on small
data</a></li>
+ <li><a href="#query-optimizer-and-execution"
id="markdown-toc-query-optimizer-and-execution">Query Optimizer and
Execution</a></li>
+ <li><a href="#python-performance"
id="markdown-toc-python-performance">Python Performance</a></li>
+ <li><a href="#stability" id="markdown-toc-stability">Stability</a></li>
+ <li><a href="#security" id="markdown-toc-security">Security</a></li>
+ </ul>
+ </li>
+ <li><a href="#declarative-pipelines"
id="markdown-toc-declarative-pipelines">Declarative Pipelines</a></li>
+ <li><a href="#infrastructure"
id="markdown-toc-infrastructure">Infrastructure</a> <ul>
+ <li><a href="#build-and-python-upgrades"
id="markdown-toc-build-and-python-upgrades">Build and Python Upgrades</a></li>
+ <li><a href="#observability"
id="markdown-toc-observability">Observability</a></li>
+ </ul>
+ </li>
+ <li><a href="#deployment" id="markdown-toc-deployment">Deployment</a> <ul>
+ <li><a href="#kubernetes"
id="markdown-toc-kubernetes">Kubernetes</a></li>
+ <li><a href="#other" id="markdown-toc-other">Other</a></li>
+ </ul>
+ </li>
+ <li><a href="#credits" id="markdown-toc-credits">Credits</a></li>
+</ul>
+
+<h3 id="highlights">Highlights</h3>
+<ul>
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-58635">[SPARK-58635]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-57228">[SPARK-57228]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-58508">[SPARK-58508]</a></strong>
Stateful Real-Time Mode — streaming aggregation, <code
class="language-plaintext highlighter-rouge">dropDuplicates</code>, and <code
class="language-plaintext highlighter-rouge">transformWithState</c [...]
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-58321">[SPARK-58321]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-58247">[SPARK-58247]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-58261">[SPARK-58261]</a></strong>
SCD Type 2 in Auto CDC — preserve the history of every record through Python,
SQL, and Spark Connect</li>
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-57851">[SPARK-57851]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-57268">[SPARK-57268]</a></strong>
Faster queries on small data — opt-in shuffle-free single-task execution for
small queries and Arrow-native Dataset caching for faster iteration on small
data</li>
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-57941">[SPARK-57941]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-58547">[SPARK-58547]</a></strong>
Spark Connect REST endpoints to list and inspect sessions and operations, with
operation IDs for end-to-end request attribution</li>
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-58366">[SPARK-58366]</a></strong>,
<strong><a
href="https://issues.apache.org/jira/browse/SPARK-57804">[SPARK-57804]</a></strong>
<code class="language-plaintext highlighter-rouge">JSON_TABLE</code> and <code
class="language-plaintext highlighter-rouge">VARIANT</code> transformation
functions (<code class="language-plaintext
highlighter-rouge">variant_set</code>, <code class="language-plaintext
highlighter-rouge">variant_ [...]
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-58092">[SPARK-58092]</a></strong>
SQL <code class="language-plaintext highlighter-rouge">ASOF JOIN</code> for
point-in-time matching of events to the information available at the time</li>
+ <li><strong><a
href="https://issues.apache.org/jira/browse/SPARK-56876">[SPARK-56876]</a></strong>
Nanosecond-precision timestamps <code class="language-plaintext
highlighter-rouge">TIMESTAMP_NTZ(p)</code> / <code class="language-plaintext
highlighter-rouge">TIMESTAMP_LTZ(p)</code> (preview)</li>
+</ul>
+
+<hr />
+
+<h3 id="sql-foundation">SQL Foundation</h3>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58366">[SPARK-58366]</a>
Support the <code class="language-plaintext
highlighter-rouge">JSON_TABLE</code> table-valued function</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58419">[SPARK-58419]</a>
Support ANSI SQL <code class="language-plaintext
highlighter-rouge">UNNEST</code> in the <code class="language-plaintext
highlighter-rouge">FROM</code> clause</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57421">[SPARK-57421]</a>
Support <code class="language-plaintext highlighter-rouge">@</code>-syntax
version time travel on table names</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58012">[SPARK-58012]</a>
Support <code class="language-plaintext highlighter-rouge">@</code>-syntax
timestamp time travel on table names</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-52719">[SPARK-52719]</a>
Support scalar UDFs in table-valued function arguments</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-54831">[SPARK-54831]</a>
Support <code class="language-plaintext highlighter-rouge">COMMENT ON
COLUMN</code> for table columns</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58065">[SPARK-58065]</a>
Support column lists for <code class="language-plaintext
highlighter-rouge">INSERT ... REPLACE WHERE</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57681">[SPARK-57681]</a>
Support dynamic table options for <code class="language-plaintext
highlighter-rouge">UPDATE</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58007">[SPARK-58007]</a>
Support dynamic table options for <code class="language-plaintext
highlighter-rouge">MERGE</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58008">[SPARK-58008]</a>
Support dynamic table options for <code class="language-plaintext
highlighter-rouge">DELETE</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57993">[SPARK-57993]</a>
Support an advisory partition size in the <code class="language-plaintext
highlighter-rouge">REBALANCE</code> hint</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-53840">[SPARK-53840]</a> Add
<code class="language-plaintext highlighter-rouge">AS JSON</code> output for
<code class="language-plaintext highlighter-rouge">SHOW TABLES</code> and <code
class="language-plaintext highlighter-rouge">SHOW TABLE EXTENDED</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-54710">[SPARK-54710]</a> Add
<code class="language-plaintext highlighter-rouge">AS JSON</code> output for
<code class="language-plaintext highlighter-rouge">SHOW PARTITIONS</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58260">[SPARK-58260]</a> Add
a config to render Hive DDL in <code class="language-plaintext
highlighter-rouge">SHOW CREATE TABLE</code> for Hive tables</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58052">[SPARK-58052]</a>
Throw overflow errors for <code class="language-plaintext
highlighter-rouge">ceil</code> and <code class="language-plaintext
highlighter-rouge">floor</code> in ANSI mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57775">[SPARK-57775]</a>
Return one row for <code class="language-plaintext highlighter-rouge">GROUP BY
GROUPING SETS (())</code> over empty input</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57194">[SPARK-57194]</a> Add
the <code class="language-plaintext
highlighter-rouge">preOperatorOptimizationRules</code> extension point to the
Optimizer</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56147">[SPARK-56147]</a> Use
a parser-based SQL statement splitter in the <code class="language-plaintext
highlighter-rouge">spark-sql</code> CLI</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57518">[SPARK-57518]</a> Make
Thrift server JDBC metadata operations DataSource V2 catalog-aware (opt-in via
<code class="language-plaintext
highlighter-rouge">spark.sql.thriftServer.catalogMetadata.enabled</code>)</li>
+ <li><strong>SQL support for ASOF join</strong> (<a
href="https://issues.apache.org/jira/browse/SPARK-58092">[SPARK-58092]</a>)
+ <ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58122">[SPARK-58122]</a> Add
SQL <code class="language-plaintext highlighter-rouge">ASOF JOIN ...
MATCH_CONDITION</code> syntax, enabled with <code class="language-plaintext
highlighter-rouge">spark.sql.join.asofJoin.enabled</code> (off by default)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58124">[SPARK-58124]</a>
Sort-merge physical execution for SQL <code class="language-plaintext
highlighter-rouge">ASOF JOIN</code></li>
+ </ul>
+ </li>
+ <li><strong>Add BIN BY relation operator for aligning range-typed rows to
fixed bin boundaries</strong> (<a
href="https://issues.apache.org/jira/browse/SPARK-57133">[SPARK-57133]</a>)
+ <ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57440">[SPARK-57440]</a>
<code class="language-plaintext highlighter-rouge">BIN BY</code> is available
behind <code class="language-plaintext
highlighter-rouge">spark.sql.binByRelationOperator.enabled</code> (off by
default)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58003">[SPARK-58003]</a>
Physical execution for <code class="language-plaintext highlighter-rouge">BIN
BY</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58064">[SPARK-58064]</a>
Filter pushdown for <code class="language-plaintext highlighter-rouge">BIN
BY</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58138">[SPARK-58138]</a>
Preserve child output partitioning through <code class="language-plaintext
highlighter-rouge">BIN BY</code></li>
+ </ul>
+ </li>
+</ul>
+
+<h4 id="built-in-functions">Built-in Functions</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56820">[SPARK-56820]</a> Add
the <code class="language-plaintext highlighter-rouge">counter_diff</code>
window function for converting cumulative counters to deltas</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57253">[SPARK-57253]</a> Add
the <code class="language-plaintext
highlighter-rouge">jaro_winkler_similarity</code> function</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58399">[SPARK-58399]</a> Add
the <code class="language-plaintext highlighter-rouge">collect_union</code>
aggregate function</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58295">[SPARK-58295]</a> Add
the <code class="language-plaintext highlighter-rouge">to_base32</code> and
<code class="language-plaintext highlighter-rouge">from_base32</code>
functions</li>
+</ul>
+
+<h4 id="variant-manipulation">VARIANT manipulation</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57804">[SPARK-57804]</a> Add
the <code class="language-plaintext highlighter-rouge">variant_set</code>
expression, plus <code class="language-plaintext
highlighter-rouge">try_variant_set</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-58221">[SPARK-58221]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57641">[SPARK-57641]</a> Add
the <code class="language-plaintext highlighter-rouge">variant_insert</code>
expression, plus <code class="language-plaintext
highlighter-rouge">try_variant_insert</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-58035">[SPARK-58035]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57417">[SPARK-57417]</a> Add
the <code class="language-plaintext highlighter-rouge">variant_delete</code>
expression</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58061">[SPARK-58061]</a> Add
the <code class="language-plaintext
highlighter-rouge">variant_array_append</code> expression, plus <code
class="language-plaintext highlighter-rouge">try_variant_array_append</code>
(<a
href="https://issues.apache.org/jira/browse/SPARK-58222">[SPARK-58222]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58062">[SPARK-58062]</a> Add
the <code class="language-plaintext
highlighter-rouge">variant_strip_nulls</code> expression</li>
+</ul>
+
+<h4 id="time-type">TIME type</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57551">[SPARK-57551]</a>
Extend <code class="language-plaintext highlighter-rouge">TIME</code> precision
to nanoseconds (<code class="language-plaintext
highlighter-rouge">TIME(p)</code> with <code class="language-plaintext
highlighter-rouge">p</code> up to 9)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57558">[SPARK-57558]</a> Add
the <code class="language-plaintext highlighter-rouge">LOCALTIME</code>
datetime value function</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57618">[SPARK-57618]</a>
Support casting between <code class="language-plaintext
highlighter-rouge">TIME(p)</code> and <code class="language-plaintext
highlighter-rouge">TIMESTAMP_NTZ(q)</code> / <code class="language-plaintext
highlighter-rouge">TIMESTAMP_LTZ(q)</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-57660">[SPARK-57660]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57572">[SPARK-57572]</a>
Infer <code class="language-plaintext highlighter-rouge">TIME</code> during CSV
and JSON schema inference, and in XML (<a
href="https://issues.apache.org/jira/browse/SPARK-57634">[SPARK-57634]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57555">[SPARK-57555]</a>
Support <code class="language-plaintext highlighter-rouge">TIME</code> in the
built-in JDBC data source and the Protobuf connector (<a
href="https://issues.apache.org/jira/browse/SPARK-57573">[SPARK-57573]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57571">[SPARK-57571]</a>
<code class="language-plaintext highlighter-rouge">TIME</code> predicate
pushdown to ORC, and Parquet/ORC aggregate pushdown (<a
href="https://issues.apache.org/jira/browse/SPARK-57568">[SPARK-57568]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57557">[SPARK-57557]</a>
Support <code class="language-plaintext highlighter-rouge">TIME</code> in
quantile aggregates, <code class="language-plaintext
highlighter-rouge">approx_top_k</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-57848">[SPARK-57848]</a>),
and <code class="language-plaintext highlighter-rouge">to_char</code>/<code
class="language-plaintext highlighter-rouge">to_varchar</code> (<a
href="https://issu [...]
+</ul>
+
+<h4 id="nanosecond-precision-timestamps-preview">Nanosecond-precision
timestamps (preview)</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56876">[SPARK-56876]</a> Add
the <code class="language-plaintext highlighter-rouge">TIMESTAMP_NTZ(p)</code>
and <code class="language-plaintext highlighter-rouge">TIMESTAMP_LTZ(p)</code>
types with precision <code class="language-plaintext
highlighter-rouge">p</code> from 7 through 9, as a preview enabled with <code
class="language-plaintext
highlighter-rouge">spark.sql.timestampNanosTypes.enabled</code> (off by
default)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56965">[SPARK-56965]</a> SQL
parser support for <code class="language-plaintext
highlighter-rouge">TIMESTAMP_NTZ(p)</code> and <code class="language-plaintext
highlighter-rouge">TIMESTAMP_LTZ(p)</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57293">[SPARK-57293]</a>
Casts between nanosecond- and microsecond-precision timestamps, and to and from
<code class="language-plaintext highlighter-rouge">DATE</code>, <code
class="language-plaintext highlighter-rouge">TIME</code>, and strings</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57454">[SPARK-57454]</a> Type
coercion and widening rules for nanosecond-precision timestamps</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57526">[SPARK-57526]</a> Add
the <code class="language-plaintext highlighter-rouge">timestamp_nanos</code>
and <code class="language-plaintext highlighter-rouge">unix_nanos</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-57527">[SPARK-57527]</a>)
functions, and the precision argument in <code class="language-plaintext
highlighter-rouge">current_timestamp(p)</code> (<a
href="https://issues.apache.org/jira/browse/SPARK [...]
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57816">[SPARK-57816]</a>
Datetime functions (<code class="language-plaintext
highlighter-rouge">date_format</code>, <code class="language-plaintext
highlighter-rouge">date_trunc</code>, <code class="language-plaintext
highlighter-rouge">EXTRACT</code>, <code class="language-plaintext
highlighter-rouge">hour</code>/<code class="language-plaintext
highlighter-rouge">minute</code>/<code class="language-plaintext
highlighter-rouge">s [...]
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57501">[SPARK-57501]</a> ANSI
interval arithmetic and timestamp subtraction for nanosecond-precision
timestamps</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57102">[SPARK-57102]</a>
Parquet support, including the vectorized reader (<a
href="https://issues.apache.org/jira/browse/SPARK-57828">[SPARK-57828]</a>) and
predicate pushdown (<a
href="https://issues.apache.org/jira/browse/SPARK-57822">[SPARK-57822]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57455">[SPARK-57455]</a> ORC,
JSON, CSV, XML, Avro, and JDBC data source support for nanosecond-precision
timestamps</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57160">[SPARK-57160]</a>
Arrow and Spark Connect support for nanosecond-precision timestamp types and
literals</li>
+</ul>
+
+<h4 id="notable-fixes">Notable fixes</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-60304">[SPARK-60304]</a> Fix
subplan merging losing a shared lower filter or a reused filter attribute (<a
href="https://issues.apache.org/jira/browse/SPARK-60084">[SPARK-60084]</a>),
which could produce wrong results</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57956">[SPARK-57956]</a> Fix
AQE incorrectly eliminating a global <code class="language-plaintext
highlighter-rouge">LIMIT</code> over a non-materialized query stage</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-60508">[SPARK-60508]</a> Fix
wrong results from <code class="language-plaintext
highlighter-rouge">semanticEquals</code> of higher-order functions capturing
lambda variables</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58384">[SPARK-58384]</a>
Preserve null semantics in <code class="language-plaintext
highlighter-rouge">OptimizeJoinCondition</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58481">[SPARK-58481]</a> Fix
<code class="language-plaintext highlighter-rouge">IN (subquery)</code>
returning wrong results when the subquery output contains NULLs</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59434">[SPARK-59434]</a> Do
not push non-deterministic predicates into CTE definitions</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58959">[SPARK-58959]</a> Fix
<code class="language-plaintext highlighter-rouge">PIVOT</code> with duplicate
values corrupting the aggregation buffer</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58816">[SPARK-58816]</a> Fix
<code class="language-plaintext highlighter-rouge">INSERT</code> with a column
list to resolve structs inside arrays and maps positionally</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57727">[SPARK-57727]</a> Fix
incorrect query results from constraint inference substituting attributes with
non-binary-stable collations</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58204">[SPARK-58204]</a> Fix
expressions that held unprotected mutable state (regex, <code
class="language-plaintext highlighter-rouge">translate</code>, <code
class="language-plaintext highlighter-rouge">format_number</code>, JSON/CSV/XML
(<a
href="https://issues.apache.org/jira/browse/SPARK-58205">[SPARK-58205]</a>),
<code class="language-plaintext
highlighter-rouge">ApplyFunctionExpression</code> (<a
href="https://issues.apach [...]
+</ul>
+
+<h3 id="query-api">Query API</h3>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-55886">[SPARK-55886]</a> Add
<code class="language-plaintext highlighter-rouge">DataFrame.zip</code> for
merging column-projected DataFrames</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57284">[SPARK-57284]</a> Add
Scala and Python bindings for vector functions</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-46776">[SPARK-46776]</a>
Support <code class="language-plaintext
highlighter-rouge">pa.ChunkedArray</code> columns in <code
class="language-plaintext highlighter-rouge">createDataFrame</code> from
pandas</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57190">[SPARK-57190]</a> Fix
the API inconsistency of the 4-argument <code class="language-plaintext
highlighter-rouge">regexp_replace</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57860">[SPARK-57860]</a> Make
the storage level of MLlib intermediate datasets configurable for KMeans and
ALS</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-34591">[SPARK-34591]</a> Add
decision tree pruning as a parameter</li>
+</ul>
+
+<h4 id="pandas-api-on-spark">Pandas API on Spark</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-40336">[SPARK-40336]</a>
Implement <code class="language-plaintext
highlighter-rouge">DataFrameGroupBy.cov</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-40337">[SPARK-40337]</a>
Implement <code class="language-plaintext
highlighter-rouge">SeriesGroupBy.describe</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-40340">[SPARK-40340]</a>
Implement <code class="language-plaintext highlighter-rouge">Rolling.sem</code>
and <code class="language-plaintext highlighter-rouge">Expanding.sem</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-40341">[SPARK-40341]</a>
Implement <code class="language-plaintext
highlighter-rouge">Rolling.median</code> and <code class="language-plaintext
highlighter-rouge">Expanding.median</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-40373">[SPARK-40373]</a>
Implement <code class="language-plaintext
highlighter-rouge">ps.show_versions</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-46158">[SPARK-46158]</a>
Support axis columns in <code class="language-plaintext
highlighter-rouge">DataFrame.xs</code></li>
+</ul>
+
+<h3 id="connectors">Connectors</h3>
+
+<h4 id="data-source-v2-framework">Data Source V2 framework</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57644">[SPARK-57644]</a>
Support generated column values on V2 table writes</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-55347">[SPARK-55347]</a> Pass
generated column expressions to DSv2 connectors</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58523">[SPARK-58523]</a> Add
a Catalyst runtime filtering interface for DSv2 scans</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-55617">[SPARK-55617]</a> Push
down <code class="language-plaintext highlighter-rouge">variant_get</code>
filters to DSv2 sources</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56660">[SPARK-56660]</a>
Decompose struct equality into field predicates for data source pushdown</li>
+</ul>
+
+<h4 id="reading-from-archives-opt-in">Reading from archives (opt-in)</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57135">[SPARK-57135]</a> Read
CSV files inside tar archives, with schema inference (<a
href="https://issues.apache.org/jira/browse/SPARK-57321">[SPARK-57321]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57419">[SPARK-57419]</a> Read
and infer JSON schema from tar archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57479">[SPARK-57479]</a> Read
and infer XML schema from tar archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57478">[SPARK-57478]</a> Read
text files from tar archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57481">[SPARK-57481]</a> Read
and infer Avro schema from tar archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57590">[SPARK-57590]</a> Read
and infer Parquet schema from archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57591">[SPARK-57591]</a> Read
and infer ORC schema from archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57705">[SPARK-57705]</a> Read
CSV, JSON, text, and XML files from zip archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58246">[SPARK-58246]</a> Read
files from 7z archives</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58382">[SPARK-58382]</a>
Support archives in the <code class="language-plaintext
highlighter-rouge">binaryFile</code> source via the <code
class="language-plaintext highlighter-rouge">wholeFile</code> option</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58609">[SPARK-58609]</a> Add
the <code class="language-plaintext highlighter-rouge">archivePathFilter</code>
option to select entries inside archives</li>
+</ul>
+
+<h4 id="file-sources">File Sources</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57354">[SPARK-57354]</a> Add
the <code class="language-plaintext
highlighter-rouge">ignoredPathSegmentRegex</code> data source option and
config</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57678">[SPARK-57678]</a> Add
the <code class="language-plaintext
highlighter-rouge">explodeEmbeddedArray</code> JSON option for streaming scans
of embedded arrays</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57802">[SPARK-57802]</a> Make
XML schema inference incremental, aligning with CSV</li>
+</ul>
+
+<h4 id="jdbc-hive-and-kafka">JDBC, Hive, and Kafka</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-52825">[SPARK-52825]</a>
Register existing JDBC dialects for additional URL prefixes</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59330">[SPARK-59330]</a> Add
an option to defer Hive <code class="language-plaintext
highlighter-rouge">InputFormat</code> and <code class="language-plaintext
highlighter-rouge">OutputFormat</code> class initialization</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59328">[SPARK-59328]</a> Add
an option to reject specified Kafka options in the Kafka data source</li>
+</ul>
+
+<h4 id="notable-fixes-1">Notable fixes</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58112">[SPARK-58112]</a> Keep
nondeterministic filters out of DSv2 Catalyst and V2 filter pushdown (<a
href="https://issues.apache.org/jira/browse/SPARK-58207">[SPARK-58207]</a>), so
Spark evaluates them after the scan</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59108">[SPARK-59108]</a> Fix
Avro positional field matching under column pruning</li>
+</ul>
+
+<h3 id="udf-user-defined-functions">UDF (User Defined Functions)</h3>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-55206">[SPARK-55206]</a>
Experimental transpilation of simple Python UDFs to Catalyst expressions (e.g.
<code class="language-plaintext highlighter-rouge">len()</code> to <code
class="language-plaintext highlighter-rouge">length</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-55214">[SPARK-55214]</a>)),
enabled with <code class="language-plaintext
highlighter-rouge">spark.sql.experimental.optimizer.transpilePyUDFs</c [...]
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56413">[SPARK-56413]</a> Add
a gRPC UDF execution protocol and logical/physical planning nodes (<a
href="https://issues.apache.org/jira/browse/SPARK-56661">[SPARK-56661]</a>) as
groundwork for language-agnostic UDFs</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57271">[SPARK-57271]</a>
Optionally print local variables in Python UDF tracebacks</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57593">[SPARK-57593]</a>
Bound pickled Python UDF input batches by size, with instrumentation</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58601">[SPARK-58601]</a>
Require <code class="language-plaintext highlighter-rouge">mapInPandas</code>
functions to return a strict iterator</li>
+</ul>
+
+<h3 id="streaming">Streaming</h3>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58724">[SPARK-58724]</a>
Incremental state cleanup for streaming <code class="language-plaintext
highlighter-rouge">dropDuplicates</code></li>
+</ul>
+
+<h4 id="real-time-mode">Real-Time Mode</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58508">[SPARK-58508]</a>
Support stateful queries in Real-Time Mode, including <code
class="language-plaintext highlighter-rouge">dropDuplicates</code>, by running
them on pipelined shuffles</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58635">[SPARK-58635]</a>
Support streaming aggregation in Real-Time Mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57228">[SPARK-57228]</a>
Support Scala/Java <code class="language-plaintext
highlighter-rouge">transformWithState</code> in Real-Time Mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57281">[SPARK-57281]</a>
Remove the <code class="language-plaintext
highlighter-rouge">@Experimental</code> annotation from Real-Time Mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58131">[SPARK-58131]</a>
Support async progress tracking in Real-Time Mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58185">[SPARK-58185]</a> Add
<code class="language-plaintext
highlighter-rouge">PipelinedShuffleDependency</code>, letting consumers read
while producers are still running, and route each shuffle to a shuffle manager
by dependency type</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58263">[SPARK-58263]</a>
Natively schedule pipelined stage groups concurrently in the DAGScheduler, with
group-atomic failure and fail-fast admission when task slots are insufficient
(<a
href="https://issues.apache.org/jira/browse/SPARK-58398">[SPARK-58398]</a>),
decoupled from the <code class="language-plaintext
highlighter-rouge">MapOutputTracker</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-58454">[SPARK-58454]< [...]
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57141">[SPARK-57141]</a> Add
<code class="language-plaintext
highlighter-rouge">StreamingShuffleManager</code>, the built-in pipelined
shuffle implementation, with a Netty-based wire protocol (<a
href="https://issues.apache.org/jira/browse/SPARK-56674">[SPARK-56674]</a>)</li>
+</ul>
+
+<h4 id="notable-fixes-2">Notable fixes</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59315">[SPARK-59315]</a> Fix
the correctness of <code class="language-plaintext
highlighter-rouge">dropDuplicates</code> with a watermark after query
restart</li>
+</ul>
+
+<h3 id="spark-connect-framework">Spark Connect Framework</h3>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57941">[SPARK-57941]</a>
Expose Spark Connect sessions and operations via a REST API</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57021">[SPARK-57021]</a> Add
a <code class="language-plaintext highlighter-rouge">SQLContext</code> wrapper
for Spark Connect</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57787">[SPARK-57787]</a>
Reuse a persistent local Spark Connect server across runs for faster local
PySpark startup (opt-in via <code class="language-plaintext
highlighter-rouge">SPARK_LOCAL_CONNECT_REUSE=1</code>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58159">[SPARK-58159]</a>
Support the <code class="language-plaintext highlighter-rouge">with</code>
statement for Spark Connect sessions</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58547">[SPARK-58547]</a>
Expose operation IDs for end-to-end request attribution</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57247">[SPARK-57247]</a>
Support <code class="language-plaintext highlighter-rouge">DataFrame.zip</code>
in Spark Connect</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58251">[SPARK-58251]</a> Size
estimates for nearly all Spark ML models (feature transformers, tree,
clustering, ALS, LDA, FPGrowth, and tuning models), with fixes for pipeline
overcounting (<a
href="https://issues.apache.org/jira/browse/SPARK-58281">[SPARK-58281]</a>),
for better Spark Connect ML model cache management</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-44462">[SPARK-44462]</a> Fix
the session passed to <code class="language-plaintext
highlighter-rouge">foreachBatch</code> in Spark Connect</li>
+</ul>
+
+<h4 id="jdbc-driver">JDBC driver</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57380">[SPARK-57380]</a>
Support complex types (<code class="language-plaintext
highlighter-rouge">ARRAY</code>, <code class="language-plaintext
highlighter-rouge">MAP</code>, <code class="language-plaintext
highlighter-rouge">STRUCT</code>) in the Connect JDBC driver</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57627">[SPARK-57627]</a>
Support primary key, foreign key, and SQL type metadata in <code
class="language-plaintext highlighter-rouge">DatabaseMetaData</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57274">[SPARK-57274]</a>
Support fetch/type accessors and <code class="language-plaintext
highlighter-rouge">getMoreResults</code> for <code class="language-plaintext
highlighter-rouge">SparkConnectStatement</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57933">[SPARK-57933]</a> Map
Spark Connect client failures to <code class="language-plaintext
highlighter-rouge">SQLException</code>, and connection errors by SQLSTATE class
<code class="language-plaintext highlighter-rouge">08</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-58619">[SPARK-58619]</a>)</li>
+</ul>
+
+<h3 id="performance-and-stability">Performance and stability</h3>
+<ul>
+ <li><strong>Add bulk read+convert paths for type-converting Parquet vector
updaters</strong> (<a
href="https://issues.apache.org/jira/browse/SPARK-56800">[SPARK-56800]</a>)
+ <ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56791">[SPARK-56791]</a> Bulk
read-and-convert paths in the vectorized Parquet reader for widening (INT32 to
Long/Double, FLOAT to Double), INT64 <code class="language-plaintext
highlighter-rouge">DECIMAL</code> narrowing, and <code
class="language-plaintext highlighter-rouge">DATE</code> to <code
class="language-plaintext highlighter-rouge">TIMESTAMP_NTZ</code>
conversions</li>
+ </ul>
+ </li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56893">[SPARK-56893]</a>
Faster vectorized Parquet decoding: dictionary decoding fast paths, bulk <code
class="language-plaintext highlighter-rouge">DELTA_BINARY_PACKED</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-56892">[SPARK-56892]</a>) and
timestamp/date (<a
href="https://issues.apache.org/jira/browse/SPARK-56896">[SPARK-56896]</a>)
reads, and lower-allocation RLE decoding (<a href="https://issues.apache.org/j
[...]
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56894">[SPARK-56894]</a> Add
a vectorized Parquet <code class="language-plaintext
highlighter-rouge">BYTE_STREAM_SPLIT</code> reader</li>
+</ul>
+
+<h4 id="faster-queries-on-small-data">Faster queries on small data</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57851">[SPARK-57851]</a>
Opt-in shuffle-free single-task execution for eligible small, single-partition
query fragments (<code class="language-plaintext
highlighter-rouge">spark.sql.optimizer.singleTaskExecution.enabled</code>),
covering scans, aggregations, distinct, windows, sorts, and limits</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57268">[SPARK-57268]</a>
Opt-in Arrow-native in-memory Dataset caching with compression and column
statistics (<code class="language-plaintext
highlighter-rouge">spark.sql.cache.serializer=org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer</code>)</li>
+</ul>
+
+<h4 id="query-optimizer-and-execution">Query Optimizer and Execution</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58210">[SPARK-58210]</a>
Enable <code class="language-plaintext
highlighter-rouge">ReplaceHashWithSortAgg</code> and combining adjacent
aggregations (<a
href="https://issues.apache.org/jira/browse/SPARK-43317">[SPARK-43317]</a>) by
default</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56805">[SPARK-56805]</a>
Enable whole-stage codegen for <code class="language-plaintext
highlighter-rouge">Union</code> by default</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-37019">[SPARK-37019]</a> Add
codegen support for array higher-order functions</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-40259">[SPARK-40259]</a>
Merge equivalent DataSource V2 scans in subplan merging</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56887">[SPARK-56887]</a> Add
a dedicated sort-merge physical operator for AS-OF joins</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56903">[SPARK-56903]</a>
Opt-in (<code class="language-plaintext
highlighter-rouge">spark.sql.shuffle.spreadNullJoinKeys.enabled</code>)
spreading of NULL outer join keys, and left anti join keys (<a
href="https://issues.apache.org/jira/browse/SPARK-57282">[SPARK-57282]</a>),
across shuffle partitions to avoid skew</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57370">[SPARK-57370]</a> Add
a JDK (<code class="language-plaintext highlighter-rouge">javac</code>)
compiler backend for whole-stage codegen</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58066">[SPARK-58066]</a>
Hoist streamed-side predicates in outer, anti, and existence joins</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58084">[SPARK-58084]</a>
Support converting sort merge join to shuffled hash join in the AQE physical
plan</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58099">[SPARK-58099]</a>
Remove the redundant local sort left below the shuffle added by skew join
optimization</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58233">[SPARK-58233]</a> Push
down local sorts to reduce redundant sorts within a stage</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58405">[SPARK-58405]</a> Pull
up project aliases through windows to eliminate redundant shuffles and
sorts</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-54593">[SPARK-54593]</a>
Enable dynamic partition pruning for materialized filtering sides</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58265">[SPARK-58265]</a>
Reuse projected broadcast values for dynamic partition pruning</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58272">[SPARK-58272]</a>
Enable runtime Bloom filters for materialized cached inputs</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58089">[SPARK-58089]</a> Push
variant extractions through Aggregate, Sort, and Join</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57022">[SPARK-57022]</a>
Nested column pruning for <code class="language-plaintext
highlighter-rouge">transform</code> over arrays of structs and through
array-returning functions (<a
href="https://issues.apache.org/jira/browse/SPARK-57176">[SPARK-57176]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57043">[SPARK-57043]</a>
Collapse grouped <code class="language-plaintext
highlighter-rouge">SUM(COUNT(1))</code> rollups</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57220">[SPARK-57220]</a>
Extend the segment-tree window frame to shrinking frames and to <code
class="language-plaintext highlighter-rouge">first</code>/<code
class="language-plaintext highlighter-rouge">last</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-57424">[SPARK-57424]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57937">[SPARK-57937]</a>
Improve decimal division performance by dividing at the result scale</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56769">[SPARK-56769]</a> Add
a fast path for <code class="language-plaintext
highlighter-rouge">date_trunc</code> at WEEK/MONTH/QUARTER/YEAR granularity</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57905">[SPARK-57905]</a>
Reduce the size of generated whole-stage codegen, e.g. for null-guarded row
writes and <code class="language-plaintext
highlighter-rouge">regexp_extract</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-57258">[SPARK-57258]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57688">[SPARK-57688]</a> Add
<code class="language-plaintext
highlighter-rouge">spark.sql.execution.bypassPartialAggregation</code> to skip
partial aggregation</li>
+</ul>
+
+<h4 id="python-performance">Python Performance</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56642">[SPARK-56642]</a> Add
pipelined JVM-Python UDF data transfer, with an optional Unix Domain Socket
mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58230">[SPARK-58230]</a>
Batch pandas-on-Spark column projections</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58249">[SPARK-58249]</a> Use
native Spark functions for NumPy math ufuncs, bitwise shifts (<a
href="https://issues.apache.org/jira/browse/SPARK-58464">[SPARK-58464]</a>),
<code class="language-plaintext highlighter-rouge">rint</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-58469">[SPARK-58469]</a>),
<code class="language-plaintext highlighter-rouge">trunc</code> (<a
href="https://issues.apache.org/jira/browse/SPARK- [...]
+</ul>
+
+<h4 id="stability">Stability</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57723">[SPARK-57723]</a>
Enable <code class="language-plaintext
highlighter-rouge">spark.rdd.compress</code> by default, as in PySpark</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58107">[SPARK-58107]</a>
Detect and repair non-deterministic local checkpoints via RDD-block content
checksums</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57491">[SPARK-57491]</a>
Prevent and detect stale push-based shuffle data from duplicate map task
attempts</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58848">[SPARK-58848]</a>
Prevent unbounded shuffle reader connection waits</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58004">[SPARK-58004]</a> Add
a hard fan-in bound to AQE shuffle partition coalescing</li>
+</ul>
+
+<h4 id="security">Security</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58054">[SPARK-58054]</a>
Enable <code class="language-plaintext
highlighter-rouge">spark.ui.contentSecurityPolicy.enabled</code> by default</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57904">[SPARK-57904]</a>
Change the default of <code class="language-plaintext
highlighter-rouge">spark.ui.xXssProtection</code> to <code
class="language-plaintext highlighter-rouge">0</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59319">[SPARK-59319]</a> Add
an opt-in restricted SQL execution mode</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57448">[SPARK-57448]</a>
Support <code class="language-plaintext
highlighter-rouge">spark.sql.reflect.allowList</code> to restrict the methods
invoked by <code class="language-plaintext
highlighter-rouge">reflect</code>/<code class="language-plaintext
highlighter-rouge">java_method</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57441">[SPARK-57441]</a>
Disallow setting JVM system properties in Spark Thrift Server sessions,
including via <code class="language-plaintext
highlighter-rouge">set:hiveconf</code> (<a
href="https://issues.apache.org/jira/browse/SPARK-57480">[SPARK-57480]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-54293">[SPARK-54293]</a> Make
the <code class="language-plaintext
highlighter-rouge">ThriftHttpCLIService</code> SNI host check configurable</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58370">[SPARK-58370]</a>
Check table write privileges when the write target is also read in the same
statement</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59329">[SPARK-59329]</a> Add
an allowlist of URI schemes for the Avro <code class="language-plaintext
highlighter-rouge">schemaUrl</code> option</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57453">[SPARK-57453]</a>
Support <code class="language-plaintext
highlighter-rouge">spark.master.rest.allowedAppResourcePatterns</code> to
restrict application resources submitted through the Master REST server, and
limit its request body size (<a
href="https://issues.apache.org/jira/browse/SPARK-58049">[SPARK-58049]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58056">[SPARK-58056]</a> Make
<code class="language-plaintext highlighter-rouge">NettyStreamManager</code>
serve only files under the registered directory</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58153">[SPARK-58153]</a>
Reject JAR entries escaping the extraction directory in <code
class="language-plaintext highlighter-rouge">RPackageUtils</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59508">[SPARK-59508]</a>
Harden Python client safety with a restricted unpickle serializer</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59320">[SPARK-59320]</a>
Optionally require an expiration claim in <code class="language-plaintext
highlighter-rouge">JWSFilter</code></li>
+</ul>
+
+<h3 id="declarative-pipelines">Declarative Pipelines</h3>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58321">[SPARK-58321]</a>
Support SCD Type 2 in Auto CDC end to end: each key’s history is kept as
versioned rows bounded by <code class="language-plaintext
highlighter-rouge">__START_AT</code> and <code class="language-plaintext
highlighter-rouge">__END_AT</code> (the target table must be in a catalog
supporting DSv2 row-level operations, e.g. Apache Iceberg)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58247">[SPARK-58247]</a>
Support SCD Type 2 in the Auto CDC Python and Spark Connect APIs (<code
class="language-plaintext highlighter-rouge">stored_as_scd_type=2</code>, <code
class="language-plaintext
highlighter-rouge">track_history_column_list</code>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58261">[SPARK-58261]</a>
Support SCD Type 2 syntax (<code class="language-plaintext
highlighter-rouge">STORED AS SCD TYPE 2</code>, <code class="language-plaintext
highlighter-rouge">TRACK HISTORY ON</code>) in SQL <code
class="language-plaintext highlighter-rouge">AUTO CDC</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57402">[SPARK-57402]</a> Add
SQL <code class="language-plaintext highlighter-rouge">AUTO CDC</code> flows
(<code class="language-plaintext highlighter-rouge">CREATE FLOW ... AS AUTO CDC
INTO</code>), with clauses accepted in any order (<a
href="https://issues.apache.org/jira/browse/SPARK-58271">[SPARK-58271]</a>)</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58319">[SPARK-58319]</a>
Derive the SCD Type 2 target table schema, and validate reserved columns (<a
href="https://issues.apache.org/jira/browse/SPARK-57251">[SPARK-57251]</a>),
track-history columns (<a
href="https://issues.apache.org/jira/browse/SPARK-58313">[SPARK-58313]</a>),
and sequencing types (<a
href="https://issues.apache.org/jira/browse/SPARK-58391">[SPARK-58391]</a>)
when an Auto CDC flow is defined</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58418">[SPARK-58418]</a>
Reconcile SCD Type 2 micro-batches that are narrower than an evolved target
schema</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57594">[SPARK-57594]</a>
Support <code class="language-plaintext highlighter-rouge">spark_confs</code>
in <code class="language-plaintext
highlighter-rouge">create_auto_cdc_flow</code></li>
+</ul>
+
+<h3 id="infrastructure">Infrastructure</h3>
+
+<h4 id="build-and-python-upgrades">Build and Python Upgrades</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56889">[SPARK-56889]</a> Drop
Python 3.10 support</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-56928">[SPARK-56928]</a>
Increase the minimum NumPy version to 1.23.2</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57877">[SPARK-57877]</a>
Support the <code class="language-plaintext
highlighter-rouge">DirectByteBuffer</code> cleaner on Java 26</li>
+</ul>
+
+<h4 id="observability">Observability</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-55876">[SPARK-55876]</a> Add
a query statistics summary bar to the SQL tab</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57987">[SPARK-57987]</a>
Expose modified configs and node descriptions in the SQL REST API</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58468">[SPARK-58468]</a>
Support on-demand loading for single event logs in the Spark History Server</li>
+</ul>
+
+<h3 id="deployment">Deployment</h3>
+
+<h4 id="kubernetes">Kubernetes</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57351">[SPARK-57351]</a>
Enable <code class="language-plaintext
highlighter-rouge">spark.kubernetes.executor.useDriverPodIP</code> by
default</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57493">[SPARK-57493]</a> Set
<code class="language-plaintext
highlighter-rouge">allowPrivilegeEscalation</code> to false for driver and
executor containers by default</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57546">[SPARK-57546]</a>
Spark Docker images now use <code class="language-plaintext
highlighter-rouge">eclipse-temurin</code> on Ubuntu 26.04 with Python 3.14 by
default</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57700">[SPARK-57700]</a>
Support opt-in <code class="language-plaintext
highlighter-rouge">jemalloc</code> in Spark Docker images</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58113">[SPARK-58113]</a>
Support <code class="language-plaintext
highlighter-rouge">publishNotReadyAddresses</code> in the driver service</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58926">[SPARK-58926]</a> Set
the driver pod’s <code class="language-plaintext
highlighter-rouge">SPARK_USER</code> from <code class="language-plaintext
highlighter-rouge">--proxy-user</code></li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-59798">[SPARK-59798]</a>
Support <code class="language-plaintext
highlighter-rouge">spark.executor.extraLibraryPath</code> for K8s executors</li>
+</ul>
+
+<h4 id="other">Other</h4>
+<ul>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-38954">[SPARK-38954]</a>
Support delegation token renewal and distribution without Kerberos</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-57387">[SPARK-57387]</a> Make
the executor JVM option <code class="language-plaintext
highlighter-rouge">-XX:OnOutOfMemoryError</code> configurable on YARN</li>
+ <li><a
href="https://issues.apache.org/jira/browse/SPARK-58192">[SPARK-58192]</a>
Support fractional <code class="language-plaintext
highlighter-rouge">spark.task.cpus</code></li>
+</ul>
+
+<hr />
+
+<h3 id="credits">Credits</h3>
+
+<p>Last but not least, this release would not have been possible without the
following contributors:
+Aaron Congo, Abdelrahman Gamal, AbhinavBattu, Adam Binford, adith-os, Adithya
Ajith, Akira Ajisaka, Akshat Shenoi, Albert Sugranyes, Alek Jarmov, Aleksandar
Trajkovic, Aleksandr Efimov, Andreas Chatzistergiou, Andreas Neumann, Andy
Grove, Andy Lam, Angerszhuuuu, Anish Mahto, Anshul Baliga, anshulbhide, Anton
Okolnychyi, Anupam Yadav, Anurag Kumar Dwivedi, Anurag Mantripragada,
art000109, Ashutosh Jindal, Attila Zsolt Piros, Avery Qi, Ayush Bilala, Ben
Hollis, Bobby Wang, Bojana Zecevic, [...]
<p>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]