Rakesh Kumar created SPARK-60763:
------------------------------------
Summary: Add migration note for the V2 scan grouping change in
Spark 4.2
Key: SPARK-60763
URL: https://issues.apache.org/jira/browse/SPARK-60763
Project: Spark
Issue Type: Documentation
Components: Documentation
Affects Versions: 4.2.0
Reporter: Rakesh Kumar
SPARK-55092 changed Spark 4.2 so that V2 scans no longer group input splits
that share a reported partition key into one task unless an operator needs that
grouping, such as a storage-partitioned join. Scans below a shuffle, or with no
consumer that uses their partitioning, now run one task per split, which can
add task overhead for connectors that report many small splits per key. Before
4.2, grouping multiple splits into one task also dropped their reported
ordering.
This change had no migration note. Add one to the "Upgrading from Spark SQL 4.1
to 4.2" section, and merge it to branch-4.2 as well as master. Raised in the
review of SPARK-60265 (https://github.com/apache/spark/pull/59522).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
