DB Tsai created SPARK-60745:
-------------------------------
Summary: Avoid row materialization for Arrow-backed pandas string
input in Spark Connect Rust
Key: SPARK-60745
URL: https://issues.apache.org/jira/browse/SPARK-60745
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Pandas string extension columns still use Python row materialization in Spark
Connect Rust. SPARK-60720 / PR144 deliberately supports concrete NumPy object
storage and excludes extension dtypes. Its exact candidate
d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723 retains the pandas 3.1 StringDtype
benchmark at 22.310792 ms warm / 24.347666 ms first construction for 10,000 x 3
input. The old benchmark did not separately record the inferred backing class,
so these timings motivate investigation rather than establish an
ArrowStringArray speedup.
Evaluate one conservative path for ordinary canonical Arrow-backed pandas
StringDtype (exact ArrowStringArray with authenticated PyArrow chunks) and an
explicit native all-StringType / UTF8_BINARY schema. Reuse the existing
internal inline IPC transport without Python row materialization. Preserve
field names, logical schema/metadata, nullable/missing semantics, offsets,
chunks/slices and standard UTF8 representation. Guard customized
DataFrame/columns/conversion/schema/storage hooks before touching data; Python
StringArray, generic ArrowDtype, mixed and unsupported inputs retain the old
path.
This is a distinct extension-storage follow-up to PR144, not a widening of its
frozen object-string measurements. SPARK-55294 concerns pandas-on-Spark type
restoration and does not implement this Rust-client input path. Bounded
upstream PR/JIRA searches and a source-only assessment are preserved in the
campaign's pandas-arrow-string-feasibility directory.
Validate canonical storage identities explicitly on pandas 2.3.3 and 3.1.0.
Cover pd.NA/NaN, nonnullable missing values, customized aliases/hooks and error
order, chunks/slices, string versus large_string conversion and
overflow/fallback, Unicode and inline transport/RPC behavior. Preserve
unchanged frozen workloads and validators, compare separate exact
baseline/candidate wheels/environments, and report supported gains plus guard,
cold/first-construction, fallback, memory and RPC costs. Do not route through
the public Arrow importer if that introduces Config calls or cached uploads. No
JVM/live Spark is required. Publish only a separately reviewed draft with
meaningful measured gain; otherwise preserve local results.
The campaign hard cutoff is 2026-10-30T14:00:00Z (07:00 America/Los_Angeles);
all execution must stop by 13:50 UTC.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
