[
https://issues.apache.org/jira/browse/SPARK-61099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140754#comment-18140754
]
DB Tsai commented on SPARK-61099:
---------------------------------
Implementation and validation:
https://github.com/apache/spark-connect-rust/pull/382.
The final patch passed the offline Python suites (2,253 passed; eight explicit
live-server skips), Rust core/API unit tests, formatting and integration-test
compilation with and without the WASM feature. Five selected native live cases
and all 15 setup/call/teardown reports passed without selected skips or
failures, with the actual imported extension checked before and after. The same
three mock-RPC configuration-ordering cases passed 100 rounds, totaling 300
passing executions with no failures, errors or skips. These are three distinct
ordering cases, not 300 distinct cases.
The scoped official raw-transport comparison recorded 85 passing outcomes with
34 skips, including all 12 mixed Classic/Connect collection cases. Broader full
and cluster raw-transport coverage was accepted through equality of the
selected transport source files with the NumPy prerequisite: its executed
suites recorded 2,586 passing outcomes with 641 skips and one deselection, and
15 passing invocations with 14 skips, respectively. Those broader suites
executed the prerequisite's native binary; this branch's native behavior is
covered by its separate checks. The official Python converters used by the
transport suites do not establish complete native converter compatibility.
Clippy remains an advisory failure (exit 101). The prerequisite's separate
catalog-plugin drift failure and earlier failed attempts remain recorded. Linux
and Windows were not run locally. Binary-backed UDT map-key conversion remains
outside this fix.
Prepared by OpenAI Codex.
> Preserve binaryAsBytes configuration timing and terminate failed iterators
> --------------------------------------------------------------------------
>
> Key: SPARK-61099
> URL: https://issues.apache.org/jira/browse/SPARK-61099
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Initial binary result conversion landed in SPARK-61127 (PR #320). This
> follow-up implements configuration timing, Arrow-only result conversion, and
> iterator failure cleanup in
> https://github.com/apache/spark-connect-rust/pull/382.
> Eager actions read spark.sql.execution.pyspark.binaryAsBytes after execution
> and Arrow-schema conversion, matching PySpark 4.3.0. toLocalIterator()
> snapshots the setting on the first next() before execution begins and reuses
> it for later rows. The setting also applies when the server supplies only an
> Arrow schema.
> A configuration or converter-initialization failure terminates the iterator.
> Closing an incomplete iterator releases its execution while the native
> prefetch task is waiting for another response. Direct binary map keys
> preserve PySpark's TypeError when bytearray keys are unhashable.
> Regression coverage includes eager aliases, both prefetch modes, absent and
> present Spark schemas, empty/null/nested results, configuration failures,
> execution ordering and operation-matched execution release. Binary-backed UDT
> map-key parity is outside this patch; current-build behavior is not
> established by these checks. The branch includes the NumPy scalar-class
> prerequisite tracked in SPARK-61102 and preserves connect-rust-4.3.0 release
> tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
