[
https://issues.apache.org/jira/browse/SPARK-61099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61099.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 382
[https://github.com/apache/spark-connect-rust/pull/382]
> Preserve binaryAsBytes configuration timing and terminate failed iterators
> --------------------------------------------------------------------------
>
> Key: SPARK-61099
> URL: https://issues.apache.org/jira/browse/SPARK-61099
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Initial binary result conversion landed in SPARK-61127 (PR #320). This
> follow-up implements configuration timing, Arrow-only result conversion, and
> iterator failure cleanup in
> https://github.com/apache/spark-connect-rust/pull/382.
> Eager actions read spark.sql.execution.pyspark.binaryAsBytes after execution
> and Arrow-schema conversion, matching PySpark 4.3.0. toLocalIterator()
> snapshots the setting on the first next() before execution begins and reuses
> it for later rows. The setting also applies when the server supplies only an
> Arrow schema.
> A configuration or converter-initialization failure terminates the iterator.
> Closing an incomplete iterator releases its execution while the native
> prefetch task is waiting for another response. Direct binary map keys
> preserve PySpark's TypeError when bytearray keys are unhashable.
> Regression coverage includes eager aliases, both prefetch modes, absent and
> present Spark schemas, empty/null/nested results, configuration failures,
> execution ordering and operation-matched execution release. Binary-backed UDT
> map-key parity is outside this patch; current-build behavior is not
> established by these checks. The branch includes the NumPy scalar-class
> prerequisite tracked in SPARK-61102 and preserves connect-rust-4.3.0 release
> tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
