[
https://issues.apache.org/jira/browse/SPARK-61099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61099:
----------------------------
Description:
The main binaryAsBytes support landed in SPARK-61127 (PR #320, commit5ecd27e).
This ticket now tracks the remaining result-configuration behavior on current
upstream.
PySpark reads spark.sql.execution.pyspark.binaryAsBytes after an eager action
and before converting its result. Its lazy toLocalIterator generator reads the
setting on the first next(), before executing the query, and retains the value
for that iterator. The native iterator currently fetches its first row before
reading the setting; a config or converter initialization error can consume
that row and allow later next() calls to continue. The setting is also skipped
when the server omits a separate Spark schema, although PySpark infers the
result schema from Arrow and honors the setting.
The follow-up reads the setting independently of the optional Spark schema,
preserves eager versus lazy timing, and terminates/releases a failed iterator.
Direct Binary map keys continue to raise TypeError in false mode, as official
PySpark does. Binary-backed UDT map-key conversion is a separate issue and is
not claimed fixed here.
Regression cases cover eager aliases, both iterator prefetch modes,
omitted/present schema, empty/null/nested values, default/case/None handling,
config and converter failures, execution ordering, release of failed
executions, and GIL release. Native validation and independent patch review are
pending; no final pass is claimed yet.
was:
Set spark.sql.execution.pyspark.binaryAsBytes to false, then collect SELECT
X'0102' AS b or iterate it with toLocalIterator(). Official PySpark returns
bytearray; the Rust-backed Python client always constructs bytes. Honor the
setting for top-level and nested binary result values and preserve user-defined
type conversion. Cover collection, lazy iteration and both settings with
regression tests.
Observed by source inspection of public apache/spark-connect-rust master
83d74462ec41012993176cf3af99017576688eb8. Runtime regression tests will
accompany the fix.
Summary: Preserve binaryAsBytes configuration timing and iterator
failure semantics (was: Honor binaryAsBytes when collecting rows)
> Preserve binaryAsBytes configuration timing and iterator failure semantics
> --------------------------------------------------------------------------
>
> Key: SPARK-61099
> URL: https://issues.apache.org/jira/browse/SPARK-61099
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The main binaryAsBytes support landed in SPARK-61127 (PR #320,
> commit5ecd27e). This ticket now tracks the remaining result-configuration
> behavior on current upstream.
> PySpark reads spark.sql.execution.pyspark.binaryAsBytes after an eager action
> and before converting its result. Its lazy toLocalIterator generator reads
> the setting on the first next(), before executing the query, and retains the
> value for that iterator. The native iterator currently fetches its first row
> before reading the setting; a config or converter initialization error can
> consume that row and allow later next() calls to continue. The setting is
> also skipped when the server omits a separate Spark schema, although PySpark
> infers the result schema from Arrow and honors the setting.
> The follow-up reads the setting independently of the optional Spark schema,
> preserves eager versus lazy timing, and terminates/releases a failed
> iterator. Direct Binary map keys continue to raise TypeError in false mode,
> as official PySpark does. Binary-backed UDT map-key conversion is a separate
> issue and is not claimed fixed here.
> Regression cases cover eager aliases, both iterator prefetch modes,
> omitted/present schema, empty/null/nested values, default/case/None handling,
> config and converter failures, execution ordering, release of failed
> executions, and GIL release. Native validation and independent patch review
> are pending; no final pass is claimed yet.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
