[
https://issues.apache.org/jira/browse/SPARK-61215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61215:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Match streaming listener parsing and failure handling in Spark Connect Rust
> ---------------------------------------------------------------------------
>
> Key: SPARK-61215
> URL: https://issues.apache.org/jira/browse/SPARK-61215
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Malformed streaming listener events can disappear or leave registrations
> active, and terminal transport failures can clear listeners without PySpark's
> handler warning.
> Propagate synchronous start-event decoding errors from start() and toTable(),
> including with no registered listeners. Decode each asynchronous event once
> and share it across all registrations; report transport, malformed-event and
> unknown-event failures once before clearing the affected subscription.
> Per-listener Exception warnings continue dispatch. Native Rust listeners
> retain a silent default error hook and can override it. Callbacks and warning
> hooks hold no listener-bus state lock; concurrent non-last removal waits
> through event/error notification.
> The independent patch is based on master
> 925dc876f63f1f779fe176f62575a0aa96cc6886 and preserves its existing lifecycle
> and generation-scoped removal semantics.
> Executed validation: formatting, release library tests, integration-test
> compilation with and without wasm-udf, extension build, 2605 Python passes
> with 8 inherited live-server skips, all 53 required listener cases and five
> new native unit tests. The current source carries that execution through
> exactly three comment-only replacements. Fresh strict validation on the
> current artifact passed 25 distinct Python cases once each (75 passing
> setup/call/teardown reports) plus three selected native error regression unit
> cases. Live Spark validation passed 7 native Python cases with no skips and
> all three native streaming e2e callers with 1 passed / 0 ignored each. The
> live selection deselected 194 other cases and used staging-harness provenance.
> The 12-group API public-name and Row tuple/name gates and no-stubs audit
> passed. Rustdoc exited 0 with two inherited types.rs warnings (unresolved
> from_json link and unclosed int tag). Workspace/all-target Clippy exited 101;
> all 472 source diagnostics and 721 spans map to unchanged saved-master
> statements and preserved current statements. Complete source review round 12
> reported no findings. These results are scoped and do not establish full
> API/client parity or certify the README's broad parity statement.
> Retain the documented limits: normal EOF silently clears registrations;
> uncaught asynchronous BaseException or warning-as-error clears registrations,
> with best-effort terminal warning; local stop()/close() cancels and clears
> without removal RPCs or joining active callbacks. A listener can retain its
> session through listener.spark; initial RPC establishment retains transport
> deadlines/retries. The inherited single-generation dispatch-marker boundary
> is unchanged.
> This artifact's official configuration and DataFrame-creation modules ran
> through its raw-transport plugin against a live JVM: 23 passed, 1 skipped and
> no failures/errors. Its current source, saved extension and reviewed source
> match.
> Assessed common transport regression evidence is carried from a separate
> pandas artifact with all 53 shared transport/build/dependency inputs equal.
> The donor's full ordinary phase covered 87 modules: 2586 passed, 641 skipped,
> 1 manifest deselection and one expected missing-catalog drift-check failure,
> with no unexpected failures. Its separate cluster/resource phase covered 4
> modules: 15 passed and 14 skipped (10 functional passes and 5 guard passes).
> Only the donor executed the full phases; this streaming artifact
> independently executed the plugin smoke and native checks described above.
> Saved harness/source/artifact/endpoint evidence is retained, but complete
> official/dependency whole-tree start/end attestations, per-case skip
> provenance and raw per-module pytest exit codes were not all recorded. Totals
> count test executions; no whole-client parity claim is made.
> This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
