DB Tsai created SPARK-61215:
-------------------------------
Summary: Match streaming listener parsing and failure handling in
Spark Connect Rust
Key: SPARK-61215
URL: https://issues.apache.org/jira/browse/SPARK-61215
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Malformed streaming listener events can disappear or leave registrations
active, and terminal transport failures can clear listeners without PySpark's
handler warning.
Propagate synchronous start-event decoding errors from start() and toTable(),
including with no registered listeners. Decode each asynchronous event once and
share it across all registrations; report transport, malformed-event and
unknown-event failures once before clearing the affected subscription.
Per-listener Exception warnings continue dispatch. Native Rust listeners retain
a silent default error hook and can override it. Callbacks and warning hooks
hold no listener-bus state lock; concurrent non-last removal waits through
event/error notification.
The independent patch is based on master
925dc876f63f1f779fe176f62575a0aa96cc6886 and preserves its existing lifecycle
and generation-scoped removal semantics.
Executed validation: formatting, release library tests, integration-test
compilation with and without wasm-udf, extension build, 2605 Python passes with
8 inherited live-server skips, all 53 required listener cases and five new
native unit tests. The current source carries that execution through exactly
three comment-only replacements. Fresh strict validation on the current
artifact passed 25 distinct Python cases once each (75 passing
setup/call/teardown reports) plus three selected native error regression unit
cases. Live Spark validation passed 7 native Python cases with no skips and all
three native streaming e2e callers with 1 passed / 0 ignored each. The live
selection deselected 194 other cases and used staging-harness provenance.
The 12-group API public-name and Row tuple/name gates and no-stubs audit
passed. Rustdoc exited 0 with two inherited types.rs warnings (unresolved
from_json link and unclosed int tag). Workspace/all-target Clippy exited 101;
all 472 source diagnostics and 721 spans map to unchanged saved-master
statements and preserved current statements. Complete source review round 12
reported no findings. These results are scoped and do not establish full
API/client parity or certify the README's broad parity statement.
Retain the documented limits: normal EOF silently clears registrations;
uncaught asynchronous BaseException or warning-as-error clears registrations,
with best-effort terminal warning; local stop()/close() cancels and clears
without removal RPCs or joining active callbacks. A listener can retain its
session through listener.spark; initial RPC establishment retains transport
deadlines/retries. The inherited single-generation dispatch-marker boundary is
unchanged.
This artifact's official configuration and DataFrame-creation modules ran
through its raw-transport plugin against a live JVM: 23 passed, 1 skipped and
no failures/errors. Its current source, saved extension and reviewed source
match.
Assessed common transport regression evidence is carried from a separate pandas
artifact with all 53 shared transport/build/dependency inputs equal. The
donor's full ordinary phase covered 87 modules: 2586 passed, 641 skipped, 1
manifest deselection and one expected missing-catalog drift-check failure, with
no unexpected failures. Its separate cluster/resource phase covered 4 modules:
15 passed and 14 skipped (10 functional passes and 5 guard passes). Only the
donor executed the full phases; this streaming artifact independently executed
the plugin smoke and native checks described above.
Saved harness/source/artifact/endpoint evidence is retained, but complete
official/dependency whole-tree start/end attestations, per-case skip provenance
and raw per-module pytest exit codes were not all recorded. Totals count test
executions; no whole-client parity claim is made.
This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
