[
https://issues.apache.org/jira/browse/SPARK-58983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-58983:
---------------------------------
Epic Link: (was: SPARK-59035)
> [spark-connect-rust] Fix CI so all workflows run and pass
> ---------------------------------------------------------
>
> Key: SPARK-58983
> URL: https://issues.apache.org/jira/browse/SPARK-58983
> Project: Spark
> Issue Type: Bug
> Components: Project Infra
> Affects Versions: connect-rust-0.1.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.2.0
>
>
> All GitHub Actions workflows on apache/spark-connect-rust failed with
> "startup_failure"
> ("This run likely failed because of a workflow file issue") on pushes to
> master and on
> pull requests, so there was effectively no CI. Fixing that surfaced several
> genuine
> build/test failures that the default build never exercised. This issue tracks
> getting CI
> to both run *and* pass.
> h2. Root cause of startup_failure
> The ASF org restricts GitHub Actions to actions/* plus a vetted allowlist;
> the workflows
> referenced non-allowlisted third-party actions
> (actions-rust-lang/setup-rust-toolchain,
> dtolnay/rust-toolchain, arduino/setup-protoc, PyO3/maturin-action,
> pypa/gh-action-pypi-publish), so the runs failed to start.
> h2. Fixes
> * Replace all non-allowlisted actions with GitHub-authored actions
> (actions/checkout,
> actions/setup-python, actions/setup-java, actions/cache,
> actions/upload-artifact,
> actions/download-artifact) plus shell steps (rustup for the toolchain,
> apt/brew/choco
> for protoc, pip for maturin/twine).
> * Fix a pre-existing crate-name bug in build_python_connect.yml's rust-checks
> job, which
> ran {{cargo test -p spark-connect -p spark-connect-core}} using the old
> crate names
> (renamed to apache-spark-connect / apache-spark-connect-core).
> * Fix a wasm-udf compile error: a unit test named parse_python_version_output
> shadowed the
> module function of the same name (via {{use super::*}}); rename the test.
> * Run the WASM packer against the reference pyspark: isolate pyspark_wasm_udf
> on the packer
> import path so {{pyspark.cloudpickle}} and
> {{pyspark.sql.types._parse_datatype_json_value}}
> resolve to the reference pyspark. The Rust-backed pyspark skin lacks them
> and cannot emit
> an executor-compatible DataType.
> * Bump the transitive dependency ethnum 1.5.0 -> 1.5.3: 1.5.0 does
> {{mem::transmute::<(), TryFromIntError>(())}}, which no longer compiles now
> that std's
> TryFromIntError is not zero-sized (E0512).
> * Move the example crates and datasets under examples/ and update the
> workflow paths.
> * Fix a transport-injection stub hang that timed out the Connect parity gate
> on test_client.py: test_server_unreachable (client to unresolvable sc://foo
> with max_retries=0) expected a prompt UNAVAILABLE, but our RustConnectStub
> ran RPCs through the Rust client's with_retry using RetryPolicy::default()
> (15 retries, backoff to 64s). The injection stub is a drop-in for the
> single-shot grpcio stub over which the reference client layers its own retry
> handler, so retrying inside our client double-retried and ignored the client
> policy (~700s hang). Add RetryPolicy::no_retries() and set it on the
> injection stub (single-shot; reference client owns retrying), and bound
> grpc.ready()/add connect_timeout so an unreachable channel surfaces
> UNAVAILABLE instead of hanging. test_client.py now passes 41/41 through our
> transport.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
