[
https://issues.apache.org/jira/browse/SPARK-58983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-58983:
---------------------------------
Description:
All GitHub Actions workflows on apache/spark-connect-rust failed with
"startup_failure"
("This run likely failed because of a workflow file issue") on pushes to master
and on
pull requests, so there was effectively no CI. Fixing that surfaced several
genuine
build/test failures that the default build never exercised. This issue tracks
getting CI
to both run *and* pass.
h2. Root cause of startup_failure
The ASF org restricts GitHub Actions to actions/* plus a vetted allowlist; the
workflows
referenced non-allowlisted third-party actions
(actions-rust-lang/setup-rust-toolchain,
dtolnay/rust-toolchain, arduino/setup-protoc, PyO3/maturin-action,
pypa/gh-action-pypi-publish), so the runs failed to start.
h2. Fixes
* Replace all non-allowlisted actions with GitHub-authored actions
(actions/checkout,
actions/setup-python, actions/setup-java, actions/cache,
actions/upload-artifact,
actions/download-artifact) plus shell steps (rustup for the toolchain,
apt/brew/choco
for protoc, pip for maturin/twine).
* Fix a pre-existing crate-name bug in build_python_connect.yml's rust-checks
job, which
ran {{cargo test -p spark-connect -p spark-connect-core}} using the old crate
names
(renamed to apache-spark-connect / apache-spark-connect-core).
* Fix a wasm-udf compile error: a unit test named parse_python_version_output
shadowed the
module function of the same name (via {{use super::*}}); rename the test.
* Run the WASM packer against the reference pyspark: isolate pyspark_wasm_udf
on the packer
import path so {{pyspark.cloudpickle}} and
{{pyspark.sql.types._parse_datatype_json_value}}
resolve to the reference pyspark. The Rust-backed pyspark skin lacks them and
cannot emit
an executor-compatible DataType.
* Bump the transitive dependency ethnum 1.5.0 -> 1.5.3: 1.5.0 does
{{mem::transmute::<(), TryFromIntError>(())}}, which no longer compiles now
that std's
TryFromIntError is not zero-sized (E0512).
* Move the example crates and datasets under examples/ and update the workflow
paths.
was:
All GitHub Actions workflows on apache/spark-connect-rust fail with
"startup_failure"
("This run likely failed because of a workflow file issue") on pushes to master
and on
pull requests, so there is effectively no CI.
Root cause: the workflows use third-party GitHub Actions that are not on the
ASF Actions
allowlist (the ASF org restricts Actions to actions/* plus a vetted allowlist),
so the
runs fail to start:
- actions-rust-lang/setup-rust-toolchain (rust.yml, lint.yml,
build_python_connect.yml)
- dtolnay/rust-toolchain, arduino/setup-protoc, PyO3/maturin-action,
pypa/gh-action-pypi-publish (release.yml)
Fix: replace all non-allowlisted actions with GitHub-authored actions
(actions/checkout,
actions/setup-python, actions/setup-java, actions/cache,
actions/upload-artifact,
actions/download-artifact) plus shell steps (rustup for the toolchain,
apt/brew/choco for
protoc, pip for maturin/twine). Also fix a pre-existing bug in
build_python_connect.yml's
rust-checks job, which ran `cargo test -p spark-connect -p spark-connect-core`
using the
old crate names (renamed to apache-spark-connect / apache-spark-connect-core).
Summary: [spark-connect-rust] Fix CI so all workflows run and pass
(was: [spark-connect-rust] CI workflows fail to start: replace non-allowlisted
GitHub Actions)
> [spark-connect-rust] Fix CI so all workflows run and pass
> ---------------------------------------------------------
>
> Key: SPARK-58983
> URL: https://issues.apache.org/jira/browse/SPARK-58983
> Project: Spark
> Issue Type: Bug
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> All GitHub Actions workflows on apache/spark-connect-rust failed with
> "startup_failure"
> ("This run likely failed because of a workflow file issue") on pushes to
> master and on
> pull requests, so there was effectively no CI. Fixing that surfaced several
> genuine
> build/test failures that the default build never exercised. This issue tracks
> getting CI
> to both run *and* pass.
> h2. Root cause of startup_failure
> The ASF org restricts GitHub Actions to actions/* plus a vetted allowlist;
> the workflows
> referenced non-allowlisted third-party actions
> (actions-rust-lang/setup-rust-toolchain,
> dtolnay/rust-toolchain, arduino/setup-protoc, PyO3/maturin-action,
> pypa/gh-action-pypi-publish), so the runs failed to start.
> h2. Fixes
> * Replace all non-allowlisted actions with GitHub-authored actions
> (actions/checkout,
> actions/setup-python, actions/setup-java, actions/cache,
> actions/upload-artifact,
> actions/download-artifact) plus shell steps (rustup for the toolchain,
> apt/brew/choco
> for protoc, pip for maturin/twine).
> * Fix a pre-existing crate-name bug in build_python_connect.yml's rust-checks
> job, which
> ran {{cargo test -p spark-connect -p spark-connect-core}} using the old
> crate names
> (renamed to apache-spark-connect / apache-spark-connect-core).
> * Fix a wasm-udf compile error: a unit test named parse_python_version_output
> shadowed the
> module function of the same name (via {{use super::*}}); rename the test.
> * Run the WASM packer against the reference pyspark: isolate pyspark_wasm_udf
> on the packer
> import path so {{pyspark.cloudpickle}} and
> {{pyspark.sql.types._parse_datatype_json_value}}
> resolve to the reference pyspark. The Rust-backed pyspark skin lacks them
> and cannot emit
> an executor-compatible DataType.
> * Bump the transitive dependency ethnum 1.5.0 -> 1.5.3: 1.5.0 does
> {{mem::transmute::<(), TryFromIntError>(())}}, which no longer compiles now
> that std's
> TryFromIntError is not zero-sized (E0512).
> * Move the example crates and datasets under examples/ and update the
> workflow paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
