voidstackloop opened a new pull request, #1736:
URL: https://github.com/apache/datafusion-python/pull/1736
# Which issue does this PR close?
Closes #1728.
# Rationale for this change
Nothing in CI runs the top-level `examples/*.py` scripts, so they silently
drift
from the API and from each other. Two consequences are already in the tree:
- `examples/csv-read-options.py` crashes: it reads `data.csv` and
`data.csv.gz`, neither of which exists in the repository.
- Nine examples end in `assert` with no output, so a reader cannot tell a
working example from a script that does nothing.
Adding a CI job is what keeps this from recurring.
# What changes are included in this PR?
- `examples/csv-read-options.py` now writes its own small CSV and gzipped CSV
into a temporary directory, so it is self-contained and runs without
external
data.
- The nine silent examples (`export.py`, `import.py`, `python-udaf.py`,
`python-udf.py`, `query-pyarrow-data.py`, `sql-to-pandas.py`,
`sql-using-python-udaf.py`, `sql-using-python-udf.py`, `substrait.py`) now
print their result, keeping their existing assertions.
- `.github/workflows/test.yml` gains a **Run example scripts** step (Python
3.12
entry) that runs every `examples/*.py` against the built wheel. It installs
the extra runtime dependencies some examples use (`pandas`, `polars`,
`matplotlib`) and skips an explicit list of examples that cannot run in CI:
- `sql-parquet-s3.py` — needs network access and AWS credentials
- `sql-parquet.py`, `dataframe-parquet.py`, `sql-to-pandas.py` — need the
NYC
taxi parquet file documented in `examples/README.md`
- `python-udf-comparisons.py` — needs the TPC-H dataset generated later in
the
workflow
- `ray_pickle_expr.py` — needs the optional (heavy) `ray` dependency
# Are there any user-facing changes?
No API changes. The examples are documentation; they are now runnable, print
their results, and CI will catch it if they break again.
# Verification
- Ran every non-skipped example against a `datafusion==54.0.0` wheel in a
fresh
venv: all pass. `multiprocessing_pickle_expr.py` and `ray_pickle_expr.py`
were
not run locally (Windows has no `forkserver` and Ray is unavailable there);
they run on the Linux CI.
- `ruff check` / `ruff format --check` (pinned 0.15.1), `codespell` and
`actionlint` (v1.7.12) pass on the changed files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
