Developer1010x opened a new pull request, #1732:
URL: https://github.com/apache/datafusion-python/pull/1732
# Which issue does this PR close?
Closes #1728.
# Rationale for this change
Nothing in CI ran the top-level `examples/*.py`, so nothing noticed when they
stopped working. Two consequences were already in the tree, and both are the
first thing a new user copies from.
# What changes are included in this PR?
**`examples/csv-read-options.py` no longer crashes.** It read `data.csv` and
`data.csv.gz`, neither of which is in the repository, so it raised on the
first
read. It now writes what it needs into a temporary directory: a
comma-separated
CSV, a pipe-separated and single-quoted one so the builder-pattern example
actually exercises the delimiter and quote it sets, and a gzipped one with a
comment line so the advanced-options example exercises `with_comment` and
`with_file_compression_type`. Every example now calls `df.show()`, so the
script
demonstrates the options instead of only naming them.
**Nine scripts printed nothing.** `export.py`, `import.py`, `python-udaf.py`,
`python-udf.py`, `query-pyarrow-data.py`, `sql-to-pandas.py`,
`sql-using-python-udaf.py`, `sql-using-python-udf.py` and `substrait.py`
ended
in a bare `assert`. They now print their results; the asserts are unchanged.
**`substrait.py` only worked from the repository root.** It read
`./testing/data/csv/aggregate_test_100.csv`, so it failed from anywhere
else. It
now resolves that path relative to `__file__`.
**The CI job.** A `Run Python examples` step, gated to `abi3` like the FFI
and
TPC-H steps, running from the repository root against the wheel built
earlier in
the job. Placement and content are both load-bearing:
- It runs **after** the TPC-H generation step, because
`python-udf-comparisons.py` reads `examples/tpch/data/lineitem.parquet`.
The
issue did not list this one; it shows up as soon as you actually run them.
- It installs `pandas` and `polars`, which are neither runtime nor dev
dependencies but are what `import.py` and `export.py` demonstrate
converting
to and from.
- It relies on the `git submodule update --init` already done by the test
step
for `substrait.py`.
- It skips five, each for a stated reason in the workflow: `sql-parquet.py`,
`dataframe-parquet.py` and `sql-to-pandas.py` need the NYC taxi Parquet
file
the README says to download by hand; `ray_pickle_expr.py` needs a Ray
cluster;
`sql-parquet-s3.py` needs network access and AWS credentials.
- A failing example does not stop the loop. All of them run and the step
reports
every failure at once, so one broken script does not hide the next.
**`examples/README.md`** gains a short "Running the examples" section with a
table of the per-example prerequisites, and drops the links to
`sql-on-polars.py`, `sql-on-pandas.py` and `sql-on-cudf.py`, which are not in
the repository.
## Verification
All 14 non-skipped examples were run from the repository root against a
`datafusion` 54.0.0 wheel, with the TPC-H data generated and the `testing`
submodule initialised. Every one exits 0 and prints output. `ruff check`,
`ruff format --check` and `codespell` pass, and `shellcheck` is clean on the
new
step's script.
## One thing found on the way, not fixed here
`CsvReadOptions.with_null_regex` appears to have no effect at read time. With
`datafusion` 54.0.0:
```python
from pathlib import Path
from datafusion import CsvReadOptions, SessionContext
p = Path("probe.csv")
p.write_text("id,name,value\n1,alice,10\n2,N/A,20\n3,carol,30\n")
ctx = SessionContext()
options =
CsvReadOptions().with_has_header(True).with_null_regex(r"^(null|NULL|N/A)$")
ctx.read_csv(p, options=options).show()
```
`N/A` comes back as the literal string rather than NULL, and if the same
value
sits in a column inferred as `Int64` the read fails outright with
`Parser error: Error while parsing value 'N/A' as type 'Int64'`. The option
is
plumbed through `crates/core/src/options.rs` into DataFusion's
`CsvReadOptions`,
so this looks like it is below this crate. To keep this PR runnable and
honest,
the advanced-options example keeps `with_null_regex` set and puts the `N/A`
in a
string column. Happy to open a separate issue for it.
# Are there any user-facing changes?
No API changes. The examples print their results now, and
`examples/csv-read-options.py` no longer requires data files that were never
shipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
