chitralverma opened a new pull request, #7824: URL: https://github.com/apache/opendal/pull/7824
# Which issue does this PR close? Closes #. # Rationale for this change The Python binding generates its type stubs with `pyo3-stub-gen`. This has a few issues: - `just stub-gen` has not worked on `main` for months. `pyo3-stub-gen` 0.18 changed the binding's module layout (in a stale, unmerged PR), later bumps moved it to 0.23, and the generator was never re-run. The committed stubs went stale, so the break stayed hidden. Regenerating now fails with a wrong-module error. - It relies on `#[gen_stub_*]` macros and `override_type` / `override_return_type` attributes spread across the Rust code, kept in sync by hand. - It ships an extra `stub_gen` binary and a crate dependency. PyO3 0.29 can describe the extension through its `experimental-inspect` feature, and `maturin --generate-stubs` turns that into `.pyi` files. This is the direction PyO3 is moving in. It removes the extra dependency, the binary, and the hand-maintained macros, and derives the stubs straight from the built binary. # What changes are included in this PR? **Switch the generator** - Drop `pyo3-stub-gen`, the `stub_gen` binary, and `define_stub_info_gatherer!`. Enable `pyo3/experimental-inspect`. The lib is `cdylib`-only now (the `rlib` was only for the deleted binary). - Remove the `#[gen_stub_*]` attributes and convert the `override_*` attributes into inline `#[pyo3(signature = (arg: "T") -> "R")]` annotations. - Update the codegen template to emit a lean `Scheme` enum; regenerate `src/services.rs` (−5.3k lines). **Generate stubs at the public paths** - `maturin --generate-stubs` writes under `opendal/_opendal/`. `scripts/postprocess_stubs.py` moves them to `opendal/<name>.pyi`, where type checkers resolve `from opendal.operator import …`. - Wire `just stub-gen` and run ruff's fixers in `just fmt`. **Guard against drift** - Add a `stubs` CI job that runs `just stub-gen` and `git diff --exit-code`, so stale stubs fail CI. Release wheels ship the committed `python/opendal` tree as-is, so it must stay fresh. # Are there any user-facing changes? Yes. The changes are to the generated stubs only; runtime behavior is unchanged. **Pros** - **Stubs derive from the built binary**, so they stay aligned with the Rust source and the runtime types, and cannot silently drift. - **Stub generation is fast** — it reads the compiled binary instead of running a separate generator pass. - **Richer, more consistent stubs**: enum members (`EntryMode.*`, `Scheme.*`) and several dunders are now described, and `Metadata.last_modified` is correctly `datetime | None` (was `datetime`, which could not represent `None`). - **Less code to maintain**: no `#[gen_stub_*]` macros, no `override_*` attributes, no `stub_gen` binary, one fewer dependency. - **Standard tooling**: generation goes through `maturin` and PyO3 directly, the future-facing path as `experimental-inspect` matures. - **Freshness is enforced** by the new CI drift check, so the published stubs always match the code. **Cons** - The per-service typed constructors are gone. The old stubs had 104 `@overload def __new__(...)` entries (one per service, with a typed `scheme=Literal[...]` and per-service config kwargs); they are now a single generic `__new__(scheme: str | Scheme, **kwargs)`. PyO3 introspection has no overload support. A follow-up will regenerate these from Rust. - `scripts/postprocess_stubs.py` applies two temporary fixups: it injects the imports PyO3 omits for forward-ref annotations, and it substitutes a hand-written `exceptions.pyi` while the generated one is incomplete (`create_exception!` types are not introspectable). These are PyO3 limitations and should be removed as `experimental-inspect` improves in future releases. Please add the `breaking-changes` label for the dropped constructor overloads. # AI Usage Statement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
