Michael-J-Ward commented on PR #694: URL: https://github.com/apache/datafusion-python/pull/694#issuecomment-2110283449
Note for reproducing: make sure to remove the `.*so` file that `maturin develop` places next to `./datafusion/__init__.py` ------ @samuelcolvin, I assume this is an instance of this issue that the [maturin docs](https://www.maturin.rs/project_layout#mixed-rustpython-project) warns about: > This structure is recommended to avoid [a common ImportError pitfall](https://github.com/PyO3/maturin/issues/490) The docs recommend nesting the python source as `python/datafusion/`, and the only references I find about `lib.name` recommend keeping it the same. > To create a mixed Rust/Python project, add a directory with your package name (i.e. matching lib.name in your Cargo.toml) to contain the Python source: Most importantly, my first attempt at using your fix *did not* appear to fix it (see below) Do you have any thoughts on this solution vs the recommended one? (I'm still learning about maturin). ------ ```console ❯ maturin build ⚠ Warning: You specified maturin >=0.15, <0.16 in pyproject.toml under `build-system.requires`, but the current maturin version is 1.4.0 🍹 Building a mixed python/rust project 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8 🐍 Not using a specific python interpreter 📡 Using build options features, locked from pyproject.toml Compiling pyo3-build-config v0.20.2 Compiling pyo3-ffi v0.20.2 Compiling pyo3 v0.20.2 Compiling datafusion-python v37.1.0 (/home/mike/workspace/datafusion-python/flake) Compiling arrow v51.0.0 Compiling datafusion-common v37.1.0 Compiling datafusion-expr v37.1.0 Compiling datafusion-execution v37.1.0 Compiling datafusion-sql v37.1.0 Compiling datafusion-physical-expr v37.1.0 Compiling datafusion-functions v37.1.0 Compiling datafusion-optimizer v37.1.0 Compiling datafusion-physical-plan v37.1.0 Compiling datafusion-functions-array v37.1.0 Compiling datafusion v37.1.0 Compiling datafusion-substrait v37.1.0 Finished dev [unoptimized + debuginfo] target(s) in 1m 02s ⚠ Warning: No compatible platform tag found, using the linux tag instead. You won't be able to upload those wheels to PyPI. 📦 Built wheel for abi3 Python ≥ 3.8 to /home/mike/workspace/datafusion-python/flake/target/wheels/datafusion-37.1.0-cp38-abi3-linux_x86_64.whl flake on flake [$!?⇕] is 📦 v37.1.0 via 🐍 v3.11.9 (venv) via 🦀 v1.77.1 via ❄ impure (nix-shell-env) on ☁ (us-east-1) took 1m27s ❯ pip install . Processing /home/mike/workspace/datafusion-python/flake Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: pyarrow>=11.0.0 in /nix/store/llp7z339ix0lm7nlrndn1yvdmhskcsyk-rust-toolchain/lib/python3.11/site-packages (from datafusion==37.1.0) (15.0.0) Requirement already satisfied: numpy<2,>=1.16.6 in /nix/store/llp7z339ix0lm7nlrndn1yvdmhskcsyk-rust-toolchain/lib/python3.11/site-packages (from pyarrow>=11.0.0->datafusion==37.1.0) (1.26.4) Building wheels for collected packages: datafusion Building wheel for datafusion (pyproject.toml) ... done Created wheel for datafusion: filename=datafusion-37.1.0-cp38-abi3-linux_x86_64.whl size=17875320 sha256=b417c0d9b4662a0f687359a2e751580d4a2fc0714a02b447cfe6deff11138413 Stored in directory: /tmp/nix-shell.EXRUlg/pip-ephem-wheel-cache-tbyz75qv/wheels/8e/1b/87/2a5f750e961aa47403189d62f5e1236e0819f25e0a9068babf Successfully built datafusion Installing collected packages: datafusion Successfully installed datafusion-37.1.0 flake on flake [$!?⇕] is 📦 v37.1.0 via 🐍 v3.11.9 (venv) via 🦀 v1.77.1 via ❄ impure (nix-shell-env) on ☁ (us-east-1) took 9m50s ❯ python Python 3.11.9 (main, Apr 2 2024, 08:25:04) [GCC 13.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import datafusion Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/mike/workspace/datafusion-python/flake/datafusion/__init__.py", line 28, in <module> from ._internal import ( ModuleNotFoundError: No module named 'datafusion._internal' ``` w/ a git diff of ```console ❯ git diff diff --git a/Cargo.toml b/Cargo.toml index 9da36d7..5bb1f83 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -60,7 +60,7 @@ url = "2.2" pyo3-build-config = "0.20.0" [lib] -name = "datafusion_python" +name = "_internal" crate-type = ["cdylib", "rlib"] [profile.release] diff --git a/src/lib.rs b/src/lib.rs index a696ebf..8ce1b16 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -72,6 +72,7 @@ pub(crate) struct TokioRuntime(tokio::runtime::Runtime); /// The higher-level public API is defined in pure python files under the /// datafusion directory. #[pymodule] +#[pyo3(name="_internal")] fn _internal(py: Python, m: &PyModule) -> PyResult<()> { // Register the Tokio Runtime as a module attribute so we can reuse it m.add( ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
