jiangxt2 opened a new issue, #12211:
URL: https://github.com/apache/gravitino/issues/12211
### What would you like to be improved?
`pip install apache-gravitino` (verified against PyPI metadata for v1.3.0)
forces installation of 13 exact-pinned packages as unconditional runtime
dependencies. Several of these are development tools that are never imported at
runtime:
| Package | Actual usage | Runtime import? |
|---------|-------------|-----------------|
| `black==26.3.1` | Code formatter | No |
| `flake8==7.0.0` | Linter | No |
| `pre-commit==3.5.0` | Git hooks | No |
| `requests==2.32.5` | Integration tests only | No |
The remaining packages (`pyarrow`, `s3fs`, `gcsfs`, `ossfs`, `adlfs`) are
used at runtime but only through lazy `importlib.import_module()` calls in
specific storage handlers (`gvfs_storage_handler.py`). They are optional by
nature — a user accessing only S3 never triggers the GCS/Azure/OSS/HDFS code
paths — yet all are installed unconditionally.
The exact `==` pinning also causes frequent pip resolution conflicts when
users have other packages requiring different versions of shared transitive
deps like `fsspec`.
For comparison, SDKs like `databricks-sdk` use flexible version ranges
(`requests<3,>=2.28.1`) and only declare true runtime deps, with optional
backends under extras.
### How should we improve?
Separate dependencies into core runtime + optional extras:
```
# Core runtime (always installed; use CI-validated lower bounds,
# add upper bounds only where known incompatibilities exist)
dataclasses-json>=0.6.7
readerwriterlock>=1.0.9
fsspec>=2024.1.0
cachetools>=7.0
# Optional storage extras (user opts in per provider)
[s3] → s3fs (version-compatible with fsspec floor)
[gcs] → gcsfs
[oss] → ossfs
[azure] → adlfs
[hdfs] → pyarrow
[cloud] → s3fs, gcsfs, ossfs, adlfs (all cloud backends, no HDFS)
# Dev/test (CI and contributors)
[test] → black, flake8, pre-commit, pytest, requests, pyarrow,
s3fs, gcsfs, ossfs, adlfs, pandas, coverage, ...
```
Note: The project already has a `[lance]` extra. The aggregation extra is
named `[cloud]` rather than `[all]` to avoid ambiguity about whether it
includes `[hdfs]` and `[lance]`.
**Breaking change mitigation:**
Removing cloud backends from the default install is a user-visible breaking
change — existing users who run `pip install apache-gravitino` and then access
S3/GCS/GVFS will get a runtime `ModuleNotFoundError`. To mitigate:
1. Each storage handler's `get_filesystem()` should catch
`ModuleNotFoundError` and raise a descriptive error with install guidance,
e.g.: `Install "apache-gravitino[s3]" to use s3a:// locations.`
2. Target this change for the next major release (2.0.0), or introduce a
two-phase migration:
- Phase 1 (minor release): Add extras, emit a deprecation warning when a
lazily-imported storage backend is resolved from the default install rather
than an explicit extra.
- Phase 2 (major release): Remove cloud/hdfs packages from core
`install_requires`.
**Version constraints:** Use CI-validated lower bounds for each runtime dep.
Add upper bounds only where known incompatibilities exist. The `fsspec`
ecosystem packages (`gcsfs`, `s3fs`, `ossfs`, `adlfs`) have tight cross-version
coupling and should be validated via an installation matrix in CI, not just by
sharing a single floor version.
**Test environment:** Gradle currently installs `.[dev]`
(`build.gradle.kts:161`). After this change, CI/Gradle should install `.[test]`
which aggregates dev tools + all storage backends + HDFS, ensuring no test
breakage.
**Relationship with #7779:** This is complementary to the `uv` migration.
#7779 addresses build-side tooling (Gradle + Miniforge → uv); this issue
addresses the user-facing install contract. PR #7811 could incorporate this
layering in `pyproject.toml`'s `[project.dependencies]` +
`[project.optional-dependencies]`, or it can be implemented independently.
Related: #7779, #7811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
