DoDiODev opened a new pull request, #9352: URL: https://github.com/apache/devlake/pull/9352
### ⚠️ Pre Checklist > Please complete _ALL_ items in this checklist, and remove before submitting - [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/). - [x] I have added relevant tests. (existing suites adapted; no new behavior) - [x] I have added relevant documentation. (N/A — no user-facing change) - [x] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc. (I can't set labels myself; suggested: `pr-type/refactor`, `component/plugins`) ### Summary Moves the Python plugin framework from Pydantic 1 to Pydantic 2, as agreed in #9092. Pydantic 1 does not support Python 3.14, so this is the first step before the Python runtime (3.11, EOL 2027-10) can be upgraded. | Package | Before | After | |---|---|---| | pydantic | 1.10.27 | 2.14.3 | | sqlmodel | 0.0.8 | 0.0.38 | | SQLAlchemy | 1.4.41 | 2.0.57 | Ported: `pydevlake`, `azuredevops`, `glpi`, `redmine`, `zephyr` and the e2e `fakeplugin`. The three Poetry locks only change the Pydantic stack (plus its new dependencies `pydantic-core`, `annotated-types`, `typing-inspection`; `sqlalchemy2-stubs` is dropped). **pydevlake** - Pydantic 2 APIs: `model_config`/`ConfigDict`, `field_validator`, `model_fields`, `model_dump_json`, `RootModel`, `Literal` instead of `Field(const=True)`. - `Optional[...]` fields get an explicit `= None` default (in Pydantic 2 they are required otherwise). This accounts for most of the diff in the domain layer and plugin models/migrations. - `sa_column=Column(...)` → `sa_type=...` (+ `sa_column_kwargs` for the `created_at`/`updated_at` defaults), so columns are not shared between inherited tables. - `ToolTable` sets `coerce_numbers_to_str=True` to keep the Pydantic 1 behavior for APIs that return ids as numbers. - `autoextract` uses `model_validate`: SQLModel table models skip validation in `__init__`, so raw strings would no longer be coerced to `datetime`/enums. - `RawModel` keeps its column types (`data`/`input` binary, `url` TEXT), so existing raw tables stay compatible. Two tests wrote `str` into `data`; they now encode to bytes like the collector does (SQLAlchemy 2 rejects `str` for binary columns). - `message.py`: `SerializeAsAny` for `RemoteScope.data`, otherwise Pydantic 2 serializes the base type and drops plugin-specific scope fields. - `model_info.py`: reads `json_schema_extra` / annotations instead of `field.field_info.extra` / `field.required`. - Test harness: expected and actual models are re-validated before comparing, so both sides have the same field types. **Plugins** - `glpi`: GLPI expands dropdowns to `'Name (id)'`; these values are reduced to the id before validation of the integer columns. The names are still read from the raw data for enrichment, so the output does not change. - `redmine`, `zephyr`: `class Config` → `model_config`. - `fakeplugin`: ids as strings, `Optional` defaults. **Dependabot** - The `pydantic` major-version ignore is removed. - The `sqlmodel` ignore stays: 0.0.48 rejects naive datetimes (pydevlake uses `datetime.now()`/`utcnow()`) and pulls in SQLAlchemy 2.1. That is a separate follow-up. ### Does this close any open issues? Part of #9092 (Pydantic step; Python runtime upgrade follows separately). ### Screenshots N/A ### Other Information **Tests** - `cd backend/python && sh ./run_tests.sh` — passed: azuredevops 8 passed / 1 skipped, glpi 72 passed / 1 skipped, redmine 31 passed / 1 skipped, zephyr 26 passed, pydevlake 13 passed. - `make unit-test` and `make e2e-test` (MySQL and PostgreSQL, includes the Python `fakeplugin`) — passed in the fork CI: [run 42621392320](https://github.com/DoDiODev/devlake/actions/runs/42621392320) (16/16 jobs) **Not in this PR** - Python runtime 3.11 → 3.13/3.14 (next PR). - sqlmodel ≥ 0.0.48 / SQLAlchemy 2.1 (timezone-aware datetimes). - Pinning the uv installer in `backend/Dockerfile` and the lake-builder image. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
