sadpandajoe commented on code in PR #45823:
URL: https://github.com/apache/superset/pull/45823#discussion_r4404163949
##########
Dockerfile:
##########
@@ -303,7 +303,8 @@ COPY requirements/base.txt requirements/
COPY superset-core superset-core
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
- /app/docker/pip-install.sh --requires-build-essential -r
requirements/base.txt
+ --mount=type=bind,source=.,target=/src \
Review Comment:
Bind-mounting the whole build context (`source=.`) into the
dependency-install `RUN` ties that layer's cache key to every non-ignored file,
so any source edit reruns the `requirements/base.txt` install even on the
default Python 3.11, where the script never reads `/src`. That undoes the
ordering the comment above describes (install dependencies before copying
source). Could the mount be limited to what the 3.15 path needs (e.g.
`pyproject.toml` and `superset-core`), or the 3.15 branch moved into its own
`RUN`?
##########
tests/unit_tests/db_engine_specs/test_databricks.py:
##########
@@ -151,6 +151,7 @@ def test_parameters_json_schema() -> None:
assert json_schema == {
"type": "object",
+ "additionalProperties": False,
Review Comment:
These schema assertions now expect `additionalProperties: False` (and
`additionalProperties: {}` for dict fields), but `requirements/base.txt` still
pins `apispec==6.6.1`. On 6.6.1 the marshmallow converter only emits
`additionalProperties` when `Meta.unknown` is set; a plain `Schema` with a
`Dict` field renders as `{'type': 'object', ...}` there. So these equality
asserts (and the matching ones in `postgres_tests.py`, `databases/api_test.py`
and `test_available`) would fail on the existing Python 3.11 lockfile baseline
and only pass with the unpinned resolve used for 3.15. Should the expected
shape be derived from the installed apispec, or should the apispec bound move
so both baselines produce the same schema?
##########
superset/utils/pandas_postprocessing/resample.py:
##########
@@ -240,10 +240,19 @@ def resample( # pylint: disable=too-many-arguments
_validate_projected_rows(df.index.min(), df.index.max(), rule)
if method == "asfreq" and fill_value is not None:
+ string_columns = [
+ column
+ for column in df.columns
+ if isinstance(df[column].dtype, pd.StringDtype)
+ ]
+ if len(string_columns):
+ df = df.astype(dict.fromkeys(string_columns, object))
_df = df.resample(rule).asfreq(fill_value=fill_value)
_df = _df.fillna(fill_value)
elif method == "linear":
- _df = df.resample(rule).interpolate()
+ _df = df.resample(rule).asfreq()
Review Comment:
`linear` now interpolates only numeric columns, so a datetime column that
used to be filled between rows comes back as `NaT`. On pandas 2.3.3, with index
`2024-01-01, 2024-01-03` and a column `2024-02-01, 2024-02-03`, daily
`Resampler.interpolate()` gives `2024-02-02` for the middle row and this path
gives `NaT`. Should datetime columns stay in the interpolated set?
##########
docker/install-app-dependencies.sh:
##########
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
Review Comment:
This new script and `tests/unit_tests/docker_app_dependencies_test.py` have
no ASF license header, and neither path is covered by `.rat-excludes`, so the
Apache RAT license check will flag them; sibling `docker/pip-install.sh` and
the new `test_lint_metadata.py` both carry it. Can the standard header be added
to both files?
##########
docker/install-app-dependencies.sh:
##########
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SOURCE_ROOT="${1:?Application source root is required}"
+
+if python -c 'import sys; raise SystemExit(sys.version_info < (3, 15))'; then
+ if [[ -d "${SOURCE_ROOT}" && ! -w "${SOURCE_ROOT}" ]]; then
+ BUILD_ROOT="$(mktemp -d)"
+ trap 'rm -rf "${BUILD_ROOT}"' EXIT
+ cp -a "${SOURCE_ROOT}/." "${BUILD_ROOT}/"
+ SOURCE_ROOT="${BUILD_ROOT}"
+ fi
+ /app/docker/pip-install.sh --requires-build-essential \
+ --no-sources --no-editable "${SOURCE_ROOT}/superset-core"
"${SOURCE_ROOT}"
Review Comment:
`--no-editable` is not accepted by every uv the Dockerfile allows: it
installs `uv>=0.11.0`, and `uv pip install --no-editable` fails with
`unexpected argument '--no-editable' found` on 0.11.3. A 3.15 image build with
a cached or pinned older uv would fail at this step. Both targets are plain
paths and `--no-sources` already skips the editable source entries, so is the
flag needed, or should the uv floor be raised?
##########
docs/admin_docs/installation/pypi.mdx:
##########
@@ -15,6 +15,143 @@ import useBaseUrl from '@docusaurus/useBaseUrl';
This page describes how to install Superset using the `apache_superset`
package [published on PyPI](https://pypi.org/project/apache_superset/).
+## Python Compatibility
+
+The minimum Python version is 3.11. The Python version classifiers in
+[pyproject.toml](https://github.com/apache/superset/blob/master/pyproject.toml)
+list the officially supported versions; the open-ended `requires-python` lower
+bound does not certify every newer interpreter.
+
+The source checkout includes experimental Python 3.15 compatibility. Its
package
+requirements select NumPy `>=2.5.4,<2.6` and pandas `>=3.1.0,<3.2` on Python
3.15
+and newer. Python 3.11 through 3.14 retain NumPy `>=1.23.5,<2.5` and pandas
+`>=2.3.3,<2.4`; the minimum Python version has not changed. These source
changes
+do not imply that an already published PyPI release supports Python 3.15.
+
+The focused
+[compatibility
workflow](https://github.com/apache/superset/blob/master/.github/workflows/python-metadata-compatibility.yml)
+tests Python 3.11 through 3.15. It installs the application without dependency
+overrides, checks dependency consistency, migrates a fresh SQLite metadata
+database, checks `/health`, and runs engine metadata, pandas post-processing
+(including Prophet), result-set, and core SDK tests. The same 278 tests and
fresh
+SQLite migrations passed locally on Windows for all five interpreter versions,
+including CPython 3.15.0 final.
+
+A local Podman container running Debian 13 with CPython 3.15.2 also passed
Review Comment:
From here the page reads as a private test log (Podman/Hyper-V host notes,
counts like 32,821 passed, an unnamed "raw-cursor join truncation fix",
Playwright case counts) that readers cannot reproduce from the linked workflow,
which runs a much narrower set, and the numbers go stale on the next test
change. Should the run-specific evidence live in the PR description, leaving
this page with the install command, the support policy and the known
limitations?
##########
.github/workflows/python-metadata-compatibility.yml:
##########
@@ -0,0 +1,93 @@
+name: Python-Metadata-Compatibility
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - .github/workflows/python-metadata-compatibility.yml
+ - superset/db_engine_specs/**
+ - tests/unit_tests/db_engine_specs/test_lint_metadata.py
+ - pyproject.toml
+ - superset/result_set.py
+ - superset/utils/pandas_postprocessing/**
+ - tests/unit_tests/pandas_postprocessing/**
+ - tests/unit_tests/result_set_test.py
+ - superset-core/**
+ workflow_dispatch: {}
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number ||
github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ metadata-parser:
+ runs-on: ubuntu-26.04
+ timeout-minutes: 10
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: [ "3.11", "3.12", "3.13", "3.14", "3.15" ]
+ env:
+ PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
+ steps:
+ - name: Checkout
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ persist-credentials: false
+ - name: Setup Python
+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 #
v7.0.0
+ with:
+ python-version: ${{ matrix.python-version }}
+ allow-prereleases: true
+ - name: Install standalone test dependencies
+ run: python -m pip install pytest==7.4.4
+ - name: Test metadata parser without application dependencies
+ run: python -m pytest --noconftest -o addopts=
tests/unit_tests/db_engine_specs/test_lint_metadata.py -q -W
default::DeprecationWarning
+
+ runtime-compatibility:
+ runs-on: ubuntu-26.04
+ timeout-minutes: 25
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: [ "3.11", "3.12", "3.13", "3.14", "3.15" ]
+ env:
+ SUPERSET_HOME: ${{ runner.temp }}/superset-python-compatibility
+ SUPERSET_SECRET_KEY: python-compatibility-ci-only-not-for-production
+ FLASK_APP: superset
+ steps:
+ - name: Checkout
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ persist-credentials: false
+ - name: Setup Python
+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 #
v7.0.0
+ with:
+ python-version: ${{ matrix.python-version }}
+ - name: Install application and focused test dependencies
+ run: >-
+ python -m pip install -e ./superset-core -e . pytest==7.4.4
pytest-mock freezegun parameterized responses rich prophet
+ - name: Check dependency consistency
+ run: python -m pip check
+ - name: Migrate a fresh metadata database
+ run: python -m flask db upgrade
+ - name: Test application health and runtime compatibility
+ run: |
+ python - <<'PY'
+ import pytest
+ from superset.app import create_app
+
+ app = create_app()
+ assert app.test_client().get('/health').status_code == 200
+ with app.app_context():
+ raise SystemExit(pytest.main([
+ '--noconftest', '-o', 'addopts=',
+ 'tests/unit_tests/db_engine_specs/test_lint_metadata.py',
Review Comment:
This is the only job that runs on 3.15/pandas 3, but the PR also changes
pandas-sensitive code in `superset/commands/dataset/importers/v1/utils.py`,
`superset/models/helpers.py` and the metadb cursor wrapper, whose tests
(`tests/unit_tests/commands/importers/v1/utils_test.py`,
`tests/unit_tests/models/helpers_test.py`,
`test_superset_joins_with_limit_drops_fanout_matches` in
`tests/unit_tests/extensions/test_sqlalchemy.py`) are not in this selection and
otherwise run only on 3.11. The `paths:` filter also omits `Dockerfile` and
`docker/**`. A later change to those files would never run against pandas 3.1.
Should those tests and paths be added so the experimental 3.15 claim stays
checked in CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
