amoghrajesh commented on code in PR #73912: URL: https://github.com/apache/airflow/pull/73912#discussion_r4145272475
########## dev/breeze/doc/ci/05_workflows.md: ########## @@ -312,6 +313,31 @@ Special tests (integration and system tests) run selectively: - In canary runs for scheduled quality checks - When dependency upgrades require thorough testing +**`(4)` Agent Framework Tests** Review Comment: We also would want to add `run-agent-framework-tests` to `04_selective_checks.md` ########## scripts/in_container/run_agent_framework_tests.sh: ########## @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Runs the common.ai adapter tests for agent frameworks that cannot join the workspace lock. +# +# Strands Agents caps mcp, and Google ADK caps opentelemetry and websockets, below the versions +# uv.lock resolves, so their adapter tests are skipped everywhere else in CI. This installs one +# framework into the CI image and runs the tests of the framework-neutral tools and their adapters; +# the other framework's tests skip themselves. One framework per invocation, so a bad release of +# one cannot mask the other. +# +# By default every package already in the image is held at its installed version with uv's +# --override, so the framework is tested against the same dependencies as the rest of Airflow and +# its caps on them are overridden. With --framework-pins the framework's own requirements win +# instead, which is the environment a user who installs it gets. +# +# The newest framework release older than the repository's uv exclude-newer window is installed. +set -euo pipefail + +TEST_PATH="providers/common/ai/tests/unit/common/ai/tools" + +framework="${1:-}" +case "${framework}" in + strands-agents) import_check="import strands" ;; + google-adk) import_check="import google.adk" ;; + *) + echo "Usage: $0 <strands-agents|google-adk> [--framework-pins]" >&2 + exit 1 + ;; +esac +FRAMEWORKS=("${framework}") + +framework_pins="false" +if [[ ${2:-} == "--framework-pins" ]]; then + framework_pins="true" +elif [[ -n ${2:-} ]]; then + echo "Unknown argument: ${2}. The only option after the framework is --framework-pins." >&2 + exit 1 +fi + +cd "${AIRFLOW_SOURCES:-/opt/airflow}" + +if [[ ${framework_pins} == "true" ]]; then + echo "Installing ${FRAMEWORKS[*]} with their own dependency pins" + uv pip install "${FRAMEWORKS[@]}" +else + overrides=$(mktemp) + before=$(mktemp) + after=$(mktemp) + trap 'rm -f "${overrides}" "${before}" "${after}"' EXIT + uv pip freeze | sort > "${before}" + # Only name==version lines: editable and local installs cannot be expressed as an override, + # and the frameworks do not depend on any of them. Overriding the rest means nothing the + # image ships should change; the check below is there in case something still does. + grep -E '^[A-Za-z0-9_.-]+==' "${before}" > "${overrides}" + echo "Installing ${FRAMEWORKS[*]}, holding the image's $(wc -l < "${overrides}") installed packages" + uv pip install --override "${overrides}" "${FRAMEWORKS[@]}" + uv pip freeze | sort > "${after}" + changed=$(comm -23 "${before}" "${after}") + if [[ -n ${changed} ]]; then + echo "Installing the frameworks changed packages the image already had:" >&2 + echo "${changed}" >&2 + exit 1 + fi +fi + +uv pip freeze | grep -iE '^(strands-agents|google-adk|mcp|opentelemetry-(api|sdk)|websockets|google-genai)==' + +# The adapter tests skip themselves when their framework is missing, so a broken install would +# otherwise pass as green. +python -c "${import_check}" + +# --skip-db-tests: the job runs with backend "none", which has no database to set up. +pytest "${TEST_PATH}" --skip-db-tests -p no:cacheprovider --color=yes -ra Review Comment: Each matrix leg runs the whole directory, so `test_tools.py`, `test_tracing.py` and `test__from_toolset.py` run twice, once per framework. Only the adapter file differs between legs. Not worth blocking on, but a per-framework path plus one shared leg would halve it. ########## .github/workflows/agent-framework-tests.yml: ########## @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Agent framework tests +on: # yamllint disable-line rule:truthy + workflow_call: + inputs: + runners: + description: "The array of labels (in json form) determining runners." + required: true + type: string + platform: + description: "Platform for the build - 'linux/amd64' or 'linux/arm64'" + required: true + type: string + default-python-version: + description: "Which version of python should be used by default" + required: true + type: string + use-uv: + description: "Whether to use uv" + required: true + type: string + canary-run: + description: > + On a canary run the frameworks' own dependency pins win, which is the environment a + user who installs them gets. Otherwise every package in the CI image keeps its version + and the frameworks' caps on them are overridden. + required: true + type: string +permissions: + contents: read +jobs: + tests: + timeout-minutes: 30 + name: >- + Agent framework tests: ${{ matrix.framework }} + (${{ inputs.canary-run == 'true' && 'framework pins' || 'image versions' }}) + runs-on: ${{ fromJSON(inputs.runners) }} + # amd64 only for now: ci-amd.yml and ci-arm.yml must stay in sync, so the platform gate lives here. + if: inputs.platform == 'linux/amd64' Review Comment: Good call putting the gate here rather than dropping the job from `ci-arm.yml` ########## scripts/in_container/run_agent_framework_tests.sh: ########## @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Runs the common.ai adapter tests for agent frameworks that cannot join the workspace lock. +# +# Strands Agents caps mcp, and Google ADK caps opentelemetry and websockets, below the versions +# uv.lock resolves, so their adapter tests are skipped everywhere else in CI. This installs one +# framework into the CI image and runs the tests of the framework-neutral tools and their adapters; +# the other framework's tests skip themselves. One framework per invocation, so a bad release of +# one cannot mask the other. +# +# By default every package already in the image is held at its installed version with uv's +# --override, so the framework is tested against the same dependencies as the rest of Airflow and +# its caps on them are overridden. With --framework-pins the framework's own requirements win +# instead, which is the environment a user who installs it gets. +# +# The newest framework release older than the repository's uv exclude-newer window is installed. +set -euo pipefail + +TEST_PATH="providers/common/ai/tests/unit/common/ai/tools" + +framework="${1:-}" +case "${framework}" in + strands-agents) import_check="import strands" ;; + google-adk) import_check="import google.adk" ;; + *) + echo "Usage: $0 <strands-agents|google-adk> [--framework-pins]" >&2 + exit 1 + ;; +esac +FRAMEWORKS=("${framework}") + +framework_pins="false" +if [[ ${2:-} == "--framework-pins" ]]; then + framework_pins="true" +elif [[ -n ${2:-} ]]; then + echo "Unknown argument: ${2}. The only option after the framework is --framework-pins." >&2 + exit 1 +fi + +cd "${AIRFLOW_SOURCES:-/opt/airflow}" + +if [[ ${framework_pins} == "true" ]]; then + echo "Installing ${FRAMEWORKS[*]} with their own dependency pins" + uv pip install "${FRAMEWORKS[@]}" +else + overrides=$(mktemp) + before=$(mktemp) + after=$(mktemp) + trap 'rm -f "${overrides}" "${before}" "${after}"' EXIT + uv pip freeze | sort > "${before}" + # Only name==version lines: editable and local installs cannot be expressed as an override, + # and the frameworks do not depend on any of them. Overriding the rest means nothing the + # image ships should change; the check below is there in case something still does. + grep -E '^[A-Za-z0-9_.-]+==' "${before}" > "${overrides}" + echo "Installing ${FRAMEWORKS[*]}, holding the image's $(wc -l < "${overrides}") installed packages" + uv pip install --override "${overrides}" "${FRAMEWORKS[@]}" + uv pip freeze | sort > "${after}" + changed=$(comm -23 "${before}" "${after}") + if [[ -n ${changed} ]]; then + echo "Installing the frameworks changed packages the image already had:" >&2 + echo "${changed}" >&2 + exit 1 + fi +fi + +uv pip freeze | grep -iE '^(strands-agents|google-adk|mcp|opentelemetry-(api|sdk)|websockets|google-genai)==' Review Comment: This is a grep under `set -euo pipefail`. If it matches nothing, the script dies with no message and no context. `|| true` if it's only for the log, or turn it into a real check with a real error. ########## scripts/in_container/run_agent_framework_tests.sh: ########## @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Runs the common.ai adapter tests for agent frameworks that cannot join the workspace lock. +# +# Strands Agents caps mcp, and Google ADK caps opentelemetry and websockets, below the versions +# uv.lock resolves, so their adapter tests are skipped everywhere else in CI. This installs one +# framework into the CI image and runs the tests of the framework-neutral tools and their adapters; +# the other framework's tests skip themselves. One framework per invocation, so a bad release of +# one cannot mask the other. +# +# By default every package already in the image is held at its installed version with uv's +# --override, so the framework is tested against the same dependencies as the rest of Airflow and +# its caps on them are overridden. With --framework-pins the framework's own requirements win +# instead, which is the environment a user who installs it gets. +# +# The newest framework release older than the repository's uv exclude-newer window is installed. +set -euo pipefail + +TEST_PATH="providers/common/ai/tests/unit/common/ai/tools" + +framework="${1:-}" +case "${framework}" in + strands-agents) import_check="import strands" ;; + google-adk) import_check="import google.adk" ;; + *) + echo "Usage: $0 <strands-agents|google-adk> [--framework-pins]" >&2 + exit 1 + ;; +esac +FRAMEWORKS=("${framework}") Review Comment: `FRAMEWORKS=("${framework}")` is an array of one, left over from the old shape. Just use `${framework}` at the two call sites below? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
