potiuk commented on PR #28317: URL: https://github.com/apache/airflow/pull/28317#issuecomment-1347420793
> Do we care about provider deps in k8s tests? > > Or more specifically, why should changing the deps of (say) the google provider invalidate the k8s cache? Very good question. But I am afraid the answer is long. It's only cncf.kuberntes and postgres that count: https://github.com/apache/airflow/blob/main/scripts/ci/kubernetes/k8s_requirements.txt#L1 so technically we could base it on ``` hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','setup.cfg','setup.py','pyproject.toml','airflow/providers/cncf/kubernetes/provider.yaml', 'airflow/providers/postgres/provider.yaml') }}" ``` However, if (when) we move providers and change the way wehere we manage dependencies (for example in pyproject.toml of each provider) we could have easily missed changing those. Some time ago I added "generated/provider_dependencies.json" as a "stable proxy" for the dependencies of "airflow extras" in general (it was really added to decouple airflow extras and setup.py from provider.yaml files so that we can move those around and change their meaning independently on whatever build process change for airflow we come up with or wherever we move provider.yaml files - or maybe even get rid of those deps in provider.yaml) There is a little potential overhead for PRs - just venv installation will get a bit longer every time we change dependencies. But in most cases it is not slowing down PRs because K8S tests are only run when k8s tests or helm chart changes so this is a "rare" event in PR. Once a PR with provider dependencies gets merged and runs the K8S tests and canary runs succed, the cache is rebuilt and it gets back to normal. So I figured, depending on "all provider dependencies" is quite acceptable and easier to avoid mistakes when we manipulate providers. And yes, I am not joking, this is the actual thought process I did when adding this PR :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
