Hi all, The command 'airflow db clean' currently hardcodes some provider-owned tables in Core. This causes a bug when those tables are configurable. Specifically 'celery_taskmeta' and 'celery_tasksetmeta' may not be correctly located if the user sets the '[celery] result_backend_schema' configuration since the config is not respected in core.
I have proposed #70639 to address this. This builds on #70147 (already merged), which allows the core clean command to handle a schema-qualified table name. The new PR #70639 adds a provider.yml key db-cleanup-tables key pointing at a callable that returns cleanup configs. ProvidersManager discovers them (like db-managers) and db clean merges them in; a provider resolves its own settings, so core needn't read provider config for tables it contributes. The PR introduces the extension point and moves Celery onto it. I also identified a couple of possibilities: * FAB. The session table is currently hard-coded in core as a carry-over from Airflow 2, when FAB was a required dependency. * Edge3. edge_logs looks like a fit, edge_job would need a recency/terminal-state decision, and edge_worker probably isn't a cleanup target. What do y'all think? Especially for the Edge3 tables; I'm not involved in the provider enough to make a call. The FAB one is a toss-up to me; it seems clean to move it to the provider, but also OK to be kept as-is since it's always had special status compared to most providers. Also, feel free to raise any suggestions to the interface put out in #70639. Thanks, TP [#70639]: https://github.com/apache/airflow/pull/70639 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
