samraj2k commented on PR #73209: URL: https://github.com/apache/airflow/pull/73209#issuecomment-6066284528
> Who needs Dag bundles configured at runtime, and how does this relate to #71111? This adds a `[dag_processor] dag_bundle_provider` option, a public `DagBundleProvider` base class and a `team_name` field on `DagFileParseRequest`, all permanent surface, but on its own it changes no behavior and links no issue. #71111 already proposes dynamic bundle configuration with a different design (reading it from a file path), and both shouldn't land. Is there an issue or dev-list discussion where a maintainer has picked an approach? Converting to draft until there is. Thanks @kaxil for the initial comments. We maintain 100s of bundles, and runtime bundles will be huge help for us. Along with that, I have attached list of discussion that require the same feature of bundles at runtime. > how does this relate to #71111? This PR too is on similar lines, but there, the only way to configure runtime bundles is through file. I think this approach though it works, works for limited case. For examples, it can be a read from S3, or consuming through API. In my PR, we are introducing abstraction such that Airflow is not aware or coupled to how to fetch list of bundles, and thus we exposed public interface for the same. This way, all the cases can be catered, whether reading from K8 path, filepath, S3, GCS etc > dev-list discussion where a maintainer has picked an approach I have sent the email for the same titled: > [DISCUSS] Dynamic discovery of DAG bundle configurations without restart so maintainers can align on the approach, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
