simi commented on PR #52002: URL: https://github.com/apache/airflow/pull/52002#issuecomment-3096247635
@potiuk I was looking mainly at those comments. https://github.com/apache/airflow/blob/0f0ea3b5c444465777a7c5eb5188b301faed1aee/airflow-core/src/airflow/models/connection.py#L99-L103 https://github.com/apache/airflow/blob/0f0ea3b5c444465777a7c5eb5188b301faed1aee/airflow-core/src/airflow/models/connection.py#L254 https://github.com/apache/airflow/blob/0f0ea3b5c444465777a7c5eb5188b301faed1aee/airflow-core/src/airflow/models/connection.py#L469-L471 Looking at connection unification, for some providers crucial fields are "standard". <img width="1552" height="1196" alt="image" src="https://github.com/user-attachments/assets/51b45d0c-9372-4c95-a5b0-5db7a89b71c1"; /> For some providers fields are "extra". <img width="1552" height="1196" alt="image" src="https://github.com/user-attachments/assets/9fa7c4a1-565f-4203-bd4d-5e131b611744"; /> It is quite hard to navigate for me in this. > And again - documenting the patterns that you described where you want to change schemes in prod/dev (either by changing connection id, or definition of connection) is a good idea. There is a good reason why Dags are written in Python because rather than declaratively describe and strongly couple such relations you can "loosely" couple them and add either automated deriving or validation in the Python code of yours - up to you. @potiuk I'm not 100% happy with any current solution and documenting my helper method is not good idea IMHO. It should be decided good pattern and merged or left on side, since it will be questionable why this helper is not part of the codebase if documented. And as mentioned it introduces some requirements on extra columns, which is not standard. IMHO the ship on unifying IO access (ObjectStorage) has sailed already, sooner or later this will need unification also. If Connection is not the best place, maybe new entity like Filesystem connected to Connection and getting more info (like bucket/base_path) could be the way. But that's out of scope of my capabilities to introduce. Btw. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html# mentions. > Airflow provides a generic abstraction on top of object stores, like s3, gcs, and azure blob storage. This abstraction allows you to use a variety of object storage systems in your dags without having to change your code to deal with every different object storage system. IMHO it is not 100% true and that's exactly what am I looking for. The last piece is still missing (how to initialize ObjectStorage simply from already configured access in generic way) to be able to use variety of object storage systems without having to change your code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
