potiuk commented on PR #52002:
URL: https://github.com/apache/airflow/pull/52002#issuecomment-3095638532
> I understand current approach is super brittle and provides almost no
guarantees. Indeed Connection is too wide abstraction and doesn't currently
couple with ObjectStoragePath. I have seen similar brittle coupling in other
Airflow parts and considered that would be maybe ok in here also.
I would say "loose" coupling rather than "brittle". "Brittle" suggest
instability. "Loose" means that the coupling is deliberately loose. As
everything in IT choices are often trade-offs. And "strong coupling" is a
choice you can make, "loose coupling" is another. Each of them comes with pros
and cons, and "strong coupling" is not always the best - because - for example
- it causes unnecessary ripple effects when you make changes that are not
applicable to "all entities". This is exactly the same case with DRY -> DRY
means strong coupling and while wildely considered as good practice, it's not
always the best and you can sometimes (quite often in fact) choose to copy
things if close coupling and DRY makes your code unnecessary complex.
In this case - connection providing authentication information mostly is
deliberate choice - it's used in many places and many ways so strong coupling
it with particular "users", where such coupling is not needed and "noise" for
other "users" is something we do not want.
> Is there any best practice on configuring Object Storage initialization?
Maybe we can just improve
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html
with best patterns. I'm trying to avoid unnecessary branching as in this
pseudo-code.
I would say there are likely several practices you can use and we do not
want to limit our users to choose only one approach. I can for example imagine
(easily and I know people are doing it) that you are using different
connection_id for local and remote (you chose a different pattern). But yes
decribing dos on how you could potentially do it, might be a good idea.
So if you are willing to describe few of those patterns and possible usages
- feel absolutely free. We do it already in a number of places - for example
https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#handling-conflicting-complex-python-dependencies
where we describe various ways you can handle conflicting requirements or
https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#dynamic-dag-generation
where we are describing various patterns of dynamic dag generation without
falling into "doing too much while parsing" trap.
Feel absolutely free to make PR with such documentation, find the right
place where to add it, describe a few variants with examples, that would be
cool contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
