potiuk commented on code in PR #58825:
URL: https://github.com/apache/airflow/pull/58825#discussion_r2574312283
##########
shared/import_utils/src/airflow_shared/import_utils/module_loading.py:
##########
@@ -67,3 +67,25 @@ def qualname(o: object | Callable) -> str:
def iter_namespace(ns: ModuleType):
return pkgutil.iter_modules(ns.__path__, ns.__name__ + ".")
+
+
+def is_valid_dotpath(path: str) -> bool:
Review Comment:
1) I am not so sure to be honest. We have almost completed the shared code
extraction - there is not much left
2) I am not sure if it is a problem if we have more of those. The setup we
have now is rally easy for sharing and it's super easy to "reuse" a code
without any of the side effects. I think it's **precisely** as I imagine
sharing code even for small modules. No coupling, perfect IDE support, prek
hook to make sure that imports are proper, building works out of the box,
testing is isolated and does not depend on anything else (and uv sync takes
care of it). Also each "module" has it's own dependencies that are individually
automatically contributed to the user's dependencies when the users starts
using particular module.
I think we now should see what else we can extract this way - rather than
what to copy - especially if those are isolated, independent functtions.
This setup we have now has pretty much all DRY benefits and no DRY drawbacks
(coupling) - I see no reason why "shared" shoud not have more of those isolated
"util" things.
There is a **huge** problem of course if "util" is a **single bag of
everything** and we should **definitely** not do it. I literally hate when you
have it and when utils starts using each other in uncontrolled way (which
happens very quickly when you have shared bag of utils). And when using one
util means that you are signing up for all of them even if you do not want. But
it's not our case.
But in this case we have not "bag of everything" but "single purpose easily
shareable and isolated modules" - which is very different than "bag of utils".
mostly because those separate shared modules will be independent from each
other (mostly) and each of the "users" will only use those specific modules
when needed - without necessarily importing all the different kinds of utils
they do not need.
This is **very** different than `bag of utils` IMHO.
##########
shared/import_utils/src/airflow_shared/import_utils/module_loading.py:
##########
@@ -67,3 +67,25 @@ def qualname(o: object | Callable) -> str:
def iter_namespace(ns: ModuleType):
return pkgutil.iter_modules(ns.__path__, ns.__name__ + ".")
+
+
+def is_valid_dotpath(path: str) -> bool:
Review Comment:
1) I am not so sure to be honest. We have almost completed the shared code
extraction - there is not much left
2) I am not sure if it is a problem if we have more of those. The setup we
have now is rally easy for sharing and it's super easy to "reuse" a code
without any of the side effects. I think it's **precisely** as I imagine
sharing code even for small modules. No coupling, perfect IDE support, prek
hook to make sure that imports are proper, building works out of the box,
testing is isolated and does not depend on anything else (and uv sync takes
care of it). Also each "module" has it's own dependencies that are individually
automatically contributed to the user's dependencies when the users starts
using particular module.
I think we now should see what else we can extract this way - rather than
what to copy - especially if those are isolated, independent functtions.
This setup we have now has pretty much all DRY benefits and no DRY drawbacks
(coupling) - I see no reason why "shared" shoud not have more of those isolated
"util" things.
There is a **huge** problem of course if "util" is a **single bag of
everything** and we should **definitely** not do it. I literally hate when you
have it and when utils starts using each other in uncontrolled way (which
happens very quickly when you have shared bag of utils). And when using one
util means that you are signing up for all of them even if you do not want. But
it's not our case on either front.
But in this case we have not "bag of everything" but "single purpose easily
shareable and isolated modules" - which is very different than "bag of utils".
mostly because those separate shared modules will be independent from each
other (mostly) and each of the "users" will only use those specific modules
when needed - without necessarily importing all the different kinds of utils
they do not need.
This is **very** different than `bag of utils` IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
