potiuk commented on code in PR #24591: URL: https://github.com/apache/airflow/pull/24591#discussion_r911417103
########## airflow/cli/__main__.py: ########## @@ -18,6 +18,7 @@ # under the License. from airflow.cli import airflow_cmd +from airflow.cli.commands import version # noqa: F401 Review Comment: Yeah 100ms might be a challenge. Or maybe not? Actually I participated last week in PyWaw meetup with none other than Łukasz Langa (the entrepreneur-in-residence of Python. The talk was precisely about speeding up Python code. They are in the very advanced phase of speeding up Python wiht Python 3.11 but this one was about something else - namely about speeding up Python code by using `mypyc`. This would be HUGE win if we would manage to pull-it-off. The super-nice thing about mypyc is that it does not require you to change the way you write python code, it has a number of limitations (for example we would not be able to use it with the metaclasses we use here and there and dynamic loading of classes we do with "airflow" module and only single-inheritance is supported). They (Łukasz was part of the team) managed to make `black` compatible with mypyc and speed it up. In short MyPyc allows to turn fully type-hinted Python code using mypy into much faster compied C-code. And (comparing to other similar approaches it actually allows to do that with very little limitations on what you can use in the python code you compile. The main "prrof" and goal they had was to make `black` faster, and I think they succeeded: <img width="896" alt="Screenshot 2022-06-30 at 22 05 36" src="https://user-images.githubusercontent.com/595491/176768599-4d0d53f2-64cf-4ca5-9422-1e9d12752967.png";> You should definitely watch the talk: https://www.youtube.com/watch?v=-Kku_hFUtY0 (you can hear me by the end asking about how we can use it in airflow). Besides having a TON of useful information and experiences about speeding up Python in general, it is very entertaining, Łukasz is a great speaker and he is a great person in general. We went to after-party with a group of Pythonistas and I picked a lot of Łukasz's brain about the subject. It was over (more than) couple of beers, so take it with a grain of salt :). But what I brought from that is that if we have a separate python module with CLI-only functionalitty that we could separate out from rest of airflow and compile with mypyc, that would be a huge win for the speed of CLI. This is even more scope creep on the PR but maybe the first founding PR should also explore this possibility - this way we could see if we CAN do such aa speedup and make super-fast CLI which would be compiled with mypyc. Łukasz - answering my question specifically confirmed my gues that for the core of airflow (mostly DB and networking) mypyc will have very limited impact - but the CLI part might heavily benefit it (of course that would complicate our build setup but - if we are going to rewrite the CLI to click, maybe it's worth checking if it can be done with mypyc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/595491/176768599-4d0d53f2-64cf-4ca5-9422-1e9d12752967.png"](https://user-images.githubusercontent.com/595491/176768599-4d0d53f2-64cf-4ca5-9422-1e9d12752967.png"){kind=link}