potiuk commented on PR #26486: URL: https://github.com/apache/airflow/pull/26486#issuecomment-1253736318
I thought a lot about it (while running yesterday - 18km is long enough to clear the head). And I am still not really convinced we should do it this way. I am even willing to try to attempt to make a smal POC of "tab" approach where we show both "Classic" approach and "TaskFlow" one when we click on a separate tab (and even apply it to many examples that @josh-fell had already converted to the TaskFlow). Looking at the comments in this thread I tried to convince myself that not showing at all TaskFlow there is a good idea, but - I could not. Why I think so? There are few reasons, let me elaborate on my thinking: I think there are many more users that use currently mostly the Classic approach. And this is fine. And it will stay like that. And it gives huge competitve edge over other solutions that work the other way round - using decorator's approach as a primary way of doing things. Airflow has literally thousands of operators/sensors/transfers that work out-of-the-box and you can treat them mostly like black-box). And we want to attract those users for sure (however for those users declarative approach would be even better and this is something that we will hopefully have some day as another way of writing DAGs. Some day, possibly not that far in the future). And I fully agree when those users open the Airlfow "main" page, showing the "classic" approach is way better. However I think we do not only want to attract those kinds of users, we also want to attract those that know and love Python well and they have gone (similarly like us) through the recent Python 2.7 -> Python 3.7+ and they are eager to have more flexibility, less boilerplate, more functional/immutable approach - all those things that make your Python code slick and neat and much easier to read and reason about actually. Those users are often using other solutions possibly, that they can write their DAGs more "pythonically". And we want to promote Airflow to those users, showing the more "modenr" and "slick" face of Airflow. And I think if I were using such a solution currently and "Airlfow" woudl become an option for me, I would likely take a look at the main page and try to figure out if Airflow is for me - without going any deeper. And (trying to put myself in the shoes of such person) - I coudl be put off, if it was not clear from the main page that you can do things in a more "slick" way. Have you actually tried to find out that Airflow has a task flow using the current documentation. It's referred in a few places. There is a "Task Flow Tutorial" but It's not quite prominent enought that a busy person who does a first pass evaluation of a tool like Airlfow would find it easily. And for many of those "power users" this might end up with "meh, not worth it". I think it is something that we cannot forget that not everyone there knows what Airflow is about. They've heard something maybe, maybe they read a blog post listing Airlfow as a "must have" tool or maybe they saw it in some "Modern Data Stack" diagrams among around a million other logos. But we should assume they came to the main page of airlfow docs to learn what Airlfow is. And they might stop right there. Unfortunately - we do not know who is looking. I think if you consider someone who does not know Airflow already - they will end up on the main page to learn what Airflow is - REGARDLESS if they are power Python users or newbies. We have no way to check it, but I think this is what will happen for both types of users. And not giving a clue that Airlfow DAGs can be written in task flow way is a risk that they will leave the page with that impression in their mind - that it works using "kinda dated approach to define DAGs". What's even more - those Power users are more likely than Python newbies to spread the news. There are various ways information about a tool spreads, but one of the most powerful ways is that your local respected Python Guru says "Yeah, I looked at it and Airflow is popular but it does not really have a modern way of writing your DAGs" vs. "Yeah I have not used it but from a quick look it seems to be popular and ALSO suports the modenr way of writing your DAGs". This is what I am afraid of when it comes to the "first imipression". There is this very true statement - "You never get a second chance to make the First impression". And what I am really about is to give a chance to make good first impression for both kinds of users. One more comment that I want to add and something that we should take into account. I think making airflow "main" page showing only the "classic' way, make our docs extremely inconistent in the way how we treat "new users". For example when new people want to learn about all kinds of [PythonOperators](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#) including ShortCircuit operator - they will ONLY learn about the TaskFlow approach. There are not even examples there on how you can use the "classic version" of those any more. Given the popularity of Python operators, this is far more confusing for users who do not know how to use decorators. Imagine you just looked at Airlfow is, you saw the main page with "classic" approach, and then you want to try your first Python operator and look at the examples and you see the decorators. Confusing? Like crazy. I think no matter what approach we take, but whatever we do - we should do it consistently. Like Now. The "current" 2.4.0 where classic and taskflow apprach are mixed at least have a chance to prepare the user that there is something like TaskFlow. And I think working out an approach where you can somehow address both kind of users is important to make a conscious decision that we apply consistently. cc: @rossturk, I know you've been thinking a lot about the main page of Airlfow and with your "newcomer" view you can probably provide a bit more of a "fresh" look/view. Would love to hear your opinion on that one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
