Really good feedback, Jakob and Anton. As Jarek said, it would be really helpful to identify the areas of documentation that we are lacking greatly. There is a term called "expert blindness <https://ux.stackexchange.com/questions/9802/is-there-a-name-or-a-phrase-for-not-being-able-to-think-like-a-new-user>", where the people who are most knowledgeable about the technology (committers) can’t see the shortcomings because they can’t experience what it’s like not to know and be new to the concept. Therefore *YOUR* feedback is very valuable.
Jarek, I am thinking of creating the inventory of the Airflow documentation and maybe there we can mark documentation pages that need to be updated and the ones that need to be created from scratch, also have a record of who is working/can work on each of them. WDYT? On Mon, Jan 13, 2020 at 7:38 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Oh yeah. We are definitely on a track to improve documentation. When I look > back at what we had even half a year ago, we are on a really good track. > > @Jacob @Anton (and others - especially newcomers) - maybe you could list > also other areas/concepts/ that you found documentation is > lacking/unclear? > > It would be great to start building such a list and fix it. For committers > a lot of the concepts are clear because they know it by heart, and they > simply do not realise there are missing things. So while comitters are best > to understand how things work, the users are best to point out the > problems. > > Anyone? > > J. > > On Mon, Jan 13, 2020 at 2:59 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Hi Jacob, > > > > I added the automation around Documenting all the Airflow configs. As a > > next step, I am going to add documentation around each of these configs. > > > > Will open some PRs shortly :) around this. > > > > Regards, > > Kaxil > > > > On Mon, Jan 13, 2020 at 12:21 PM Jacob Ward <jw...@brandwatch.com> > wrote: > > > > > As someone who is interested in contributing, but has not been able to > > yet, > > > I agree massively with Tomasz. There is still so much I need to > > understand > > > about how Airflow works and pretty much everything I have learned is > from > > > trial and error. While I do agree that a certain amount of testing is > > > helpful in understanding a technology, there is so much lacking in the > > > documentation that I often resort to code diving, or worse guesswork, > > when > > > trying to understand a new concept. This not only raises the barrier > for > > > entry for using Airflow and developing for Airflow, but I also feel it > > > reduces my confidence in being able to commit as there is so much of > the > > > codebase which just seems mysterious and unknown to me. > > > > > > Take for example the config options. Only recently has someone made a > > > documentation page which explains every option, but this itself is very > > > lacking; it has no more information than the comments on the default > > > airflow config file. Improving and expanding the documentation can be > > best > > > done by those who understand Airflow the best; committers. It's a great > > > investment in development as it accelerates the learning for new and > > > intermediate users, who can sooner become contributing members of the > > > community. > > > > > > Just a few thoughts that I hope will be considered. > > > > > > Jacob > > > > > > On Sun, 29 Dec 2019 at 18:50, Tomasz Urbaszek < > > tomasz.urbas...@polidea.com > > > > > > > wrote: > > > > > > > Hi all, > > > > > > > > In my opinion we area great community but we are not "Airflow". If > > > someone > > > > finds us, then his or her troubles are probably solved. At least I > hope > > > so. > > > > But last survey result includes a few really important > "non-technical" > > > > points: > > > > > > > > When onboarding new members to Airflow, what is the biggest problem? > > > > - No guide on best practises on developing DAGs 51.95% > > > > - Small number of tutorials on different aspects of using Airflow > > 18.51% > > > > In your opinion, what could be improved in Airflow? > > > > - Examples, how-to, onboarding documentation 46.43% > > > > - Technical documentation 44.48% > > > > > > > > These are "users opinions" but I would also give them a +1. I think > > that > > > to > > > > encourage > > > > people to contribute to Airflow we frist should encourage to use the > > > tool. > > > > Then, > > > > when people know how to and how not to use Airflow they will be able > to > > > > spot real > > > > bugs and groundbreaking features. A user who contributes is a great > > value > > > > and > > > > it's something we should encourage. > > > > > > > > Airflow is a huge project and the easier it will be to understand how > > it > > > > works, the easier > > > > it will be to contribute. After 6 months there are still things that > I > > do > > > > not fully understand. > > > > I think our new website is a tremendous opportunity. > > > > > > > > On the other hand, I fully support the idea of meetups. I really like > > the > > > > quite new > > > > Apache Local Community[1] idea. It's something that should bring ASF > > and > > > > OSS > > > > closer to students, devs, business people and anyone who's > interested. > > > > Moreover, > > > > it's something that will bring value to our communities because those > > > > meetings > > > > will be a great opportunity to exchange experiences between projects. > > > > > > > > T. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/COMDEV/Apache+Local+Community+-+ALC > > > > > > > > On Sun, Dec 29, 2019 at 7:12 PM Felix Uellendall > > <felue...@pm.me.invalid > > > > > > > > wrote: > > > > > > > > > Hey Jarek, > > > > > > > > > > I really like the points you bring up. While reading your mail I > > > thought > > > > > about the same things. For me at the beginning it was really hard > to > > > get > > > > > into this community and how everything works mostly because of the > > > > > language. I am not sure but maybe it would also be a good idea to > > > > organize > > > > > meetings in native languages first for people who do not feel > > > comfortable > > > > > speaking/reading/writing in English - just to help them to feel > > > > > comfortable. And after they got a basic understanding of the whole > > > > process > > > > > of contribution and how the project works they can start learning > > > English > > > > > if they want to join the community. > > > > > > > > > > Of course English is a requirement but I think people would be more > > > > > willing to improve their English if they have no other barriers > left > > > that > > > > > are keeping them from joining this community. > > > > > > > > > > Felix > > > > > > > > > > Sent from ProtonMail Mobile > > > > > > > > > > On Sun, Dec 29, 2019 at 18:37, Jarek Potiuk < > > jarek.pot...@polidea.com> > > > > > wrote: > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > TL; DR; I wanted to start a non-technical discussion about being > > > (even > > > > > > more) welcoming community. > > > > > > > > > > > > It's a long read - following some deep discussions I had recently > > and > > > > you > > > > > > might not be interested in it, so feel free to skip the entirety > of > > > it. > > > > > > > > > > > > I also believe this might become quickly a controversial topic > and > > > > > > mis-communication over email can easily happen - so I would like > to > > > ask > > > > > > everyone to be considerate and open-minded when responding. > > > > > > > > > > > > *Some context - how welcoming are we now ?* > > > > > > > > > > > > First of all I think we are doing a lot as community to be really > > > > > welcoming > > > > > > and friendly. A lot that we do is really opening up in various > ways > > > to > > > > > new > > > > > > community members, users, existing contributors etc. We are > > > responsive, > > > > > > helpful, we try to actively reach-out to get users opinions (the > > > > survey). > > > > > > We are open to invite non-code-committers to get "committer > status" > > > > > (that's > > > > > > highly encouraged by the Apache Software Foundation!) or even PMC > > > > members > > > > > > (yeah!). We organise events (Meetups and upcoming Airflow > Summits), > > > > > > workshops for users and new contributors. We are making it easier > > for > > > > new > > > > > > contributors to start contributing - by environment and > > documentation > > > > > > improvements. > > > > > > > > > > > > At the same time we have certain expectations/barrier of entry. > > It's > > > > not > > > > > > super easy to join the community and you must really earn your > > status > > > > to > > > > > > become a committer/PMC member. I think we are fairly good as a > > > > community > > > > > in > > > > > > enforcing that in deliberate and firm ways - and all this without > > > being > > > > > > rude or aggressive. I remember one of the first emails when I > > joined > > > > the > > > > > > community where I was firmly but friendly reminded that in this > > > > community > > > > > > decisions are made by the community and not a bunch of people > > talking > > > > at > > > > > > slack and agreeing to something between them. That was a very > > > important > > > > > > lesson to me - and first trigger to learn what ApacheWay is. And > it > > > was > > > > > > super cool even though I felt I have to apologize for my lack of > > > > > > understanding how this all works (which I did). > > > > > > > > > > > > We have certain expectations for PRs/code - some enforced > > > > automatically, > > > > > > some by comments/discussions/review process. And we have > > expectations > > > > for > > > > > > engagement of people submitting the code. They are supposed to > > > > follow-up > > > > > > their PRs - being responsible to get the PRs to submission and > > engage > > > > > > committers when they need it. We also encourage people not only > to > > > > > > finger-point things to fix but also engage and help with fixing > > > things > > > > > they > > > > > > find or even improve the processes. > > > > > > > > > > > > I think it's rather good mixture of openness/barrier of entry. > When > > > > > someone > > > > > > new joins any community - has to first adapt and show how they > can > > be > > > > > > valuable for the community before he or she can influence the way > > > > > community > > > > > > works. So it's great that there are firm boundaries and > > expectations > > > > and > > > > > > that we clearly explain them to anyone that tries to join and we > > > expect > > > > > > those people to follow the expectations before we invite them > > further > > > > > after > > > > > > they "earned" the status. This is best described in the > > "meritocracy" > > > > > rule > > > > > > defined here: > > > > > > https://www.apache.org/foundation/how-it-works.html#meritocracy > . > > We > > > > are > > > > > > following it really well I think. > > > > > > > > > > > > I believe in many ways we are much better than a number of other > > > > > > open-source communities and we are following ApacheWay fairly > well. > > > And > > > > > > I've heard personally a member of the board of the Apache > Software > > > > > > Foundation praising how welcoming Apache Airflow community is. > > > > > > > > > > > > *So why the discussion at all if we are in such good shape ?* > > > > > > > > > > > > I just wanted to see if we can do better than that - and whether > we > > > > need > > > > > to > > > > > > do better currently at all. > > > > > > > > > > > > I think it's fairly easy to overlook the moment when we should do > > > > > something > > > > > > more. Maybe we can change something to be even more welcoming. > > Maybe > > > we > > > > > can > > > > > > get people engaged who currently do not engage because it is too > > > > > difficult? > > > > > > Maybe we miss another point of view because of that? Maybe some > of > > > the > > > > > > rules we have should be updated? Maybe people who feel excluded > do > > > not > > > > > > speak here because they feel the barrier of entry is too big and > > they > > > > are > > > > > > afraid they will not be heard or will be ignored or will be > shouted > > > > at. I > > > > > > think it's better to discuss such things when everything looks > > great > > > > and > > > > > > when there is a good "vibe" in the community rather than being > > > > triggered > > > > > by > > > > > > people complaining after it becomes a problem and when the "vibe" > > > > > > deteriorates. > > > > > > > > > > > > The trigger for my thoughts was a looong discussion I had with > one > > of > > > > the > > > > > > attendees of PyDataWarsaw conference a few weeks ago at the > > > > after-party. > > > > > We > > > > > > talked for several hours I think, and we were the last ones to > > leave > > > > the > > > > > > party grounds (yes it was 3 am or so :D ). The person I spoke to > > > > raised a > > > > > > few important topics - like "not everyone has enough courage to > > > openly > > > > > > speak at the discussion list first" or "unconsciously people are > > > > valuing > > > > > > less contributions by women" (there is a study confirming that > > > > > > > > > > > > > > > > > > > > > https://www.theguardian.com/technology/2016/feb/12/women-considered-better-coders-hide-gender-github > > > > > ) > > > > > > and "some people need a kind of mentorship when they enter new > > > > community > > > > > > and after the introduction they become great contributors" - and > he > > > had > > > > > > some really good examples for all those statements from his own > > > > > experience. > > > > > > After the discussion he read about Apache Way (as I advised him), > > > > looked > > > > > at > > > > > > our discussions and he wrote to me a few days ago that he sees > how > > > > > > welcoming we are and that we are addressing a lot of the concerns > > he > > > > has > > > > > in > > > > > > really good way - but nevertheless it stuck with me a bit and I > > > > thought - > > > > > > maybe he is right that we should discuss it. > > > > > > > > > > > > For example - while we have two women on the PMC member list, > > almost > > > > all > > > > > > the people committing the code are male (I believe). This - of > > > course - > > > > > > reflects the state of our industry and is nothing new, but maybe > we > > > are > > > > > > (unconsciously) doing something in our discussions in devlistt or > > > slack > > > > > or > > > > > > reviews that puts off people who otherwise would be valuable to > our > > > > > > community? The friend of mine who triggered my thinking had a > great > > > > point > > > > > > that not everyone new has the courage to speak openly at the > > devlist > > > or > > > > > > slack initially. Maybe we should reach out in a different way to > > > those > > > > > > people? Or maybe we should think about some kind of mentorship > for > > > new > > > > > > people so that we can guide people through the first stages of > > > becoming > > > > > > contributors and navigate the way our community works? > > > > > > > > > > > > It looks like we already have people from all over the world - > US, > > > > > Europe, > > > > > > India, Japan, Australia, China. We have meetups in almost all of > > > those > > > > > > places. But maybe we could do more to get more people > > > > contributing/users > > > > > > invited from some places (for example we have no meetups in China > > yet > > > > and > > > > > > not a lot of people from South America I think). Again - maybe we > > can > > > > do > > > > > > something about it ?. I know there was an event in Mexico where > we > > > had > > > > > > Airflow workshop - maybe we can reach out to people there somehow > > :) > > > ? > > > > > > There was also a great presentation about Chinese user community > at > > > the > > > > > > ApacheCon Europe few months ago > > > > > > > > > > > > > > > > > > > > > https://aceu19.apachecon.com/session/inviting-apache-flinks-chinese-user-community > > > > > > on > > > > > > how difficult it is to get people in China contributing because > of > > > the > > > > > > language barrier. Maybe we should get more workshops for new > > > > contributors > > > > > > in Chinese/Mandarin in China initially and get some contributors > > from > > > > > there > > > > > > (writing description of a PR might be easier even for someone who > > has > > > > > > difficulties speaking english or you can have someone who will be > > > your > > > > > > local mentor for that). > > > > > > > > > > > > I do not have concrete proposals yet, or I do not ask you to have > > > them > > > > > > immediately. I don't even know yet if we should do something or > > not. > > > > But > > > > > I > > > > > > wanted to open up discussion to hear what others think about it - > > > both > > > > > > active members of our community and those who are just listening > > and > > > > > rarely > > > > > > discuss. > > > > > > > > > > > > Maybe we are really in a good state and we should just continue? > Or > > > > maybe > > > > > > there are some easy things we can do as a community to get better > > at > > > > > being > > > > > > more welcoming ? Also maybe we should forward the discussion > > > elsewhere > > > > > > (users@?/Slack?/Meetups?) so that others who are not reading the > > > > devlist > > > > > > can chime in ? > > > > > > > > > > > > I'd really love to hear what others think about it! > > > > > > > > > > > > Again - please be considerate and open-minded - this might > quickly > > > > > become a > > > > > > controversial subject and miscommunication is almost certain, so > > > let's > > > > > all > > > > > > be careful with words and statements. > > > > > > > > > > > > J. > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > <+48%20660%20796%20129>> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > -- > > > > > > > > Tomasz Urbaszek > > > > Polidea <https://www.polidea.com/> | Software Engineer > > > > > > > > M: +48 505 628 493 <+48%20505%20628%20493> <+48505628493 > <+48%20505%20628%20493>> > > > > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > > > > > > > Unique Tech > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > <+48%20660%20796%20129>> > [image: Polidea] <https://www.polidea.com/> >
