That's great to hear Kaxil, I am looking forward to seeing these improvements!
@Jarek, I'm happy to have a think and make a list of the areas of the documentation I find most lacking. Would you like me to post that here, or if we open a Jira ticket would it be better to have it commented there? Jacob On Mon, 13 Jan 2020 at 15:38, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Oh yeah. We are definitely on a track to improve documentation. When I look > back at what we had even half a year ago, we are on a really good track. > > @Jacob @Anton (and others - especially newcomers) - maybe you could list > also other areas/concepts/ that you found documentation is > lacking/unclear? > > It would be great to start building such a list and fix it. For committers > a lot of the concepts are clear because they know it by heart, and they > simply do not realise there are missing things. So while comitters are best > to understand how things work, the users are best to point out the > problems. > > Anyone? > > J. > > On Mon, Jan 13, 2020 at 2:59 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Hi Jacob, > > > > I added the automation around Documenting all the Airflow configs. As a > > next step, I am going to add documentation around each of these configs. > > > > Will open some PRs shortly :) around this. > > > > Regards, > > Kaxil > > > > On Mon, Jan 13, 2020 at 12:21 PM Jacob Ward <jw...@brandwatch.com> > wrote: > > > > > As someone who is interested in contributing, but has not been able to > > yet, > > > I agree massively with Tomasz. There is still so much I need to > > understand > > > about how Airflow works and pretty much everything I have learned is > from > > > trial and error. While I do agree that a certain amount of testing is > > > helpful in understanding a technology, there is so much lacking in the > > > documentation that I often resort to code diving, or worse guesswork, > > when > > > trying to understand a new concept. This not only raises the barrier > for > > > entry for using Airflow and developing for Airflow, but I also feel it > > > reduces my confidence in being able to commit as there is so much of > the > > > codebase which just seems mysterious and unknown to me. > > > > > > Take for example the config options. Only recently has someone made a > > > documentation page which explains every option, but this itself is very > > > lacking; it has no more information than the comments on the default > > > airflow config file. Improving and expanding the documentation can be > > best > > > done by those who understand Airflow the best; committers. It's a great > > > investment in development as it accelerates the learning for new and > > > intermediate users, who can sooner become contributing members of the > > > community. > > > > > > Just a few thoughts that I hope will be considered. > > > > > > Jacob > > > > > > On Sun, 29 Dec 2019 at 18:50, Tomasz Urbaszek < > > tomasz.urbas...@polidea.com > > > > > > > wrote: > > > > > > > Hi all, > > > > > > > > In my opinion we area great community but we are not "Airflow". If > > > someone > > > > finds us, then his or her troubles are probably solved. At least I > hope > > > so. > > > > But last survey result includes a few really important > "non-technical" > > > > points: > > > > > > > > When onboarding new members to Airflow, what is the biggest problem? > > > > - No guide on best practises on developing DAGs 51.95% > > > > - Small number of tutorials on different aspects of using Airflow > > 18.51% > > > > In your opinion, what could be improved in Airflow? > > > > - Examples, how-to, onboarding documentation 46.43% > > > > - Technical documentation 44.48% > > > > > > > > These are "users opinions" but I would also give them a +1. I think > > that > > > to > > > > encourage > > > > people to contribute to Airflow we frist should encourage to use the > > > tool. > > > > Then, > > > > when people know how to and how not to use Airflow they will be able > to > > > > spot real > > > > bugs and groundbreaking features. A user who contributes is a great > > value > > > > and > > > > it's something we should encourage. > > > > > > > > Airflow is a huge project and the easier it will be to understand how > > it > > > > works, the easier > > > > it will be to contribute. After 6 months there are still things that > I > > do > > > > not fully understand. > > > > I think our new website is a tremendous opportunity. > > > > > > > > On the other hand, I fully support the idea of meetups. I really like > > the > > > > quite new > > > > Apache Local Community[1] idea. It's something that should bring ASF > > and > > > > OSS > > > > closer to students, devs, business people and anyone who's > interested. > > > > Moreover, > > > > it's something that will bring value to our communities because those > > > > meetings > > > > will be a great opportunity to exchange experiences between projects. > > > > > > > > T. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/COMDEV/Apache+Local+Community+-+ALC > > > > > > > > On Sun, Dec 29, 2019 at 7:12 PM Felix Uellendall > > <felue...@pm.me.invalid > > > > > > > > wrote: > > > > > > > > > Hey Jarek, > > > > > > > > > > I really like the points you bring up. While reading your mail I > > > thought > > > > > about the same things. For me at the beginning it was really hard > to > > > get > > > > > into this community and how everything works mostly because of the > > > > > language. I am not sure but maybe it would also be a good idea to > > > > organize > > > > > meetings in native languages first for people who do not feel > > > comfortable > > > > > speaking/reading/writing in English - just to help them to feel > > > > > comfortable. And after they got a basic understanding of the whole > > > > process > > > > > of contribution and how the project works they can start learning > > > English > > > > > if they want to join the community. > > > > > > > > > > Of course English is a requirement but I think people would be more > > > > > willing to improve their English if they have no other barriers > left > > > that > > > > > are keeping them from joining this community. > > > > > > > > > > Felix > > > > > > > > > > Sent from ProtonMail Mobile > > > > > > > > > > On Sun, Dec 29, 2019 at 18:37, Jarek Potiuk < > > jarek.pot...@polidea.com> > > > > > wrote: > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > TL; DR; I wanted to start a non-technical discussion about being > > > (even > > > > > > more) welcoming community. > > > > > > > > > > > > It's a long read - following some deep discussions I had recently > > and > > > > you > > > > > > might not be interested in it, so feel free to skip the entirety > of > > > it. > > > > > > > > > > > > I also believe this might become quickly a controversial topic > and > > > > > > mis-communication over email can easily happen - so I would like > to > > > ask > > > > > > everyone to be considerate and open-minded when responding. > > > > > > > > > > > > *Some context - how welcoming are we now ?* > > > > > > > > > > > > First of all I think we are doing a lot as community to be really > > > > > welcoming > > > > > > and friendly. A lot that we do is really opening up in various > ways > > > to > > > > > new > > > > > > community members, users, existing contributors etc. We are > > > responsive, > > > > > > helpful, we try to actively reach-out to get users opinions (the > > > > survey). > > > > > > We are open to invite non-code-committers to get "committer > status" > > > > > (that's > > > > > > highly encouraged by the Apache Software Foundation!) or even PMC > > > > members > > > > > > (yeah!). We organise events (Meetups and upcoming Airflow > Summits), > > > > > > workshops for users and new contributors. We are making it easier > > for > > > > new > > > > > > contributors to start contributing - by environment and > > documentation > > > > > > improvements. > > > > > > > > > > > > At the same time we have certain expectations/barrier of entry. > > It's > > > > not > > > > > > super easy to join the community and you must really earn your > > status > > > > to > > > > > > become a committer/PMC member. I think we are fairly good as a > > > > community > > > > > in > > > > > > enforcing that in deliberate and firm ways - and all this without > > > being > > > > > > rude or aggressive. I remember one of the first emails when I > > joined > > > > the > > > > > > community where I was firmly but friendly reminded that in this > > > > community > > > > > > decisions are made by the community and not a bunch of people > > talking > > > > at > > > > > > slack and agreeing to something between them. That was a very > > > important > > > > > > lesson to me - and first trigger to learn what ApacheWay is. And > it > > > was > > > > > > super cool even though I felt I have to apologize for my lack of > > > > > > understanding how this all works (which I did). > > > > > > > > > > > > We have certain expectations for PRs/code - some enforced > > > > automatically, > > > > > > some by comments/discussions/review process. And we have > > expectations > > > > for > > > > > > engagement of people submitting the code. They are supposed to > > > > follow-up > > > > > > their PRs - being responsible to get the PRs to submission and > > engage > > > > > > committers when they need it. We also encourage people not only > to > > > > > > finger-point things to fix but also engage and help with fixing > > > things > > > > > they > > > > > > find or even improve the processes. > > > > > > > > > > > > I think it's rather good mixture of openness/barrier of entry. > When > > > > > someone > > > > > > new joins any community - has to first adapt and show how they > can > > be > > > > > > valuable for the community before he or she can influence the way > > > > > community > > > > > > works. So it's great that there are firm boundaries and > > expectations > > > > and > > > > > > that we clearly explain them to anyone that tries to join and we > > > expect > > > > > > those people to follow the expectations before we invite them > > further > > > > > after > > > > > > they "earned" the status. This is best described in the > > "meritocracy" > > > > > rule > > > > > > defined here: > > > > > > https://www.apache.org/foundation/how-it-works.html#meritocracy > . > > We > > > > are > > > > > > following it really well I think. > > > > > > > > > > > > I believe in many ways we are much better than a number of other > > > > > > open-source communities and we are following ApacheWay fairly > well. > > > And > > > > > > I've heard personally a member of the board of the Apache > Software > > > > > > Foundation praising how welcoming Apache Airflow community is. > > > > > > > > > > > > *So why the discussion at all if we are in such good shape ?* > > > > > > > > > > > > I just wanted to see if we can do better than that - and whether > we > > > > need > > > > > to > > > > > > do better currently at all. > > > > > > > > > > > > I think it's fairly easy to overlook the moment when we should do > > > > > something > > > > > > more. Maybe we can change something to be even more welcoming. > > Maybe > > > we > > > > > can > > > > > > get people engaged who currently do not engage because it is too > > > > > difficult? > > > > > > Maybe we miss another point of view because of that? Maybe some > of > > > the > > > > > > rules we have should be updated? Maybe people who feel excluded > do > > > not > > > > > > speak here because they feel the barrier of entry is too big and > > they > > > > are > > > > > > afraid they will not be heard or will be ignored or will be > shouted > > > > at. I > > > > > > think it's better to discuss such things when everything looks > > great > > > > and > > > > > > when there is a good "vibe" in the community rather than being > > > > triggered > > > > > by > > > > > > people complaining after it becomes a problem and when the "vibe" > > > > > > deteriorates. > > > > > > > > > > > > The trigger for my thoughts was a looong discussion I had with > one > > of > > > > the > > > > > > attendees of PyDataWarsaw conference a few weeks ago at the > > > > after-party. > > > > > We > > > > > > talked for several hours I think, and we were the last ones to > > leave > > > > the > > > > > > party grounds (yes it was 3 am or so :D ). The person I spoke to > > > > raised a > > > > > > few important topics - like "not everyone has enough courage to > > > openly > > > > > > speak at the discussion list first" or "unconsciously people are > > > > valuing > > > > > > less contributions by women" (there is a study confirming that > > > > > > > > > > > > > > > > > > > > > https://www.theguardian.com/technology/2016/feb/12/women-considered-better-coders-hide-gender-github > > > > > ) > > > > > > and "some people need a kind of mentorship when they enter new > > > > community > > > > > > and after the introduction they become great contributors" - and > he > > > had > > > > > > some really good examples for all those statements from his own > > > > > experience. > > > > > > After the discussion he read about Apache Way (as I advised him), > > > > looked > > > > > at > > > > > > our discussions and he wrote to me a few days ago that he sees > how > > > > > > welcoming we are and that we are addressing a lot of the concerns > > he > > > > has > > > > > in > > > > > > really good way - but nevertheless it stuck with me a bit and I > > > > thought - > > > > > > maybe he is right that we should discuss it. > > > > > > > > > > > > For example - while we have two women on the PMC member list, > > almost > > > > all > > > > > > the people committing the code are male (I believe). This - of > > > course - > > > > > > reflects the state of our industry and is nothing new, but maybe > we > > > are > > > > > > (unconsciously) doing something in our discussions in devlistt or > > > slack > > > > > or > > > > > > reviews that puts off people who otherwise would be valuable to > our > > > > > > community? The friend of mine who triggered my thinking had a > great > > > > point > > > > > > that not everyone new has the courage to speak openly at the > > devlist > > > or > > > > > > slack initially. Maybe we should reach out in a different way to > > > those > > > > > > people? Or maybe we should think about some kind of mentorship > for > > > new > > > > > > people so that we can guide people through the first stages of > > > becoming > > > > > > contributors and navigate the way our community works? > > > > > > > > > > > > It looks like we already have people from all over the world - > US, > > > > > Europe, > > > > > > India, Japan, Australia, China. We have meetups in almost all of > > > those > > > > > > places. But maybe we could do more to get more people > > > > contributing/users > > > > > > invited from some places (for example we have no meetups in China > > yet > > > > and > > > > > > not a lot of people from South America I think). Again - maybe we > > can > > > > do > > > > > > something about it ?. I know there was an event in Mexico where > we > > > had > > > > > > Airflow workshop - maybe we can reach out to people there somehow > > :) > > > ? > > > > > > There was also a great presentation about Chinese user community > at > > > the > > > > > > ApacheCon Europe few months ago > > > > > > > > > > > > > > > > > > > > > https://aceu19.apachecon.com/session/inviting-apache-flinks-chinese-user-community > > > > > > on > > > > > > how difficult it is to get people in China contributing because > of > > > the > > > > > > language barrier. Maybe we should get more workshops for new > > > > contributors > > > > > > in Chinese/Mandarin in China initially and get some contributors > > from > > > > > there > > > > > > (writing description of a PR might be easier even for someone who > > has > > > > > > difficulties speaking english or you can have someone who will be > > > your > > > > > > local mentor for that). > > > > > > > > > > > > I do not have concrete proposals yet, or I do not ask you to have > > > them > > > > > > immediately. I don't even know yet if we should do something or > > not. > > > > But > > > > > I > > > > > > wanted to open up discussion to hear what others think about it - > > > both > > > > > > active members of our community and those who are just listening > > and > > > > > rarely > > > > > > discuss. > > > > > > > > > > > > Maybe we are really in a good state and we should just continue? > Or > > > > maybe > > > > > > there are some easy things we can do as a community to get better > > at > > > > > being > > > > > > more welcoming ? Also maybe we should forward the discussion > > > elsewhere > > > > > > (users@?/Slack?/Meetups?) so that others who are not reading the > > > > devlist > > > > > > can chime in ? > > > > > > > > > > > > I'd really love to hear what others think about it! > > > > > > > > > > > > Again - please be considerate and open-minded - this might > quickly > > > > > become a > > > > > > controversial subject and miscommunication is almost certain, so > > > let's > > > > > all > > > > > > be careful with words and statements. > > > > > > > > > > > > J. > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > -- > > > > > > > > Tomasz Urbaszek > > > > Polidea <https://www.polidea.com/> | Software Engineer > > > > > > > > M: +48 505 628 493 <+48505628493> > > > > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > > > > > > > Unique Tech > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
