Hi Jacob, I added the automation around Documenting all the Airflow configs. As a next step, I am going to add documentation around each of these configs.
Will open some PRs shortly :) around this. Regards, Kaxil On Mon, Jan 13, 2020 at 12:21 PM Jacob Ward <jw...@brandwatch.com> wrote: > As someone who is interested in contributing, but has not been able to yet, > I agree massively with Tomasz. There is still so much I need to understand > about how Airflow works and pretty much everything I have learned is from > trial and error. While I do agree that a certain amount of testing is > helpful in understanding a technology, there is so much lacking in the > documentation that I often resort to code diving, or worse guesswork, when > trying to understand a new concept. This not only raises the barrier for > entry for using Airflow and developing for Airflow, but I also feel it > reduces my confidence in being able to commit as there is so much of the > codebase which just seems mysterious and unknown to me. > > Take for example the config options. Only recently has someone made a > documentation page which explains every option, but this itself is very > lacking; it has no more information than the comments on the default > airflow config file. Improving and expanding the documentation can be best > done by those who understand Airflow the best; committers. It's a great > investment in development as it accelerates the learning for new and > intermediate users, who can sooner become contributing members of the > community. > > Just a few thoughts that I hope will be considered. > > Jacob > > On Sun, 29 Dec 2019 at 18:50, Tomasz Urbaszek <tomasz.urbas...@polidea.com > > > wrote: > > > Hi all, > > > > In my opinion we area great community but we are not "Airflow". If > someone > > finds us, then his or her troubles are probably solved. At least I hope > so. > > But last survey result includes a few really important "non-technical" > > points: > > > > When onboarding new members to Airflow, what is the biggest problem? > > - No guide on best practises on developing DAGs 51.95% > > - Small number of tutorials on different aspects of using Airflow 18.51% > > In your opinion, what could be improved in Airflow? > > - Examples, how-to, onboarding documentation 46.43% > > - Technical documentation 44.48% > > > > These are "users opinions" but I would also give them a +1. I think that > to > > encourage > > people to contribute to Airflow we frist should encourage to use the > tool. > > Then, > > when people know how to and how not to use Airflow they will be able to > > spot real > > bugs and groundbreaking features. A user who contributes is a great value > > and > > it's something we should encourage. > > > > Airflow is a huge project and the easier it will be to understand how it > > works, the easier > > it will be to contribute. After 6 months there are still things that I do > > not fully understand. > > I think our new website is a tremendous opportunity. > > > > On the other hand, I fully support the idea of meetups. I really like the > > quite new > > Apache Local Community[1] idea. It's something that should bring ASF and > > OSS > > closer to students, devs, business people and anyone who's interested. > > Moreover, > > it's something that will bring value to our communities because those > > meetings > > will be a great opportunity to exchange experiences between projects. > > > > T. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/COMDEV/Apache+Local+Community+-+ALC > > > > On Sun, Dec 29, 2019 at 7:12 PM Felix Uellendall <felue...@pm.me.invalid > > > > wrote: > > > > > Hey Jarek, > > > > > > I really like the points you bring up. While reading your mail I > thought > > > about the same things. For me at the beginning it was really hard to > get > > > into this community and how everything works mostly because of the > > > language. I am not sure but maybe it would also be a good idea to > > organize > > > meetings in native languages first for people who do not feel > comfortable > > > speaking/reading/writing in English - just to help them to feel > > > comfortable. And after they got a basic understanding of the whole > > process > > > of contribution and how the project works they can start learning > English > > > if they want to join the community. > > > > > > Of course English is a requirement but I think people would be more > > > willing to improve their English if they have no other barriers left > that > > > are keeping them from joining this community. > > > > > > Felix > > > > > > Sent from ProtonMail Mobile > > > > > > On Sun, Dec 29, 2019 at 18:37, Jarek Potiuk <jarek.pot...@polidea.com> > > > wrote: > > > > > > > Hello everyone, > > > > > > > > TL; DR; I wanted to start a non-technical discussion about being > (even > > > > more) welcoming community. > > > > > > > > It's a long read - following some deep discussions I had recently and > > you > > > > might not be interested in it, so feel free to skip the entirety of > it. > > > > > > > > I also believe this might become quickly a controversial topic and > > > > mis-communication over email can easily happen - so I would like to > ask > > > > everyone to be considerate and open-minded when responding. > > > > > > > > *Some context - how welcoming are we now ?* > > > > > > > > First of all I think we are doing a lot as community to be really > > > welcoming > > > > and friendly. A lot that we do is really opening up in various ways > to > > > new > > > > community members, users, existing contributors etc. We are > responsive, > > > > helpful, we try to actively reach-out to get users opinions (the > > survey). > > > > We are open to invite non-code-committers to get "committer status" > > > (that's > > > > highly encouraged by the Apache Software Foundation!) or even PMC > > members > > > > (yeah!). We organise events (Meetups and upcoming Airflow Summits), > > > > workshops for users and new contributors. We are making it easier for > > new > > > > contributors to start contributing - by environment and documentation > > > > improvements. > > > > > > > > At the same time we have certain expectations/barrier of entry. It's > > not > > > > super easy to join the community and you must really earn your status > > to > > > > become a committer/PMC member. I think we are fairly good as a > > community > > > in > > > > enforcing that in deliberate and firm ways - and all this without > being > > > > rude or aggressive. I remember one of the first emails when I joined > > the > > > > community where I was firmly but friendly reminded that in this > > community > > > > decisions are made by the community and not a bunch of people talking > > at > > > > slack and agreeing to something between them. That was a very > important > > > > lesson to me - and first trigger to learn what ApacheWay is. And it > was > > > > super cool even though I felt I have to apologize for my lack of > > > > understanding how this all works (which I did). > > > > > > > > We have certain expectations for PRs/code - some enforced > > automatically, > > > > some by comments/discussions/review process. And we have expectations > > for > > > > engagement of people submitting the code. They are supposed to > > follow-up > > > > their PRs - being responsible to get the PRs to submission and engage > > > > committers when they need it. We also encourage people not only to > > > > finger-point things to fix but also engage and help with fixing > things > > > they > > > > find or even improve the processes. > > > > > > > > I think it's rather good mixture of openness/barrier of entry. When > > > someone > > > > new joins any community - has to first adapt and show how they can be > > > > valuable for the community before he or she can influence the way > > > community > > > > works. So it's great that there are firm boundaries and expectations > > and > > > > that we clearly explain them to anyone that tries to join and we > expect > > > > those people to follow the expectations before we invite them further > > > after > > > > they "earned" the status. This is best described in the "meritocracy" > > > rule > > > > defined here: > > > > https://www.apache.org/foundation/how-it-works.html#meritocracy . We > > are > > > > following it really well I think. > > > > > > > > I believe in many ways we are much better than a number of other > > > > open-source communities and we are following ApacheWay fairly well. > And > > > > I've heard personally a member of the board of the Apache Software > > > > Foundation praising how welcoming Apache Airflow community is. > > > > > > > > *So why the discussion at all if we are in such good shape ?* > > > > > > > > I just wanted to see if we can do better than that - and whether we > > need > > > to > > > > do better currently at all. > > > > > > > > I think it's fairly easy to overlook the moment when we should do > > > something > > > > more. Maybe we can change something to be even more welcoming. Maybe > we > > > can > > > > get people engaged who currently do not engage because it is too > > > difficult? > > > > Maybe we miss another point of view because of that? Maybe some of > the > > > > rules we have should be updated? Maybe people who feel excluded do > not > > > > speak here because they feel the barrier of entry is too big and they > > are > > > > afraid they will not be heard or will be ignored or will be shouted > > at. I > > > > think it's better to discuss such things when everything looks great > > and > > > > when there is a good "vibe" in the community rather than being > > triggered > > > by > > > > people complaining after it becomes a problem and when the "vibe" > > > > deteriorates. > > > > > > > > The trigger for my thoughts was a looong discussion I had with one of > > the > > > > attendees of PyDataWarsaw conference a few weeks ago at the > > after-party. > > > We > > > > talked for several hours I think, and we were the last ones to leave > > the > > > > party grounds (yes it was 3 am or so :D ). The person I spoke to > > raised a > > > > few important topics - like "not everyone has enough courage to > openly > > > > speak at the discussion list first" or "unconsciously people are > > valuing > > > > less contributions by women" (there is a study confirming that > > > > > > > > > > https://www.theguardian.com/technology/2016/feb/12/women-considered-better-coders-hide-gender-github > > > ) > > > > and "some people need a kind of mentorship when they enter new > > community > > > > and after the introduction they become great contributors" - and he > had > > > > some really good examples for all those statements from his own > > > experience. > > > > After the discussion he read about Apache Way (as I advised him), > > looked > > > at > > > > our discussions and he wrote to me a few days ago that he sees how > > > > welcoming we are and that we are addressing a lot of the concerns he > > has > > > in > > > > really good way - but nevertheless it stuck with me a bit and I > > thought - > > > > maybe he is right that we should discuss it. > > > > > > > > For example - while we have two women on the PMC member list, almost > > all > > > > the people committing the code are male (I believe). This - of > course - > > > > reflects the state of our industry and is nothing new, but maybe we > are > > > > (unconsciously) doing something in our discussions in devlistt or > slack > > > or > > > > reviews that puts off people who otherwise would be valuable to our > > > > community? The friend of mine who triggered my thinking had a great > > point > > > > that not everyone new has the courage to speak openly at the devlist > or > > > > slack initially. Maybe we should reach out in a different way to > those > > > > people? Or maybe we should think about some kind of mentorship for > new > > > > people so that we can guide people through the first stages of > becoming > > > > contributors and navigate the way our community works? > > > > > > > > It looks like we already have people from all over the world - US, > > > Europe, > > > > India, Japan, Australia, China. We have meetups in almost all of > those > > > > places. But maybe we could do more to get more people > > contributing/users > > > > invited from some places (for example we have no meetups in China yet > > and > > > > not a lot of people from South America I think). Again - maybe we can > > do > > > > something about it ?. I know there was an event in Mexico where we > had > > > > Airflow workshop - maybe we can reach out to people there somehow :) > ? > > > > There was also a great presentation about Chinese user community at > the > > > > ApacheCon Europe few months ago > > > > > > > > > > https://aceu19.apachecon.com/session/inviting-apache-flinks-chinese-user-community > > > > on > > > > how difficult it is to get people in China contributing because of > the > > > > language barrier. Maybe we should get more workshops for new > > contributors > > > > in Chinese/Mandarin in China initially and get some contributors from > > > there > > > > (writing description of a PR might be easier even for someone who has > > > > difficulties speaking english or you can have someone who will be > your > > > > local mentor for that). > > > > > > > > I do not have concrete proposals yet, or I do not ask you to have > them > > > > immediately. I don't even know yet if we should do something or not. > > But > > > I > > > > wanted to open up discussion to hear what others think about it - > both > > > > active members of our community and those who are just listening and > > > rarely > > > > discuss. > > > > > > > > Maybe we are really in a good state and we should just continue? Or > > maybe > > > > there are some easy things we can do as a community to get better at > > > being > > > > more welcoming ? Also maybe we should forward the discussion > elsewhere > > > > (users@?/Slack?/Meetups?) so that others who are not reading the > > devlist > > > > can chime in ? > > > > > > > > I'd really love to hear what others think about it! > > > > > > > > Again - please be considerate and open-minded - this might quickly > > > become a > > > > controversial subject and miscommunication is almost certain, so > let's > > > all > > > > be careful with words and statements. > > > > > > > > J. > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > > > Tomasz Urbaszek > > Polidea <https://www.polidea.com/> | Software Engineer > > > > M: +48 505 628 493 <+48505628493> > > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > > > Unique Tech > > Check out our projects! <https://www.polidea.com/our-work> > > >
