Hi Mike, Yes, I won¹t hesitate to ask your help. Thank you as always.
Hi Cam and everyone, I would like to clarify that the time wasting discussion is not about the project workflow at all. I am sorry, but I have been too busy to read any email thread on the workflow. The discussion in my email is about the github pull requests. https://github.com/apache/climate/pull/196 https://github.com/apache/climate/pull/197 I wish I do not have to stay up until 2 AM just to explain simple seasonal mean calculation repeatedly. Kyo On 5/12/15, 10:54 PM, "Michael Joyce" <jo...@apache.org> wrote: >Beyond my comments below I'm stepping off this thread. It's become too >toxic for my liking and most of the "discussion" on here is helping >nothing. > >I think getting feedback on our current workflow would be great. If you >have ideas anyone please do respond (or maybe just make a new thread) so >we >can talk about stuff. It's important that we make it work for us! > >@Kyo, I've said it a million times, but it's important to say it again. If >you need help with anything please don't hesitate to reach out. I'm always >happy to help out when I can. You always have this list, you know where my >office is, you know what my phone number is, and you have me on IM. If >you're ever unsure about something or just need an extra pair of eyes to >find a bug please ask. > >-- Jimmy > >On Tue, May 12, 2015 at 9:53 PM, Mattmann, Chris A (3980) < >chris.a.mattm...@jpl.nasa.gov> wrote: > >> Lewis, >> >> -----Original Message----- >> >> From: Lewis John Mcgibbney <lewis.mcgibb...@gmail.com> >> Reply-To: "dev@climate.apache.org" <dev@climate.apache.org> >> Date: Tuesday, May 12, 2015 at 8:29 PM >> To: "dev@climate.apache.org" <dev@climate.apache.org> >> Subject: Re: Project Workflow >> >> >Hi Chris, >> >Please see replies inline >> > >> >On Tue, May 12, 2015 at 6:10 PM, <dev-digest-h...@climate.apache.org> >> >wrote: >> > >> >> >> >> I¹m honestly not sure what you are talking about, >> > >> > >> >Not even one wee bit ;) maybe some of my replies will bridge the gap >> >between you not being sure about what I am talking about Vs disagreeing >> >with what I am trying to say. >> >> Yes me saying I don¹t know what you are talking about is my way >> of saying I disagree with you. That clear? >> >> > >> > >> >> and to be honest, >> >> waiting 72 hours before committing everything is not a project I¹d >> >> ever want to be on. >> > >> > >> >Well lets for a minute consider the flip side (or another possible >>side) >> >of >> >this coin. I would never want to be on a project where patches are >>merged >> >which clearly break > 50% of the tests in the entire codebase. >> >> Please, provide me a citation for this? Which PR has done >> that? Be specific too. Don¹t say ³the PMC². Use names, and PR #s. >> > >Not to speak for Lewis but I assume >https://github.com/apache/climate/pull/195 >Although that PR seems to perfectly represent receiving good feedback and >making quick changes to issues that broke the build and getting everything >resolved. Seemed like great collaboration to me. > > >> >> >It seems to >> >me, rather detrimental to the project codebase as well as other >>community >> >members who have confidence in a codebase for patches to be committed >>and >> >merged whcih essentially break everything! Additionally, this goes >>against >> >all of the good practice I've ever learned since stumbling across >>TheASF >> >some 6 years ago! I can't word this opinion any other way. >> >> Who in the world is suggesting we merge things that break 50% of the >>code? >> I¹m suggesting *we* actually merge things. Instead of having one person >> merge >> things. And mostly have others talk about how they want to merge things >>and >> then have one person end up merging things. After lots of discussion. >> > >Saying it again for the billionth time. With regards to helping merge >stuff, please do. > > >> [..snip..] >> > >> > >> >> It shouldn¹t be *imposed on me*. >> > >> > >> >It seems like imposed is a strong word given the sentiment of the >>thread >> >and the openness of Mike to open it up initially to how people want to >>do >> >things. I think what we are trying to determine here is whether people >> >feel >> >like things are being imposed upon them. If that ends up being one of >>the >> >outcomes of this thread then we need to accept, address, change, >>implement >> >and move on. >> >> I¹m raising it, b/c honestly, this PMC has failed to educate at least >>one >> of its members on his rights. And I am telling you that as the Champion >>of >> this project and the person who even suggested bringing it here, it¹s a >> collective >> issue. I did my part to try and educate that person today at the coffee >> cart, >> by telling Kyo this at JPL - I also noted this on the recent PR #: >> >> http://s.apache.org/to1 > > >Kyo was on the original team when the project went into incubator correct? >Seems like that certainly should have been brought up at some point. Don't >know if it's fair to say that the PMC failed to educate someone who has >been on the PMC since it started. I wouldn't think to check that with >people honestly. Perhaps we should make sure the current list of people >are >aware of what's going on if this is an ongoing concern? There's certainly >a >decent number of people on the PMC who aren't from a software background >so >that might be useful. > > >> >> >> >> >I consider a grace period as a politeness as well as a duration which >> >people can gauge the contribution(s) and comment accordingly. That is >>all. >> >It is not so people can shoot it down. That would be detrimental >>indeed. >> >> Realize in CTR - they can do the same thing. We are using a version >>control >> system. Since when has code committed to a version control system been >> something >> that¹s difficult to revert? You can do the exact same thing in CTR. With >> less >> barrier. Want more barrier, RTC? Use it then. Then later do CTR. It¹s >>easy. >> >> > >> > >> >> BTW Apache projects and their >> >> conversation need to happen at the ASF and I¹m seriously concerned >> >> that¹s not happening here. There is too much reliance on Github >> >> for this project. >> >> >> > >> >I understand what you are saying here Chris. There is a lot of >>development >> >chat going on at Github. This is on an issue-by-issue bases AFAIK >>however, >> >therefore I am of the opinion that essentially this is no different >>from >> >the same conversation happening over on Jira and the same messages >>being >> >shadowed over onto dev@. >> >> You¹re conflating the issue. This isn¹t a question of Github versus >>JIRA. >> This is a question of {Github, JIRA, auto whatever} versus {regular old >> dev mail}. >> >> > The reason I say that is that (with my experiences >> >of mentoring Apache Usergrid) communities should not be *forced* to use >> >Jira over Github. >> >> I¹m not suggesting that. >> >> > The same messges are shadowed to Jira and to the Mailing >> >Lists. This is a nuance of the communication workflow. If this is a >>major >> >issue (we've been here before haven't we ;)) then it needs to be dealt >> >with. This argument has a lot of precedence at the foundation and we >>can >> >dig it up if need be. >> >What is your suggestion then Chris? That all correspondence is moved to >> >Jira? That it happens on the ML? That we find a balance between the >>three? >> >> Balance of course. Important development decisions and discussions >>should >> not be in a Pull Request conversation - it should be on the dev list, >>not >> surrounding by a message saying ³Github notification from user XXXYYY². >> Small discussion around an issue on Github? Tests passed from a bot? >> Sure leave that on Github. ³I don¹t think X is a good implementation >>over >> Y² - that¹s dev list conversation *not* surrounded and marshaled by an >> automated bot. >> >> > >> > >> >> >> >> Yes. Flat out. We don¹t VOTE 72 hours on every line of code, or every >> >> patch, >> >> or waiting for a grace period to commit things. >> > >> > >> >There was no mention of VOTE'ing at all AFAIK. All commentary thus far >>has >> >referred to a 72 window for community commentary that was all. After >>this >> >72 hours (not months) it is absolutely cool to commit away. BTW, it is >> >also >> >cool to commit away before those 72 hours. >> >> Please read what you just said. I have to wait 72 hours, then it¹s cool. >> But >> before 72 hours it¹s cool too. Huh? >> > >The 72 hours is a cursory period that I said was my personal metric. >Please >check the OP again. There seems to be some confusion about this. > > >> >> > There is no Bylaws established >> >by OCW to state anything any different. This combined with the >>pre-commit >> >build for the project has kept builds stable on OCW for as long as I >>can >> >remember. >> >> Guess who suggested making that pre-commit build and told MikeJ to look >>at >> the way Spark did it? You¹re not telling me anything I don¹t already >>know >> Lewis. >> >> >It seems to be doing a reasonable job at keeping the test suite >> >stable and passing successfully. I would have thought that this >>practice >> >would have been fine given the fact that comments typically come in >>before >> >72 hours and I've seen a bunch of patches committed before 72 hours as >> >well. >> >> Citations? Were they self patches from Mike to himself? Were they Mike >>and >> Kim >> who clearly seem to agree with each other - Kim committing Mike¹s >>patches >> or >> vice versa? > > >> > >> >[..snip..] >> > >> > >> >> If I want to add a test. >> >> Update >> >> something that isn¹t being used in the code base, or that doesn¹t >>even >> >>have >> >> tests to show how it works one way or another? Someone should be >>able to >> >> press >> >> forward on that. Releases? 72 hours. New PMC/committers? 72 hours. A >>new >> >> JIRA ticket, >> >> etc.? Not sure we need that. >> >> >> > >> >I think there is an issue here though. It's not this type of thing >>that 72 >> >hours is in place for. It's proposed patches which break the build and >> >test >> >suite and mean that others working off of master cannot keep up to date >> >with master. That is what the workflow guards against. Someone please >> >correct me if I am wrong. >> >> What is to guard against? That¹s my question. What precisely are we >> guarding >> against? Do we not trust the version control system? What users have >> complained >> about a broken build thus far? Please name a user, not a developer. >> >> > >> >[..snip..] >> > >> > >> >> And Kyo not even knowing that >> >> he has direct commit access to the Git repo at the ASF >> > >> > >> >That's a clear failure of the PMC and Kyo's introduction to the PMC in >> >communicating to Kyo that he has commit rights to the canonical source >> >code >> >and that he can essentially commit whenever and whatever he likes. That >> >seriously needs to be addressed. >> >> That¹s the problem - no he can¹t. You can¹t have it both ways. If PMC >> has a 72 hour wait period then there is a barrier to him committing >> whenever >> he likes. If he must discuss everything there is a barrier to him >> committing >> whatever he likes. >> >> So, which is it? >> > >Open PR, wait a reasonable amount of time for people to check stuff out, >merge it. That's the workflow. "Reasonable amount of time" is at the >discretion of the person getting on and deciding to merge stuff. 72 hours >was my personal goto that was listed as an example in the OP. > > >> Here¹s my proposal. We start *TRUSTING* each other, and start >> doing things and moving forward instead of worrying about PR discussion >> and broken tests, and pointing people to read workflow wiki pages (that >> you seem to be citing, but have stated you haven¹t read later on in this >> email) that reference Github as the canonical source. >> > >Again. Where does it say this? Because I just read through it again and >can't find what you're talking about. > > >> > >> > >> >> , and honestly a >> >> guide >> >> that I was pointed to that says the primary source code base for the >> >> project >> >> is at Github (newsflash: it¹s not - that¹s where our *mirror* is). >> >> >> > >> >I've never seen it so no comment. I am aware that canonical source is >>at >> >Apache. >> >> Huh? You are citing the ³workflow² earlier - have you read it or not? >> >> > >> >[..snip..] >> > >> > >> >> Lewis as a member of the foundation >> >> I¹m sure you¹re privy to the recent strife and discussion related to >> >>this >> >> over >> >> the years >> > >> > >> >Absolutely I am. I've also however seen extremely successful >> >Apache-compliant Github workflows dramatically increasing development >>and >> >interest in codebases. Usergrid is an excellent example of that. We >>really >> >struggled over there for a number of months with an entire PPMC nearly >> >opting to leave the Incubator due to what they saw as ridiculous >> >constraints upon what Infra wanted them to do. My justification for >> >getting >> >involved in this thread is because I felt I learned a lot from that >> >experience and I hope I can help out here! >> >> This is not a Github problem. It¹s a community problem, that includes >> issues with Github, amongst others. Github has worked great on Nutch and >> Tika - and in certain interaction types that I¹m not seeing on OCW >>lately, >> it could work well too. I¹m calling out things I don¹t think are working >> well. >> >> > >> > >> >> - a project¹s sole source of activity and conversation related to >> >> development cannot be automated emails from bots. We should probably >>be >> >> discussing dev related stuff in emails. That¹s still the lowest >>common >> >> denominator. >> >> >> > >> >I checked out the mailing list archives and almost every message is >> >generated automatically. So your point is utterly valid. I would ask if >> >you >> >if you think disassociating all contextual development communication >>from >> >Jira or any other issue tracker is a wise thing to do? >> >> Strawman. >> You: Are you citing a problem with X, so let¹s do ^X. >> >> Ever heard of gradients that lie in-between? >> >> >[..snip..] >> > >> > >> >> I see PRs from Kyo getting many suggested >> >> revisions from Whitehall and Joyce - and then I see similarly some >> >>issues >> >> when they try and push code. I see Mike being the guy to integrate >>and >> >>push >> >> PRs after review (his own and other people¹s). That¹s scarily like a >> >>BDFL. >> >> >> > >> >I don't see it this way. I would be pretty convinced that Mike can >>speak >> >up >> >here and state that he would wish others to commit their own patches. >> >> >The instructions on how to merge are in that wiki document that is >constantly cited in this thread right below how to make a pull request. I >haven't seen any emails about being confused so I'm not certain why people >would assume people are confused? > > >> They would if they knew how, and weren¹t afraid that the PR they filed >> would get 5-6 suggestions before committing a simple date/time function. >> C¹mon. >> OCW is not Nutch. It¹s not Hadoop. Let¹s use Software Metrics and do a >>full >> SLOC count and let¹s put it Ohloh and see how many proposed developer >>hours >> have been invested in it versus Nutch and/or Hadoop and/or Tika, etc. >>It¹s >> not >> even close. >> > >Also, OCW is already on Ohloh but they have some issues with their repo >updates. I gave up a long time ago trying to get them to refresh their >stuff. Unfortunately they weren't very responsive. All the login info was >sent to private@ a long time ago I'm pretty sure. If you cant find it let >me know, I'm sure it's around somewhere. > > >> >> You have to be able to adapt in projects and to be able to lower the >> barrier >> when necessary - I absolutely understand in Nutch having 1 or 2 >>iterations >> sometimes on patches and on PRs and stuff. We have 5-6 interested >> developers >> doing things right now - and iterating and PRs and stuff and >>conversation >> is >> a great way in a distributed team across timezones and so forth to work >> together. >> We have nowhere near that situation with OCW right now. The active >>people >> working on >> OCW are all in the same time zone; 100% all work for the same company, >>and >> of which there is 1 primary active developer; with 1 or 2 occasional >> developers >> in terms of contributions that submit things that are merged by the 1 >> primary >> active developer. >> >> >> > >> > >> >> Yes I said it. Is Mike the merge master? >> > >> > >> >Not at all. If you check out my other thread I sent in reply to Kyo, >>there >> >are 29 Committers with write access to the codebase and 42 subscribers >>to >> >this list in all. >> >> Red-herring. >> >> They are not contributing to the project right now. >> >> >> >That is a hellish impressive number of people to have on >> >the PMC. The truth though Chris is that is it were not for Mike then I >> >would have had nothing committed to the codebase! >> >> This ignores the heritage of contributions that existed long >> before Mike Joyce. >> >> >[..snip..] >> > >> > >> >> He is a PMC >> >> member and I¹m sorry and not going to dance around the issue anymore >>- >> >>he¹s >> >> not being treated like a PMC member. And I¹m bringing it up and not >> >> sweeping >> >> it under the carpet. >> >> >> > >> >This is a foreign concept to me and I was not aware that Kyo was not >>being >> >treated as a PMC member. Unless I have missed this corresponence >>happening >> >on list then I was not aware of it. >> >> It¹s all on the Github - treating people like we trust them involves >>taking >> a leap of faith sometimes on what they are doing. And trusting it¹ll be >>OK. >> That¹s my impression reading the Github. You may disagree. >> >> >[..snip..] >> > >> > >> >> He¹s finding it difficult to contribute. >> > >> > >> >Well we need to help with that. That is the only way forward. The >>mailing >> >list is open for this kind of communication. Frustrations are all too >> >often >> >voiced here which is probably too late. I would like to call out Kyo >> >personally and state that if you read this, and you are having an issue >> >contributing to the OCW codebase over and above community comments then >> >please let us know. We can either sit down one-to-one, in company or >>else >> >the PMC can maybe even do a casual hangout and we can resole the >>committer >> >issues. >> >I would like to also point you to the following resources Kyo >> >http://apache.org/dev/contributors.html#providingfeedback >> >In particular it details how to submit patches and provides guidance >>for >> >what those patches should minimally do. I quote >> >" >> > >> >change the sourcefiles to incorporate your change or addition. Make >>sure >> >you also provide appropriate source code documentation (like javadoc >>for >> >java sources), and follow a project's coding conventions. >> > >> >check the software still compiles and runs correctly >> > >> >run any unit or regression tests the software may have >> >" >> >If there is any part of this or anything else which we can aid with >>then >> >please let us know on the list. >> >> >> Stop citing good software coding practices. It¹s community >> over code here at the ASF. I studied software architecture and >> development and understand plenty about it. >> >> I want to reduce Kyo¹s frustration about contributing to the codebase. >> He has stated all the discussion before being able to make a simple >>commit >> causes his frustration. I agree with him and think there is too much >> discussion >> before a commit. I¹m a PMC member on the project. >> >> > >> > >> >> >> >> I do. It¹s too long. >> > >> > >> >Too long for what? Is development on OCW time sensitive? >> > >> > >> >> Please let me know the sacred vow that breaking >> >> a test on OCW causes. >> > >> > >> >Non at all. It is development code afterall and hey that is what source >> >code management systems are for right? Sh*t I am all for breaking >>project >> >builds I done it many times before or many projects. I am however also >>for >> >taking on board a Jenkins message which tells me I've broken a number >>of >> >tests. >> >> Again, please tell how I am suggesting directly to ignore Jenkins >> considering >> I suggested to have Mike copy the Spark one and create it. Please tell >>me >> precisely how I am suggesting to ignore Jenkins? I¹m suggesting to trust >> each other. HUGE difference. >> >> > >> > >> >> We have 0 users of the project. We are our own >> >> users. >> > >> > >> >This is not true Chris I am sorry. I was only talking to someone today >>who >> >mentioned a recently committed module is being used by foreign >> >researchers. >> >> Where are they in the last 9 months on the list? The last year? >> If they are out there, I would welcome them here. Right now I can safely >> say >> having been on this project for a long time and monitoring the list that >> not a >> single non JPL email has been sent to the list in months related to any >> external >> users of the software. >> >> >> >A number of us have made efforts to grow the Climate user base and I >>think >> >we need to be very sure that no-one is using the codebase before >>stating >> >explicitly that no-one does. >> >> Please make a list of known users of the code base that aren¹t using >> Jinwon¹s >> forked version. I would seriously like to know, having been one of the >> principal >> people to help start this project, and also present on it >>internationally >> for >> years before it even came to the ASF. >> >> If you have such a list that isn¹t students who aren¹t on the list, I >>would >> love to see it. Users bring desire for releases, and new features, and >>use >> cases >> and I haven¹t seen any of that at all brought on to this list. There is >>NO >> ONE >> that would love more users for OCW than me. Period. I brought the >>project >> here, >> don¹t suggest for a second and I take extreme offense to your attempt to >> try >> and educate me on the users. If you know about the users, please >>produce a >> list >> and don¹t speak in generalities. >> >> >> > >> > >> >> I¹m going to make a scary suggestion. Push all the code! >> > >> > >> >Really ;) >> > >> > >> >> Do >> >> things! Talk on the dev list. Figure out how not to piss >> >> people like Kyo off and gain their contributions even if it means >> >> breaking some tests, compromising (Kyo too), but people on all sides. >> >> >> > >> >Chris I think we are all for doing things. I hope we are getting to a >> >stage >> >now where we will unearth what it is that is pissing Kyo off. >> >> Lewis, newsflash, I¹ve unearthed it. See all my emails and commentary. >>See >> Loikith¹s email. That¹s why. That¹s why I¹m bringing this up. >> >> >> > We all have >> >an obligation as part of the PMC to support his with any committer and >> >developer resources he requires. This includes potentially guiding him >> >towards development neutral lists such as community@/community-dev@, >>etc. >> >I >> >can't provide help to someone if I don't understand where the >> >pain/frustration stems from. >> >> I do. For all the reasons stated above. >> >> > >> > >> >> >> >> Sorry but PMC lead is no more special than any PMC member. >> > >> > >> >No-one said he was AFAIK. >> >> direct quote from you is that Mike was the ³PMC Lead². So not sure >> why you used that term - so I was following from that. >> >> > My point was that Mike was engaged in a primary >> >development role of OCW. Whereas others were/are not. >> >> Bingo. Which is precisely my problem with this. One developer engaged >> in the primary development role of a project at Apache is a serious >> concern. >> >> > >> > >> >> The person >> >> has an added responsibility of filing a board report and being the >> >> eyes and the ears of the board. I haven¹t seen it come across that >>there >> >> is a concern that he¹s merging everything. I have a concern. I¹m >> >>bringing >> >> it up. >> >> >> > >> >As you are entirely entitled to do. We are by no means at crisis point >> >here. There are issues which need addressing with regards to this >>topic. >> >I've TBH never seen a PMC with 29 members and only 1 person merging >>code. >> >> This is a precise concern point, among others that I¹ve raised. >> >> > >> > >> >> This PMC will have succeeded when Kyo Lee has merged and committed >>his >> >>own >> >> code to the repo. It will succeed when Mike¹s not committing >>everyone¹s >> >> PRs. >> >> >> > >> >Sounds like a sure aim to me. It is not difficult. All Kyo needs to do >>is >> >as follows >> > >> >$ cd climate >> >$ git remote add apache >> >https://git-wip-us.apache.org/repos/asf/climate.git >> >$ git push apache $branch >> >> You missed the part about 5-6 conversations and comments with two other >> people >> that seem to agree to disagree with him. Or to suggest that he do 5-6 >> things >> before he commit the code. With a project with this low activity, >>isolated >> to >> a single developer, it¹s a recipe for disaster. I¹m wondering how you >>are >> having >> a hard time seeing that? >> >> > >> >> >> >> > >> >> >I would suggest that we augment the workflow to accomodate a thirst >> >>step >> >> > >> >> >3) Please make best efforts to at least consider other community >> >>comments >> >> >before merging. This way we can work collaboratively to all have a >> >>better >> >> >understanding of the codebase. >> >> > >> > >> > >> >Do you Chris or does anyone else have an opinion or suggestion for >> >augmenting the workflow or abolishing it altogether as the above thread >> >would suggest? >> >I think regardsless of what we end up deciding upon, we need to have >>it in >> >black and white on the wiki. It seems like this has become a major pain >> >point for Kyo less a major concern for Chris as a champion of the >>project >> >and ongoing mentor. >> >> Uhhh, yes. I¹ve suggested it. >> >> Be flexible. Trust each other. YMMV. >> >> >I would like to work together to establish a resolution. Kyo, it would >> >mean >> >A LOT if you were part of this. This also goes for the other ~25 PMC >> >members. OCW is a significant imbalance in the community with regards >>to >> >development Vs ML correspondence and physical ML presence. An >>observation >> >and inference on my part is that this is the result of many people >> >basically not having the time and or cycles to actively discuss or >>develop >> >OCW in line with their day-to-day operations. This is not due to an >>overly >> >convoluted commit process. >> >> Disagree. It¹s a combination of both. The way you can attract new >> contributors >> is to revisit and deal with a process that has any barriers (even 1 >> barrier) >> and try and have none. Some Apache projects accept patches today from a >> mailing >> list (HTTPD) - still to this day - and credit those contributors without >> giving >> them 5 or 6 elements in the ³Bring me a rock game² for them to do before >> committing >> their patch. >> >> Code can be continuously adapted and updated (CTR); can be peer reviewed >> before >> commital (RTC); and these models can work and do work well and nicely >>with >> one >> another. >> >> >> >The fact that Kyo is struggling is our primary >> >cause for concern as loosing a valuable community member is literally a >> >disaster no matter what the community is and how many individuals are >> >associated with the community. >> >> It¹s even more than that. Kyo, as a PMC member, is also indicative of >>the >> larger set of scientists that need to use OCW and to moreover to >> contribute to it. >> He¹s having problems doing both. >> >> Cheers, >> Chris >> >>
