Re: Project management (JIRA fix version tracking) is in crisis

2021-01-19 Thread Andrew Purtell
PRs can be opened for backports and cherry picks and you can have all of the desired precommit reports available before any merging happens. Publishing changes should be relatively atomic. (Is "relatively" better? I mean on the same day.) The JIRAs that linger open in a half open state with some

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-19 Thread Nick Dimiduk
It sounds like we have rough consensus at least on removing the CHANGES and RELEASENOTES files from source control. I filed HBASE-25520 to track this work. > What is the common thing here? That the merge action is atomic! All are ready or none are ready! Don't stuff half done work in some

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-18 Thread Andrew Purtell
Thank you Wellington. We do need for committers to pay attention to JIRA. We still use JIRA for project management so we can't ignore it or allow it to get out of sync with reality anyway. As long as we keep fix versions in sync, we can continue to use today's create-release script and Yetus' nice

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-18 Thread Wellington Chevreuil
While no perfect script/tool is available to solve this problem, we may agree on a preferable commiting process to make RM's life less difficult. My take from this thread is that something like below would help: 1) Once a commit happens on any branch, resolve the jira and update fixVersions field

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-17 Thread Duo Zhang
What about a Jenkins job for detecting the diff between jira and git? And I agree that, let's not include CHANGES.md and RELEASENOTES.md in our code base directly which means we need to sink an RC only if these two files are incorrect, not the actual code. Andrew Purtell 于2021年1月16日周六

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Andrew Purtell
I think the discussion has already moved forward a lot with mention of scripts I did not know existed. One well developed git<->jira diff tool, documented as part of the RM procedure, would be much better than we have right now, with various RMs doing ad hoc things. On Fri, Jan 15, 2021 at 6:04

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Duo Zhang
Haven't fully read all the comments but I think this is the duty for the release manager? This is what I use to get the difference between jira and git git log --oneline 0167558eb31ff48308d592ef70b6d005ba6d21fb...3ee8b0c75b | grep -o -E "^[0-9a-z]*\s*HBASE-[0-9]*"|awk '{print $2}' | sort -u >

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Sean Busbey
What would the logistics look like for git as the source of truth? Every release candidate I've ever done I've had to do the jira and git reconciliation. There are always errors in both. Folks commit stuff with the wrong JIRA ID or wrong message. Would reconciliation for a RM then require revert

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Stack
On Fri, Jan 15, 2021 at 10:06 AM Andrew Purtell wrote: > I have now had to sink two 2.4.1 RCs because of errors in the change log. > > I made a pass over git history and ensured every commit was included. I had > also made a pass over JIRA to move out any unresolved issues or complete > the

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Andrew Purtell
I basically agree with all of this, except: - I believe committers are going to have a hard time with fix version management and we should not trust this - Git as source of truth was discussed in the Yetus context IIRC but docmaker uses the JIRA API, so JIRA remains the effective source of

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Nick Dimiduk
If it's possible, I would also appreciate it if we could add a pre-commit hook to our repository that rejects a commit whose title does not match our prescribed pattern. Doing so would go a long way toward simplifying the parsing tools. -n On Fri, Jan 15, 2021 at 3:18 PM Nick Dimiduk wrote: >

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Nick Dimiduk
> I have now had to sink two 2.4.1 RCs because of errors in the change log. IIRC, it's a relatively new phenomenon that we sink an RC because the CHANGES file is incomplete. > That's right... Commtters cannot be trusted to correctly maintain issue metadata in JIRA. I wouldn't go so far as to

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Nick Dimiduk
Regarding a tool, I’ve already committed one that I used for 2.3.0. IIRC, Busbey suggested on its original PR the idea of using it during our nightlies as well. I’m pretty sure I’ve mentioned it explicit already to Viraj, Andrew, and Huaxiang. Check dev-support/git-jira-release-audit on master.

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Huaxiang Sun
Yeah, after multiple release efforts of 2.3.4 failed, I finally did a manual check. 1. got git logs for jiras from 2.3.3. 2. did a dry-run to generate CHANGES.md. I compared these two outputs and removed the common ones. The one left in 1) are issues in git and fixVersion not correctly setup in

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Nick Dimiduk
Hi Andrew, I had a conversation on this same topic yesterday with Huaxiang and, I believe, came to a similar conclusion. I was advocating that git be the source of truth and Jira follow, and there be no CHANGES file in the repository at all. I’m on mobile at the moment. Let me spend some time

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Viraj Jasani
Thanks Andrew and Zach. The script tries to identify these issues: 1. commit is reverted as per commit message 2. commit does not contain Jira number format HBASE- in message 3. Jira does not have expected fixVersion 4. Jira has expected fixVersion, but it is not yet resolved 5. Jiras that

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Zach York
Awesome Viraj! Also if we have confidence in the script we could probably add it to run during a nightly run to try to catch issues earlier and remove the burden on the RM (if there isn't too much noise from open issues). On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell wrote: > Thanks, Viraj. A

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Andrew Purtell
Thanks, Viraj. A script like that will be super helpful. On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani wrote: > > This is not something RMs have had to do > > in the past and it asks a lot of them. > > +1. It is too much for RM to do if RM alone is responsible for > maintaining Git/Jira in

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Viraj Jasani
> This is not something RMs have had to do > in the past and it asks a lot of them. +1. It is too much for RM to do if RM alone is responsible for maintaining Git/Jira in sync. The reason why I was able to get the diffs for 2.3.4 and 2.4.1 RCs is because I used a small script that a) goes

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Andrew Purtell
A related factor is the practice of holding a JIRA open until all backports are complete. This is something RMs will already be familiar with, so isn't more than an inconvenience, but it does lend itself to fix version inaccuracy. Say you merge a PR or commit a cherry pick to some of the active

Re: Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Andrew Purtell
We do not have a single source of truth as to what are active branches and what was the most recent release by version number made from each branch. If we had such a resource, then committers could refer to it whenever merging PRs or cherry picking commits backward. Perhaps something can be

Project management (JIRA fix version tracking) is in crisis

2021-01-15 Thread Andrew Purtell
I have now had to sink two 2.4.1 RCs because of errors in the change log. I made a pass over git history and ensured every commit was included. I had also made a pass over JIRA to move out any unresolved issues or complete the resolution of same. What I did not do is check that every resolved