Re: Project management (JIRA fix version tracking) is in crisis

[Andrew Purtell]

I think the discussion has already moved forward a lot with mention of
scripts I did not know existed. One well developed git<->jira diff tool,
documented as part of the RM procedure, would be much better than we have
right now, with various RMs doing ad hoc things.


On Fri, Jan 15, 2021 at 6:04 PM 张铎(Duo Zhang)  wrote:

> Haven't fully read all the comments but I think this is the duty for the
> release manager? This is what I use to get the difference between jira and
> git
>
> git log --oneline 0167558eb31ff48308d592ef70b6d005ba6d21fb...3ee8b0c75b |
> grep -o -E "^[0-9a-z]*\s*HBASE-[0-9]*"|awk '{print $2}' | sort -u >
> in_git_2.0.0.txt
> cat CHANGELOG.md | grep -o -E "\[HBASE-[0-9]*\]" | grep -o -E
> "HBASE-[0-9]*" | sort -u > CHANGELOG_jiras.txt
>
> And then you could use comm to get the difference.
>
> We all make mistakes so this process can not be automatic, trust me.
> Sometimes a commit is reverted and never applied again, sometimes we
> include the wrong jira id in commit message.
> This is why I always use a jira issue to land the CHANGES and RELEASENOTES,
> tag it, and then use the release script to generate the RC.
>
> For me, it will cost me a lot of time when creating a new minor release
> because there will be a lot of commits, but for a patch release usually it
> is fine.
>
> Maybe things will be easier if we move all the development process to
> github?
>
> Thanks.
>
> Sean Busbey  于2021年1月16日周六 上午8:05写道：
>
> > What would the logistics look like for git as the source of truth?
> >
> > Every release candidate I've ever done I've had to do the jira and git
> > reconciliation. There are always errors in both. Folks commit stuff with
> > the wrong JIRA ID or wrong message.
> >
> > Would reconciliation for a RM then require revert and reapply of any such
> > changes?
> >
> > On Fri, Jan 15, 2021, 17:54 Stack  wrote:
> >
> > > On Fri, Jan 15, 2021 at 10:06 AM Andrew Purtell 
> > > wrote:
> > >
> > > > I have now had to sink two 2.4.1 RCs because of errors in the change
> > log.
> > > >
> > > > I made a pass over git history and ensured every commit was
> included. I
> > > had
> > > > also made a pass over JIRA to move out any unresolved issues or
> > complete
> > > > the resolution of same. What I did not do is check that every
> resolved
> > > JIRA
> > > > corresponded to an actual commit. This is not something RMs have had
> to
> > > do
> > > > in the past and it asks a lot of them.
> > > >
> > > >
> > > Just to say, that making RCs, I've the done the JIRA<->GIT
> reconciliation
> > > described (ugly hack scripting, sorting, and comparing). Others have
> too
> > > I've noticed. It is awful, yes, especially when issues like those found
> > by
> > > Viraj in yours and Huaxiangs RCs this week. The refguide is vague on
> what
> > > is involved; it needs more detail. The issue HBASE-22853 description on
> > the
> > > need for a tool to do the reconcile is a bit better.
> > >
> > >
> > >
> > > > I know NOW that as RM I cannot currently trust committers to get fix
> > > > versions right or care about this.
> > > >
> > > > That's right... Commtters cannot be trusted to correctly maintain
> issue
> > > > metadata in JIRA.
> > > >
> > > >
> > > Yes (says a key offender).
> > >
> > >
> > >
> > > > That is not a good situation for the project to be in. Up until now
> it
> > > has
> > > > not been the responsibility of the RM to check each and every JIRA
> > > status.
> > > > It has been the collective responsibility of committers to care about
> > the
> > > > project's release tracking insofar as to correctly update fix
> versions
> > in
> > > > JIRA. For releases containing relatively few changes, like 2.4.1,
> with
> > > ~50
> > > > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > > > versions, walk the commit history, and set back fix versions on JIRA
> to
> > > > actually correspond with what was truly committed. However, for minor
> > > > releases, with hundreds of commits, this will not be possible.
> > > >
> > > > I think the root cause is GitHub and JIRA are two separate change
> > > tracking
> > > > systems with only a minimal amount of integration. It requires manual
> > > > effort. More and more, new committers are familiar with GitHub and
> PRs
> > > and
> > > > are not familiar with JIRA and the Apache way of using JIRA to build
> > > change
> > > > logs. We need to better educate new and existing committers on their
> > > > responsibilities with regards to maintaining JIRA metadata correctly.
> > > >
> > > >
> > > Agree.
> > >
> > > Other issues are that the release process has been evolving. In the old
> > > days, JIRA was the source of truth; changelog was a JIRA report. The
> > > CHANGES.md/RELEASENOTES.md have come to the fore perhaps making
> > difference
> > > between JIRA and GIT more pronounced.
> > >
> > > I like the suggestion made by you above (and Nick in an internal
> > > discussion) that git be the source of truth. That the CHANGES.md NOT be
> > > 

Re: Project management (JIRA fix version tracking) is in crisis

[Duo Zhang]

Haven't fully read all the comments but I think this is the duty for the
release manager? This is what I use to get the difference between jira and
git

git log --oneline 0167558eb31ff48308d592ef70b6d005ba6d21fb...3ee8b0c75b |
grep -o -E "^[0-9a-z]*\s*HBASE-[0-9]*"|awk '{print $2}' | sort -u >
in_git_2.0.0.txt
cat CHANGELOG.md | grep -o -E "\[HBASE-[0-9]*\]" | grep -o -E
"HBASE-[0-9]*" | sort -u > CHANGELOG_jiras.txt

And then you could use comm to get the difference.

We all make mistakes so this process can not be automatic, trust me.
Sometimes a commit is reverted and never applied again, sometimes we
include the wrong jira id in commit message.
This is why I always use a jira issue to land the CHANGES and RELEASENOTES,
tag it, and then use the release script to generate the RC.

For me, it will cost me a lot of time when creating a new minor release
because there will be a lot of commits, but for a patch release usually it
is fine.

Maybe things will be easier if we move all the development process to
github?

Thanks.

Sean Busbey  于2021年1月16日周六 上午8:05写道：

> What would the logistics look like for git as the source of truth?
>
> Every release candidate I've ever done I've had to do the jira and git
> reconciliation. There are always errors in both. Folks commit stuff with
> the wrong JIRA ID or wrong message.
>
> Would reconciliation for a RM then require revert and reapply of any such
> changes?
>
> On Fri, Jan 15, 2021, 17:54 Stack  wrote:
>
> > On Fri, Jan 15, 2021 at 10:06 AM Andrew Purtell 
> > wrote:
> >
> > > I have now had to sink two 2.4.1 RCs because of errors in the change
> log.
> > >
> > > I made a pass over git history and ensured every commit was included. I
> > had
> > > also made a pass over JIRA to move out any unresolved issues or
> complete
> > > the resolution of same. What I did not do is check that every resolved
> > JIRA
> > > corresponded to an actual commit. This is not something RMs have had to
> > do
> > > in the past and it asks a lot of them.
> > >
> > >
> > Just to say, that making RCs, I've the done the JIRA<->GIT reconciliation
> > described (ugly hack scripting, sorting, and comparing). Others have too
> > I've noticed. It is awful, yes, especially when issues like those found
> by
> > Viraj in yours and Huaxiangs RCs this week. The refguide is vague on what
> > is involved; it needs more detail. The issue HBASE-22853 description on
> the
> > need for a tool to do the reconcile is a bit better.
> >
> >
> >
> > > I know NOW that as RM I cannot currently trust committers to get fix
> > > versions right or care about this.
> > >
> > > That's right... Commtters cannot be trusted to correctly maintain issue
> > > metadata in JIRA.
> > >
> > >
> > Yes (says a key offender).
> >
> >
> >
> > > That is not a good situation for the project to be in. Up until now it
> > has
> > > not been the responsibility of the RM to check each and every JIRA
> > status.
> > > It has been the collective responsibility of committers to care about
> the
> > > project's release tracking insofar as to correctly update fix versions
> in
> > > JIRA. For releases containing relatively few changes, like 2.4.1, with
> > ~50
> > > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > > versions, walk the commit history, and set back fix versions on JIRA to
> > > actually correspond with what was truly committed. However, for minor
> > > releases, with hundreds of commits, this will not be possible.
> > >
> > > I think the root cause is GitHub and JIRA are two separate change
> > tracking
> > > systems with only a minimal amount of integration. It requires manual
> > > effort. More and more, new committers are familiar with GitHub and PRs
> > and
> > > are not familiar with JIRA and the Apache way of using JIRA to build
> > change
> > > logs. We need to better educate new and existing committers on their
> > > responsibilities with regards to maintaining JIRA metadata correctly.
> > >
> > >
> > Agree.
> >
> > Other issues are that the release process has been evolving. In the old
> > days, JIRA was the source of truth; changelog was a JIRA report. The
> > CHANGES.md/RELEASENOTES.md have come to the fore perhaps making
> difference
> > between JIRA and GIT more pronounced.
> >
> > I like the suggestion made by you above (and Nick in an internal
> > discussion) that git be the source of truth. That the CHANGES.md NOT be
> > checked-in is also a good idea so it can be regenerated w/o need of a
> > change to the RC if JIRA is changed (I can work on this part if wanted).
> >
> > S
> >
> >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>

Re: Project management (JIRA fix version tracking) is in crisis

[Sean Busbey]

What would the logistics look like for git as the source of truth?

Every release candidate I've ever done I've had to do the jira and git
reconciliation. There are always errors in both. Folks commit stuff with
the wrong JIRA ID or wrong message.

Would reconciliation for a RM then require revert and reapply of any such
changes?

On Fri, Jan 15, 2021, 17:54 Stack  wrote:

> On Fri, Jan 15, 2021 at 10:06 AM Andrew Purtell 
> wrote:
>
> > I have now had to sink two 2.4.1 RCs because of errors in the change log.
> >
> > I made a pass over git history and ensured every commit was included. I
> had
> > also made a pass over JIRA to move out any unresolved issues or complete
> > the resolution of same. What I did not do is check that every resolved
> JIRA
> > corresponded to an actual commit. This is not something RMs have had to
> do
> > in the past and it asks a lot of them.
> >
> >
> Just to say, that making RCs, I've the done the JIRA<->GIT reconciliation
> described (ugly hack scripting, sorting, and comparing). Others have too
> I've noticed. It is awful, yes, especially when issues like those found by
> Viraj in yours and Huaxiangs RCs this week. The refguide is vague on what
> is involved; it needs more detail. The issue HBASE-22853 description on the
> need for a tool to do the reconcile is a bit better.
>
>
>
> > I know NOW that as RM I cannot currently trust committers to get fix
> > versions right or care about this.
> >
> > That's right... Commtters cannot be trusted to correctly maintain issue
> > metadata in JIRA.
> >
> >
> Yes (says a key offender).
>
>
>
> > That is not a good situation for the project to be in. Up until now it
> has
> > not been the responsibility of the RM to check each and every JIRA
> status.
> > It has been the collective responsibility of committers to care about the
> > project's release tracking insofar as to correctly update fix versions in
> > JIRA. For releases containing relatively few changes, like 2.4.1, with
> ~50
> > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > versions, walk the commit history, and set back fix versions on JIRA to
> > actually correspond with what was truly committed. However, for minor
> > releases, with hundreds of commits, this will not be possible.
> >
> > I think the root cause is GitHub and JIRA are two separate change
> tracking
> > systems with only a minimal amount of integration. It requires manual
> > effort. More and more, new committers are familiar with GitHub and PRs
> and
> > are not familiar with JIRA and the Apache way of using JIRA to build
> change
> > logs. We need to better educate new and existing committers on their
> > responsibilities with regards to maintaining JIRA metadata correctly.
> >
> >
> Agree.
>
> Other issues are that the release process has been evolving. In the old
> days, JIRA was the source of truth; changelog was a JIRA report. The
> CHANGES.md/RELEASENOTES.md have come to the fore perhaps making difference
> between JIRA and GIT more pronounced.
>
> I like the suggestion made by you above (and Nick in an internal
> discussion) that git be the source of truth. That the CHANGES.md NOT be
> checked-in is also a good idea so it can be regenerated w/o need of a
> change to the RC if JIRA is changed (I can work on this part if wanted).
>
> S
>
>
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: Project management (JIRA fix version tracking) is in crisis

[Stack]

On Fri, Jan 15, 2021 at 10:06 AM Andrew Purtell  wrote:

> I have now had to sink two 2.4.1 RCs because of errors in the change log.
>
> I made a pass over git history and ensured every commit was included. I had
> also made a pass over JIRA to move out any unresolved issues or complete
> the resolution of same. What I did not do is check that every resolved JIRA
> corresponded to an actual commit. This is not something RMs have had to do
> in the past and it asks a lot of them.
>
>
Just to say, that making RCs, I've the done the JIRA<->GIT reconciliation
described (ugly hack scripting, sorting, and comparing). Others have too
I've noticed. It is awful, yes, especially when issues like those found by
Viraj in yours and Huaxiangs RCs this week. The refguide is vague on what
is involved; it needs more detail. The issue HBASE-22853 description on the
need for a tool to do the reconcile is a bit better.



> I know NOW that as RM I cannot currently trust committers to get fix
> versions right or care about this.
>
> That's right... Commtters cannot be trusted to correctly maintain issue
> metadata in JIRA.
>
>
Yes (says a key offender).



> That is not a good situation for the project to be in. Up until now it has
> not been the responsibility of the RM to check each and every JIRA status.
> It has been the collective responsibility of committers to care about the
> project's release tracking insofar as to correctly update fix versions in
> JIRA. For releases containing relatively few changes, like 2.4.1, with ~50
> changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> versions, walk the commit history, and set back fix versions on JIRA to
> actually correspond with what was truly committed. However, for minor
> releases, with hundreds of commits, this will not be possible.
>
> I think the root cause is GitHub and JIRA are two separate change tracking
> systems with only a minimal amount of integration. It requires manual
> effort. More and more, new committers are familiar with GitHub and PRs and
> are not familiar with JIRA and the Apache way of using JIRA to build change
> logs. We need to better educate new and existing committers on their
> responsibilities with regards to maintaining JIRA metadata correctly.
>
>
Agree.

Other issues are that the release process has been evolving. In the old
days, JIRA was the source of truth; changelog was a JIRA report. The
CHANGES.md/RELEASENOTES.md have come to the fore perhaps making difference
between JIRA and GIT more pronounced.

I like the suggestion made by you above (and Nick in an internal
discussion) that git be the source of truth. That the CHANGES.md NOT be
checked-in is also a good idea so it can be regenerated w/o need of a
change to the RC if JIRA is changed (I can work on this part if wanted).

S


> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: Project management (JIRA fix version tracking) is in crisis

[Andrew Purtell]

I basically agree with all of this, except:

-  I believe committers are going to have a hard time with fix version
management and we should not trust this

- Git as source of truth was discussed in the Yetus context IIRC but
docmaker uses the JIRA API, so JIRA remains the effective source of truth
without a good replacement (there is no replacement for docmaker), unless
there is a GitHub API I'm not aware of (there could be).

- I would prefer to resolve API compatibility or changelog inconsistencies
before tagging the RC and publishing this tag

- Unless the git branches are locked there is a race between prep and
RC/tag publish. This part is fine as long as the RM can detect the
collision (git rejects the push) and so ultimately this requires all prep
happen locally with the tag/push/publish as final atomic step.



On Fri, Jan 15, 2021 at 3:19 PM Nick Dimiduk  wrote:

> > I have now had to sink two 2.4.1 RCs because of errors in the change log.
>
> IIRC, it's a relatively new phenomenon that we sink an RC because the
> CHANGES file is incomplete.
>
> > That's right... Commtters cannot be trusted to correctly maintain issue
> metadata in JIRA.
>
> I wouldn't go so far as to make a blanket statement such as this... I
> suspect there's a couple new people who missed this message. In my
> experience, a polite nudge from the RM on the affected JIRA issue to the
> offending committer resolves the knowledge gap.
>
> > I think the root cause is GitHub and JIRA are two separate change
> tracking systems with only a minimal amount of integration. It requires
> manual effort.
>
> This I 100% agree with. But then, even before GitHub, Jira and SVN were two
> separate systems, and we managed fine. Maybe, as you say, the committer
> mentality around attention to detail has laxed. Or maybe, as I suggest,
> there's just an education gap involving a couple people, which can be
> addressed.
>
> > We do not have a single source of truth as to what are active branches
> and what was the most recent release by version number made from each
> branch.
>
> Ah, now the discussion gets interesting! IMHO, we do have a source of truth
> re: this information -- it's git.
>
> > Perhaps something can be created that takes the union of git branch
> listing and reporter.a.o release version history.
>
> I did create such a tool, it scrapes git and Jira and produces a database
> containing this information, from which reports are generated and ad-hoc
> queries are issued. I committed it (nearly a year ago to the day) via
> HBASE-22853 (Viraj, you signed off on this commit...). As I mentioned
> earlier, it's still young, but it was invaluable for me in establishing a
> baseline for 2.3.0, and includes a readme with lots of examples.
>
> I am of the opinion that the git repository is the authority on what is in
> a release. After all, a release is defined as a permanent "rel" tag in git
> (plus some ASF ceremony). IMHO, Jira needs to follow git, not the other way
> around. Likewise, tracking the CHANGES file in git -- a file generated from
> a Jira query -- is very silly. Instead, I propose that the changes file be
> generated on demand, perhaps as part building the tarball. The only
> alternative, if you want to retain the CHANGES file in git, is to include
> an update to the CHANGES file with every git commit, which I think is
> completely untenable.
>
> One other stumbling point is that our release notes are generated from a
> field in Jira. We could generate release notes from git commit messages,
> but I think that is an unrealistic burden to place on committers at the
> time of commit.
>
> Following the logic that git is the source of truth, I propose a change to
> our release generation process so that it looks something like
> 1. set the version number in poms, commit.
> 2. apply the RC tag.
> 3. run the audit tool to generate a report.
> 4. resolve inconsistencies (manually for now, eventually the tool could
> update Jira -- it has an API)
> 5. build artifacts, including generating any files from metadata in Jira.
>
> Step 3, when automated, can even be run during our nightly branch builds.
> At that point, a committer doesn't need to worry at all about a fixVersion,
> they just commit to their branches and automation takes care of the rest.
>
> The above has the issue around when to close a ticket. IMHO, a ticket
> should be closed as soon as there's a commit to any branch, but maybe there
> are other complexities to consider.
>
> Thanks for bringing this up, Andrew. It's bothered me since 2.3.0 :)
>
> -n
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: Project management (JIRA fix version tracking) is in crisis

[Nick Dimiduk]

If it's possible, I would also appreciate it if we could add a pre-commit
hook to our repository that rejects a commit whose title does not match our
prescribed pattern. Doing so would go a long way toward simplifying the
parsing tools.

-n

On Fri, Jan 15, 2021 at 3:18 PM Nick Dimiduk  wrote:

> > I have now had to sink two 2.4.1 RCs because of errors in the change log.
>
> IIRC, it's a relatively new phenomenon that we sink an RC because the
> CHANGES file is incomplete.
>
> > That's right... Commtters cannot be trusted to correctly maintain issue
> metadata in JIRA.
>
> I wouldn't go so far as to make a blanket statement such as this... I
> suspect there's a couple new people who missed this message. In my
> experience, a polite nudge from the RM on the affected JIRA issue to the
> offending committer resolves the knowledge gap.
>
> > I think the root cause is GitHub and JIRA are two separate change
> tracking systems with only a minimal amount of integration. It requires
> manual effort.
>
> This I 100% agree with. But then, even before GitHub, Jira and SVN were
> two separate systems, and we managed fine. Maybe, as you say, the committer
> mentality around attention to detail has laxed. Or maybe, as I suggest,
> there's just an education gap involving a couple people, which can be
> addressed.
>
> > We do not have a single source of truth as to what are active branches
> and what was the most recent release by version number made from each
> branch.
>
> Ah, now the discussion gets interesting! IMHO, we do have a source of
> truth re: this information -- it's git.
>
> > Perhaps something can be created that takes the union of git branch
> listing and reporter.a.o release version history.
>
> I did create such a tool, it scrapes git and Jira and produces a database
> containing this information, from which reports are generated and ad-hoc
> queries are issued. I committed it (nearly a year ago to the day) via
> HBASE-22853 (Viraj, you signed off on this commit...). As I mentioned
> earlier, it's still young, but it was invaluable for me in establishing a
> baseline for 2.3.0, and includes a readme with lots of examples.
>
> I am of the opinion that the git repository is the authority on what is in
> a release. After all, a release is defined as a permanent "rel" tag in git
> (plus some ASF ceremony). IMHO, Jira needs to follow git, not the other way
> around. Likewise, tracking the CHANGES file in git -- a file generated from
> a Jira query -- is very silly. Instead, I propose that the changes file be
> generated on demand, perhaps as part building the tarball. The only
> alternative, if you want to retain the CHANGES file in git, is to include
> an update to the CHANGES file with every git commit, which I think is
> completely untenable.
>
> One other stumbling point is that our release notes are generated from a
> field in Jira. We could generate release notes from git commit messages,
> but I think that is an unrealistic burden to place on committers at the
> time of commit.
>
> Following the logic that git is the source of truth, I propose a change to
> our release generation process so that it looks something like
> 1. set the version number in poms, commit.
> 2. apply the RC tag.
> 3. run the audit tool to generate a report.
> 4. resolve inconsistencies (manually for now, eventually the tool could
> update Jira -- it has an API)
> 5. build artifacts, including generating any files from metadata in Jira.
>
> Step 3, when automated, can even be run during our nightly branch builds.
> At that point, a committer doesn't need to worry at all about a fixVersion,
> they just commit to their branches and automation takes care of the rest.
>
> The above has the issue around when to close a ticket. IMHO, a ticket
> should be closed as soon as there's a commit to any branch, but maybe there
> are other complexities to consider.
>
> Thanks for bringing this up, Andrew. It's bothered me since 2.3.0 :)
>
> -n
>

Re: Project management (JIRA fix version tracking) is in crisis

[Nick Dimiduk]

> I have now had to sink two 2.4.1 RCs because of errors in the change log.

IIRC, it's a relatively new phenomenon that we sink an RC because the
CHANGES file is incomplete.

> That's right... Commtters cannot be trusted to correctly maintain issue
metadata in JIRA.

I wouldn't go so far as to make a blanket statement such as this... I
suspect there's a couple new people who missed this message. In my
experience, a polite nudge from the RM on the affected JIRA issue to the
offending committer resolves the knowledge gap.

> I think the root cause is GitHub and JIRA are two separate change
tracking systems with only a minimal amount of integration. It requires
manual effort.

This I 100% agree with. But then, even before GitHub, Jira and SVN were two
separate systems, and we managed fine. Maybe, as you say, the committer
mentality around attention to detail has laxed. Or maybe, as I suggest,
there's just an education gap involving a couple people, which can be
addressed.

> We do not have a single source of truth as to what are active branches
and what was the most recent release by version number made from each
branch.

Ah, now the discussion gets interesting! IMHO, we do have a source of truth
re: this information -- it's git.

> Perhaps something can be created that takes the union of git branch
listing and reporter.a.o release version history.

I did create such a tool, it scrapes git and Jira and produces a database
containing this information, from which reports are generated and ad-hoc
queries are issued. I committed it (nearly a year ago to the day) via
HBASE-22853 (Viraj, you signed off on this commit...). As I mentioned
earlier, it's still young, but it was invaluable for me in establishing a
baseline for 2.3.0, and includes a readme with lots of examples.

I am of the opinion that the git repository is the authority on what is in
a release. After all, a release is defined as a permanent "rel" tag in git
(plus some ASF ceremony). IMHO, Jira needs to follow git, not the other way
around. Likewise, tracking the CHANGES file in git -- a file generated from
a Jira query -- is very silly. Instead, I propose that the changes file be
generated on demand, perhaps as part building the tarball. The only
alternative, if you want to retain the CHANGES file in git, is to include
an update to the CHANGES file with every git commit, which I think is
completely untenable.

One other stumbling point is that our release notes are generated from a
field in Jira. We could generate release notes from git commit messages,
but I think that is an unrealistic burden to place on committers at the
time of commit.

Following the logic that git is the source of truth, I propose a change to
our release generation process so that it looks something like
1. set the version number in poms, commit.
2. apply the RC tag.
3. run the audit tool to generate a report.
4. resolve inconsistencies (manually for now, eventually the tool could
update Jira -- it has an API)
5. build artifacts, including generating any files from metadata in Jira.

Step 3, when automated, can even be run during our nightly branch builds.
At that point, a committer doesn't need to worry at all about a fixVersion,
they just commit to their branches and automation takes care of the rest.

The above has the issue around when to close a ticket. IMHO, a ticket
should be closed as soon as there's a commit to any branch, but maybe there
are other complexities to consider.

Thanks for bringing this up, Andrew. It's bothered me since 2.3.0 :)

-n

[VOTE] Third release candidate for HBase 2.4.1 (RC2) is available

[Andrew Purtell]

[ Thanks to Viraj Jasani for ensuring an accurate change log and for
scripting a tool to do the same. ]

Please vote on this Apache hbase release candidate, hbase-2.4.1RC2.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 2.4.1
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.4.1RC2:

https://github.com/apache/hbase/tree/2.4.1RC2

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC2/

The JIRA change log for this release can be found at:

https://s.apache.org/0nopx

Temporary Maven artifacts are available in the staging repository:

https://repository.apache.org/content/repositories/orgapachehbase-1434/

Artifacts were signed with the apurt...@apache.org key which can be found
in:

https://dist.apache.org/repos/dist/release/hbase/KEYS

The API compatibility report for this RC can be found at:


https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC2/api_compare_2.4.0_to_2.4.1RC2.html

To learn more about Apache HBase, please see

http://hbase.apache.org/

Thanks,
Your HBase Release Manager

Re: Project management (JIRA fix version tracking) is in crisis

[Nick Dimiduk]

Regarding a tool, I’ve already committed one that I used for 2.3.0. IIRC,
Busbey suggested on its original PR the idea of using it during our
nightlies as well. I’m pretty sure I’ve mentioned it explicit already to
Viraj, Andrew, and Huaxiang. Check dev-support/git-jira-release-audit on
master. There’s a readme in the directory.

Of course, if Viraj’s tool is better, let’s move to that one. But please,
take a look here first. I found it very useful for the new minor branch
scenario with 100’s of commits that Andrew mentioned earlier in this thread.

On Fri, Jan 15, 2021 at 11:20 Viraj Jasani  wrote:

> Thanks Andrew and Zach.
> The script tries to identify these issues:
>
> 1. commit is reverted as per commit message
> 2. commit does not contain Jira number format HBASE- in message
> 3. Jira does not have expected fixVersion
> 4. Jira has expected fixVersion, but it is not yet resolved
> 5. Jiras that are marked resolved with given fixVersion but have no
> corresponding
> commit present
>
> IMHO, issues like
> 1. reverted commit and relevant Jira being resolved with
> expected fixVersion which it is actually reverted but Jira is
> not reopened
> 2. some false alarm like preparation of snapshot version
> which is not tied to any Jira
>
> might make it bit difficult to automate this script with
> create-release or nightly build because these kind of discrepancies
> are for committers/RMs to manually check and resolve.
> Having said that, I am sure we can make something better and
> meaningful out of this that will make our life easier going forward.
>
>
> On 2021/01/15 19:07:35, Zach York  wrote:
> > Awesome Viraj! Also if we have confidence in the script we could probably
> > add it to run during a nightly run to try to catch issues earlier and
> > remove the burden on the RM (if there isn't too much noise from open
> > issues).
> >
> > On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell 
> wrote:
> >
> > > Thanks, Viraj. A script like that will be super helpful.
> > >
> > > On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani 
> wrote:
> > >
> > > > > This is not something RMs have had to do
> > > > > in the past and it asks a lot of them.
> > > >
> > > > +1. It is too much for RM to do if RM alone is responsible for
> > > > maintaining Git/Jira in sync.
> > > >
> > > > The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> > > > RCs is because I used a small script that a) goes through all
> > > > commits, b) gets Jira number, c) search for the Jira fields using
> it's
> > > > API and d) finds the diff. And similarly using JQL, it finds all
> Jiras
> > > > with expected fixVersion but has no git commit present. Mostly I
> > > > will raise PR for this script tomorrow early as part of HBASE-25514.
> > > >
> > > > If we have enough trust on this, when RM given heads up on
> > > > dev mailing list for a new release, more committers can be
> > > > encouraged to start using this script to find any discrepancy,
> resolve
> > > > it if possible and keep in mind the upcoming release version while
> > > > resolving a Jira.
> > > > However, this is just a script that can help specifically just before
> > > > spinning RCs. More precautionary steps would be bringing
> > > > awareness of all active branches and how they relate to active
> > > > releases as mentioned by Andrew.
> > > >
> > > >
> > > > On 2021/01/15 18:39:18, Andrew Purtell  wrote:
> > > > > We do not have a single source of truth as to what are active
> branches
> > > > and
> > > > > what was the most recent release by version number made from each
> > > branch.
> > > > > If we had such a resource, then committers could refer to it
> whenever
> > > > > merging PRs or cherry picking commits backward. Perhaps something
> can
> > > be
> > > > > created that takes the union of git branch listing and reporter.a.o
> > > > release
> > > > > version history. The downside of course it this would be something
> new
> > > to
> > > > > maintain. Committers would also need to be educated about the
> existence
> > > > of
> > > > > this tool and the requirement to use it, but that could be solved
> with
> > > an
> > > > > update to the How To Release section of the online book. I filed
> > > > > HBASE-25515 for this idea.
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell <
> apurt...@apache.org>
> > > > wrote:
> > > > >
> > > > > > I have now had to sink two 2.4.1 RCs because of errors in the
> change
> > > > log.
> > > > > >
> > > > > > I made a pass over git history and ensured every commit was
> > > included. I
> > > > > > had also made a pass over JIRA to move out any unresolved issues
> or
> > > > > > complete the resolution of same. What I did not do is check that
> > > every
> > > > > > resolved JIRA corresponded to an actual commit. This is not
> something
> > > > RMs
> > > > > > have had to do in the past and it asks a lot of them.
> > > > > >
> > > > > > I know NOW that as RM I cannot currently trust committers 

Re: Project management (JIRA fix version tracking) is in crisis

[Huaxiang Sun]

Yeah, after multiple release efforts of 2.3.4 failed, I finally did a
manual check.
1. got git logs for jiras from 2.3.3.
2. did a dry-run to generate CHANGES.md.

I compared these two outputs and removed the common ones. The one left in
1) are issues in git and fixVersion not correctly setup in jira.
The one left in 2) is fixVersion set to 2.3.4 but patch is not committed to
the branch.

The logic is essentially based on the fact that git is the only source of
truth.


On Fri, Jan 15, 2021 at 11:30 AM Nick Dimiduk  wrote:

> Hi Andrew,
>
> I had a conversation on this same topic yesterday with Huaxiang and, I
> believe, came to a similar conclusion. I was advocating that git be the
> source of truth and Jira follow, and there be no CHANGES file in the
> repository at all.
>
> I’m on mobile at the moment. Let me spend some time this afternoon to catch
> up this thread and write up my own thoughts. I actually planned to raise a
> discuss thread today anyway.
>
> Thank you all for caring about this!
> -n
>
> On Fri, Jan 15, 2021 at 11:20 Viraj Jasani  wrote:
>
> > Thanks Andrew and Zach.
> > The script tries to identify these issues:
> >
> > 1. commit is reverted as per commit message
> > 2. commit does not contain Jira number format HBASE- in message
> > 3. Jira does not have expected fixVersion
> > 4. Jira has expected fixVersion, but it is not yet resolved
> > 5. Jiras that are marked resolved with given fixVersion but have no
> > corresponding
> > commit present
> >
> > IMHO, issues like
> > 1. reverted commit and relevant Jira being resolved with
> > expected fixVersion which it is actually reverted but Jira is
> > not reopened
> > 2. some false alarm like preparation of snapshot version
> > which is not tied to any Jira
> >
> > might make it bit difficult to automate this script with
> > create-release or nightly build because these kind of discrepancies
> > are for committers/RMs to manually check and resolve.
> > Having said that, I am sure we can make something better and
> > meaningful out of this that will make our life easier going forward.
> >
> >
> > On 2021/01/15 19:07:35, Zach York  wrote:
> > > Awesome Viraj! Also if we have confidence in the script we could
> probably
> > > add it to run during a nightly run to try to catch issues earlier and
> > > remove the burden on the RM (if there isn't too much noise from open
> > > issues).
> > >
> > > On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell 
> > wrote:
> > >
> > > > Thanks, Viraj. A script like that will be super helpful.
> > > >
> > > > On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani 
> > wrote:
> > > >
> > > > > > This is not something RMs have had to do
> > > > > > in the past and it asks a lot of them.
> > > > >
> > > > > +1. It is too much for RM to do if RM alone is responsible for
> > > > > maintaining Git/Jira in sync.
> > > > >
> > > > > The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> > > > > RCs is because I used a small script that a) goes through all
> > > > > commits, b) gets Jira number, c) search for the Jira fields using
> > it's
> > > > > API and d) finds the diff. And similarly using JQL, it finds all
> > Jiras
> > > > > with expected fixVersion but has no git commit present. Mostly I
> > > > > will raise PR for this script tomorrow early as part of
> HBASE-25514.
> > > > >
> > > > > If we have enough trust on this, when RM given heads up on
> > > > > dev mailing list for a new release, more committers can be
> > > > > encouraged to start using this script to find any discrepancy,
> > resolve
> > > > > it if possible and keep in mind the upcoming release version while
> > > > > resolving a Jira.
> > > > > However, this is just a script that can help specifically just
> before
> > > > > spinning RCs. More precautionary steps would be bringing
> > > > > awareness of all active branches and how they relate to active
> > > > > releases as mentioned by Andrew.
> > > > >
> > > > >
> > > > > On 2021/01/15 18:39:18, Andrew Purtell 
> wrote:
> > > > > > We do not have a single source of truth as to what are active
> > branches
> > > > > and
> > > > > > what was the most recent release by version number made from each
> > > > branch.
> > > > > > If we had such a resource, then committers could refer to it
> > whenever
> > > > > > merging PRs or cherry picking commits backward. Perhaps something
> > can
> > > > be
> > > > > > created that takes the union of git branch listing and
> reporter.a.o
> > > > > release
> > > > > > version history. The downside of course it this would be
> something
> > new
> > > > to
> > > > > > maintain. Committers would also need to be educated about the
> > existence
> > > > > of
> > > > > > this tool and the requirement to use it, but that could be solved
> > with
> > > > an
> > > > > > update to the How To Release section of the online book. I filed
> > > > > > HBASE-25515 for this idea.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 15, 2021 at 10:05 AM Andrew 

Re: [VOTE] HBase 2.3.4 release candidate (RC4) is available

[Huaxiang Sun]

Yeah, that is true. It is fixed by HBASE-25238, I did the manual check.
Leaving this question to dev discussion. There are multiple rounds of
effort for this 2.3.4 release.
Unless it is a real issue, hope this vote does not sink just because of
this issue.

But again, it is up to dev's agreement and we need to put the decisions
into the release guidance.


On Fri, Jan 15, 2021 at 4:24 AM Viraj Jasani  wrote:

> HBASE-25234 doesn't have corresponding commit
> present on branch-2.3 but perhaps it is fine as it
> is resolved with comment: "Fixed by HBASE-25238" ?
>
> Are we find with HBASE-25234 showing up in CHANGELOG?
>
> On 2021/01/15 03:03:47, Huaxiang Sun  wrote:
> > Please vote on this Apache hbase release candidate,
> >
> > hbase-2.3.4RC4
> >
> >
> > The VOTE will remain open for at least 72 hours.
> >
> >
> > [ ] +1 Release this package as Apache hbase 2.3.4
> >
> > [ ] -1 Do not release this package because ...
> >
> >
> > The tag to be voted on is 2.3.4RC4:
> >
> >
> >   https://github.com/apache/hbase/tree/2.3.4RC4
> >
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> >
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.3.4RC4/
> >
> >
> > Maven artifacts are available in a staging repository at:
> >
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1433/
> >
> >
> > Artifacts were signed with the huaxiang...@apache.org key which can be
> > found in:
> >
> >
> >   https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> >
> > To learn more about Apache hbase, please see
> >
> >
> >   http://hbase.apache.org/
> >
> >
> > Thanks,
> >
> > Your HBase Release Manager
> >
>

Re: Project management (JIRA fix version tracking) is in crisis

[Nick Dimiduk]

Hi Andrew,

I had a conversation on this same topic yesterday with Huaxiang and, I
believe, came to a similar conclusion. I was advocating that git be the
source of truth and Jira follow, and there be no CHANGES file in the
repository at all.

I’m on mobile at the moment. Let me spend some time this afternoon to catch
up this thread and write up my own thoughts. I actually planned to raise a
discuss thread today anyway.

Thank you all for caring about this!
-n

On Fri, Jan 15, 2021 at 11:20 Viraj Jasani  wrote:

> Thanks Andrew and Zach.
> The script tries to identify these issues:
>
> 1. commit is reverted as per commit message
> 2. commit does not contain Jira number format HBASE- in message
> 3. Jira does not have expected fixVersion
> 4. Jira has expected fixVersion, but it is not yet resolved
> 5. Jiras that are marked resolved with given fixVersion but have no
> corresponding
> commit present
>
> IMHO, issues like
> 1. reverted commit and relevant Jira being resolved with
> expected fixVersion which it is actually reverted but Jira is
> not reopened
> 2. some false alarm like preparation of snapshot version
> which is not tied to any Jira
>
> might make it bit difficult to automate this script with
> create-release or nightly build because these kind of discrepancies
> are for committers/RMs to manually check and resolve.
> Having said that, I am sure we can make something better and
> meaningful out of this that will make our life easier going forward.
>
>
> On 2021/01/15 19:07:35, Zach York  wrote:
> > Awesome Viraj! Also if we have confidence in the script we could probably
> > add it to run during a nightly run to try to catch issues earlier and
> > remove the burden on the RM (if there isn't too much noise from open
> > issues).
> >
> > On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell 
> wrote:
> >
> > > Thanks, Viraj. A script like that will be super helpful.
> > >
> > > On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani 
> wrote:
> > >
> > > > > This is not something RMs have had to do
> > > > > in the past and it asks a lot of them.
> > > >
> > > > +1. It is too much for RM to do if RM alone is responsible for
> > > > maintaining Git/Jira in sync.
> > > >
> > > > The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> > > > RCs is because I used a small script that a) goes through all
> > > > commits, b) gets Jira number, c) search for the Jira fields using
> it's
> > > > API and d) finds the diff. And similarly using JQL, it finds all
> Jiras
> > > > with expected fixVersion but has no git commit present. Mostly I
> > > > will raise PR for this script tomorrow early as part of HBASE-25514.
> > > >
> > > > If we have enough trust on this, when RM given heads up on
> > > > dev mailing list for a new release, more committers can be
> > > > encouraged to start using this script to find any discrepancy,
> resolve
> > > > it if possible and keep in mind the upcoming release version while
> > > > resolving a Jira.
> > > > However, this is just a script that can help specifically just before
> > > > spinning RCs. More precautionary steps would be bringing
> > > > awareness of all active branches and how they relate to active
> > > > releases as mentioned by Andrew.
> > > >
> > > >
> > > > On 2021/01/15 18:39:18, Andrew Purtell  wrote:
> > > > > We do not have a single source of truth as to what are active
> branches
> > > > and
> > > > > what was the most recent release by version number made from each
> > > branch.
> > > > > If we had such a resource, then committers could refer to it
> whenever
> > > > > merging PRs or cherry picking commits backward. Perhaps something
> can
> > > be
> > > > > created that takes the union of git branch listing and reporter.a.o
> > > > release
> > > > > version history. The downside of course it this would be something
> new
> > > to
> > > > > maintain. Committers would also need to be educated about the
> existence
> > > > of
> > > > > this tool and the requirement to use it, but that could be solved
> with
> > > an
> > > > > update to the How To Release section of the online book. I filed
> > > > > HBASE-25515 for this idea.
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell <
> apurt...@apache.org>
> > > > wrote:
> > > > >
> > > > > > I have now had to sink two 2.4.1 RCs because of errors in the
> change
> > > > log.
> > > > > >
> > > > > > I made a pass over git history and ensured every commit was
> > > included. I
> > > > > > had also made a pass over JIRA to move out any unresolved issues
> or
> > > > > > complete the resolution of same. What I did not do is check that
> > > every
> > > > > > resolved JIRA corresponded to an actual commit. This is not
> something
> > > > RMs
> > > > > > have had to do in the past and it asks a lot of them.
> > > > > >
> > > > > > I know NOW that as RM I cannot currently trust committers to get
> fix
> > > > > > versions right or care about this.
> > > > > >
> > > > > > That's 

Re: Project management (JIRA fix version tracking) is in crisis

[Viraj Jasani]

Thanks Andrew and Zach.
The script tries to identify these issues:

1. commit is reverted as per commit message
2. commit does not contain Jira number format HBASE- in message
3. Jira does not have expected fixVersion
4. Jira has expected fixVersion, but it is not yet resolved
5. Jiras that are marked resolved with given fixVersion but have no 
corresponding
commit present

IMHO, issues like 
1. reverted commit and relevant Jira being resolved with
expected fixVersion which it is actually reverted but Jira is
not reopened
2. some false alarm like preparation of snapshot version
which is not tied to any Jira

might make it bit difficult to automate this script with
create-release or nightly build because these kind of discrepancies
are for committers/RMs to manually check and resolve.
Having said that, I am sure we can make something better and
meaningful out of this that will make our life easier going forward.


On 2021/01/15 19:07:35, Zach York  wrote: 
> Awesome Viraj! Also if we have confidence in the script we could probably
> add it to run during a nightly run to try to catch issues earlier and
> remove the burden on the RM (if there isn't too much noise from open
> issues).
> 
> On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell  wrote:
> 
> > Thanks, Viraj. A script like that will be super helpful.
> >
> > On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani  wrote:
> >
> > > > This is not something RMs have had to do
> > > > in the past and it asks a lot of them.
> > >
> > > +1. It is too much for RM to do if RM alone is responsible for
> > > maintaining Git/Jira in sync.
> > >
> > > The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> > > RCs is because I used a small script that a) goes through all
> > > commits, b) gets Jira number, c) search for the Jira fields using it's
> > > API and d) finds the diff. And similarly using JQL, it finds all Jiras
> > > with expected fixVersion but has no git commit present. Mostly I
> > > will raise PR for this script tomorrow early as part of HBASE-25514.
> > >
> > > If we have enough trust on this, when RM given heads up on
> > > dev mailing list for a new release, more committers can be
> > > encouraged to start using this script to find any discrepancy, resolve
> > > it if possible and keep in mind the upcoming release version while
> > > resolving a Jira.
> > > However, this is just a script that can help specifically just before
> > > spinning RCs. More precautionary steps would be bringing
> > > awareness of all active branches and how they relate to active
> > > releases as mentioned by Andrew.
> > >
> > >
> > > On 2021/01/15 18:39:18, Andrew Purtell  wrote:
> > > > We do not have a single source of truth as to what are active branches
> > > and
> > > > what was the most recent release by version number made from each
> > branch.
> > > > If we had such a resource, then committers could refer to it whenever
> > > > merging PRs or cherry picking commits backward. Perhaps something can
> > be
> > > > created that takes the union of git branch listing and reporter.a.o
> > > release
> > > > version history. The downside of course it this would be something new
> > to
> > > > maintain. Committers would also need to be educated about the existence
> > > of
> > > > this tool and the requirement to use it, but that could be solved with
> > an
> > > > update to the How To Release section of the online book. I filed
> > > > HBASE-25515 for this idea.
> > > >
> > > >
> > > >
> > > > On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell 
> > > wrote:
> > > >
> > > > > I have now had to sink two 2.4.1 RCs because of errors in the change
> > > log.
> > > > >
> > > > > I made a pass over git history and ensured every commit was
> > included. I
> > > > > had also made a pass over JIRA to move out any unresolved issues or
> > > > > complete the resolution of same. What I did not do is check that
> > every
> > > > > resolved JIRA corresponded to an actual commit. This is not something
> > > RMs
> > > > > have had to do in the past and it asks a lot of them.
> > > > >
> > > > > I know NOW that as RM I cannot currently trust committers to get fix
> > > > > versions right or care about this.
> > > > >
> > > > > That's right... Commtters cannot be trusted to correctly maintain
> > issue
> > > > > metadata in JIRA.
> > > > >
> > > > > That is not a good situation for the project to be in. Up until now
> > it
> > > has
> > > > > not been the responsibility of the RM to check each and every JIRA
> > > status.
> > > > > It has been the collective responsibility of committers to care about
> > > the
> > > > > project's release tracking insofar as to correctly update fix
> > versions
> > > in
> > > > > JIRA. For releases containing relatively few changes, like 2.4.1,
> > with
> > > ~50
> > > > > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > > > > versions, walk the commit history, and set back fix versions on JIRA
> > to
> > > > > actually 

Re: Project management (JIRA fix version tracking) is in crisis

[Zach York]

Awesome Viraj! Also if we have confidence in the script we could probably
add it to run during a nightly run to try to catch issues earlier and
remove the burden on the RM (if there isn't too much noise from open
issues).

On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell  wrote:

> Thanks, Viraj. A script like that will be super helpful.
>
> On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani  wrote:
>
> > > This is not something RMs have had to do
> > > in the past and it asks a lot of them.
> >
> > +1. It is too much for RM to do if RM alone is responsible for
> > maintaining Git/Jira in sync.
> >
> > The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> > RCs is because I used a small script that a) goes through all
> > commits, b) gets Jira number, c) search for the Jira fields using it's
> > API and d) finds the diff. And similarly using JQL, it finds all Jiras
> > with expected fixVersion but has no git commit present. Mostly I
> > will raise PR for this script tomorrow early as part of HBASE-25514.
> >
> > If we have enough trust on this, when RM given heads up on
> > dev mailing list for a new release, more committers can be
> > encouraged to start using this script to find any discrepancy, resolve
> > it if possible and keep in mind the upcoming release version while
> > resolving a Jira.
> > However, this is just a script that can help specifically just before
> > spinning RCs. More precautionary steps would be bringing
> > awareness of all active branches and how they relate to active
> > releases as mentioned by Andrew.
> >
> >
> > On 2021/01/15 18:39:18, Andrew Purtell  wrote:
> > > We do not have a single source of truth as to what are active branches
> > and
> > > what was the most recent release by version number made from each
> branch.
> > > If we had such a resource, then committers could refer to it whenever
> > > merging PRs or cherry picking commits backward. Perhaps something can
> be
> > > created that takes the union of git branch listing and reporter.a.o
> > release
> > > version history. The downside of course it this would be something new
> to
> > > maintain. Committers would also need to be educated about the existence
> > of
> > > this tool and the requirement to use it, but that could be solved with
> an
> > > update to the How To Release section of the online book. I filed
> > > HBASE-25515 for this idea.
> > >
> > >
> > >
> > > On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell 
> > wrote:
> > >
> > > > I have now had to sink two 2.4.1 RCs because of errors in the change
> > log.
> > > >
> > > > I made a pass over git history and ensured every commit was
> included. I
> > > > had also made a pass over JIRA to move out any unresolved issues or
> > > > complete the resolution of same. What I did not do is check that
> every
> > > > resolved JIRA corresponded to an actual commit. This is not something
> > RMs
> > > > have had to do in the past and it asks a lot of them.
> > > >
> > > > I know NOW that as RM I cannot currently trust committers to get fix
> > > > versions right or care about this.
> > > >
> > > > That's right... Commtters cannot be trusted to correctly maintain
> issue
> > > > metadata in JIRA.
> > > >
> > > > That is not a good situation for the project to be in. Up until now
> it
> > has
> > > > not been the responsibility of the RM to check each and every JIRA
> > status.
> > > > It has been the collective responsibility of committers to care about
> > the
> > > > project's release tracking insofar as to correctly update fix
> versions
> > in
> > > > JIRA. For releases containing relatively few changes, like 2.4.1,
> with
> > ~50
> > > > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > > > versions, walk the commit history, and set back fix versions on JIRA
> to
> > > > actually correspond with what was truly committed. However, for minor
> > > > releases, with hundreds of commits, this will not be possible.
> > > >
> > > > I think the root cause is GitHub and JIRA are two separate change
> > tracking
> > > > systems with only a minimal amount of integration. It requires manual
> > > > effort. More and more, new committers are familiar with GitHub and
> PRs
> > and
> > > > are not familiar with JIRA and the Apache way of using JIRA to build
> > change
> > > > logs. We need to better educate new and existing committers on their
> > > > responsibilities with regards to maintaining JIRA metadata correctly.
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, 

Re: Project management (JIRA fix version tracking) is in crisis

[Andrew Purtell]

Thanks, Viraj. A script like that will be super helpful.

On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani  wrote:

> > This is not something RMs have had to do
> > in the past and it asks a lot of them.
>
> +1. It is too much for RM to do if RM alone is responsible for
> maintaining Git/Jira in sync.
>
> The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> RCs is because I used a small script that a) goes through all
> commits, b) gets Jira number, c) search for the Jira fields using it's
> API and d) finds the diff. And similarly using JQL, it finds all Jiras
> with expected fixVersion but has no git commit present. Mostly I
> will raise PR for this script tomorrow early as part of HBASE-25514.
>
> If we have enough trust on this, when RM given heads up on
> dev mailing list for a new release, more committers can be
> encouraged to start using this script to find any discrepancy, resolve
> it if possible and keep in mind the upcoming release version while
> resolving a Jira.
> However, this is just a script that can help specifically just before
> spinning RCs. More precautionary steps would be bringing
> awareness of all active branches and how they relate to active
> releases as mentioned by Andrew.
>
>
> On 2021/01/15 18:39:18, Andrew Purtell  wrote:
> > We do not have a single source of truth as to what are active branches
> and
> > what was the most recent release by version number made from each branch.
> > If we had such a resource, then committers could refer to it whenever
> > merging PRs or cherry picking commits backward. Perhaps something can be
> > created that takes the union of git branch listing and reporter.a.o
> release
> > version history. The downside of course it this would be something new to
> > maintain. Committers would also need to be educated about the existence
> of
> > this tool and the requirement to use it, but that could be solved with an
> > update to the How To Release section of the online book. I filed
> > HBASE-25515 for this idea.
> >
> >
> >
> > On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell 
> wrote:
> >
> > > I have now had to sink two 2.4.1 RCs because of errors in the change
> log.
> > >
> > > I made a pass over git history and ensured every commit was included. I
> > > had also made a pass over JIRA to move out any unresolved issues or
> > > complete the resolution of same. What I did not do is check that every
> > > resolved JIRA corresponded to an actual commit. This is not something
> RMs
> > > have had to do in the past and it asks a lot of them.
> > >
> > > I know NOW that as RM I cannot currently trust committers to get fix
> > > versions right or care about this.
> > >
> > > That's right... Commtters cannot be trusted to correctly maintain issue
> > > metadata in JIRA.
> > >
> > > That is not a good situation for the project to be in. Up until now it
> has
> > > not been the responsibility of the RM to check each and every JIRA
> status.
> > > It has been the collective responsibility of committers to care about
> the
> > > project's release tracking insofar as to correctly update fix versions
> in
> > > JIRA. For releases containing relatively few changes, like 2.4.1, with
> ~50
> > > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > > versions, walk the commit history, and set back fix versions on JIRA to
> > > actually correspond with what was truly committed. However, for minor
> > > releases, with hundreds of commits, this will not be possible.
> > >
> > > I think the root cause is GitHub and JIRA are two separate change
> tracking
> > > systems with only a minimal amount of integration. It requires manual
> > > effort. More and more, new committers are familiar with GitHub and PRs
> and
> > > are not familiar with JIRA and the Apache way of using JIRA to build
> change
> > > logs. We need to better educate new and existing committers on their
> > > responsibilities with regards to maintaining JIRA metadata correctly.
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: Project management (JIRA fix version tracking) is in crisis

[Viraj Jasani]

> This is not something RMs have had to do
> in the past and it asks a lot of them.

+1. It is too much for RM to do if RM alone is responsible for
maintaining Git/Jira in sync.

The reason why I was able to get the diffs for 2.3.4 and 2.4.1
RCs is because I used a small script that a) goes through all
commits, b) gets Jira number, c) search for the Jira fields using it's
API and d) finds the diff. And similarly using JQL, it finds all Jiras
with expected fixVersion but has no git commit present. Mostly I
will raise PR for this script tomorrow early as part of HBASE-25514.

If we have enough trust on this, when RM given heads up on
dev mailing list for a new release, more committers can be
encouraged to start using this script to find any discrepancy, resolve
it if possible and keep in mind the upcoming release version while
resolving a Jira.
However, this is just a script that can help specifically just before
spinning RCs. More precautionary steps would be bringing
awareness of all active branches and how they relate to active
releases as mentioned by Andrew.


On 2021/01/15 18:39:18, Andrew Purtell  wrote: 
> We do not have a single source of truth as to what are active branches and
> what was the most recent release by version number made from each branch.
> If we had such a resource, then committers could refer to it whenever
> merging PRs or cherry picking commits backward. Perhaps something can be
> created that takes the union of git branch listing and reporter.a.o release
> version history. The downside of course it this would be something new to
> maintain. Committers would also need to be educated about the existence of
> this tool and the requirement to use it, but that could be solved with an
> update to the How To Release section of the online book. I filed
> HBASE-25515 for this idea.
> 
> 
> 
> On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell  wrote:
> 
> > I have now had to sink two 2.4.1 RCs because of errors in the change log.
> >
> > I made a pass over git history and ensured every commit was included. I
> > had also made a pass over JIRA to move out any unresolved issues or
> > complete the resolution of same. What I did not do is check that every
> > resolved JIRA corresponded to an actual commit. This is not something RMs
> > have had to do in the past and it asks a lot of them.
> >
> > I know NOW that as RM I cannot currently trust committers to get fix
> > versions right or care about this.
> >
> > That's right... Commtters cannot be trusted to correctly maintain issue
> > metadata in JIRA.
> >
> > That is not a good situation for the project to be in. Up until now it has
> > not been the responsibility of the RM to check each and every JIRA status.
> > It has been the collective responsibility of committers to care about the
> > project's release tracking insofar as to correctly update fix versions in
> > JIRA. For releases containing relatively few changes, like 2.4.1, with ~50
> > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > versions, walk the commit history, and set back fix versions on JIRA to
> > actually correspond with what was truly committed. However, for minor
> > releases, with hundreds of commits, this will not be possible.
> >
> > I think the root cause is GitHub and JIRA are two separate change tracking
> > systems with only a minimal amount of integration. It requires manual
> > effort. More and more, new committers are familiar with GitHub and PRs and
> > are not familiar with JIRA and the Apache way of using JIRA to build change
> > logs. We need to better educate new and existing committers on their
> > responsibilities with regards to maintaining JIRA metadata correctly.
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
> 
> 
> -- 
> Best regards,
> Andrew
> 
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
> 

Re: Project management (JIRA fix version tracking) is in crisis

[Andrew Purtell]

A related factor is the practice of holding a JIRA open until all backports
are complete. This is something RMs will already be familiar with, so isn't
more than an inconvenience, but it does lend itself to fix version
inaccuracy. Say you merge a PR or commit a cherry pick to some of the
active branches, but not all, and do not resolve the JIRA. If someone wants
to release from one of the branches where the commit was committed, the
change will not appear in the change log. The JIRA status for the issue
must be set to resolved in order for the issue to be included in the change
log.

What I do as RM, and what I expect all RMs do, is run a report of all
issues with a fix version of what I am about to release, and if any are in
unresolved state, will resolve the JIRA at that time and advise further
commits be made on subtasks or new issues.

It would be better, when you are considering holding an issue open for a
while for some backport to come in, to not hold the issue open, to resolve
it at the time the first commits are complete. Open new issues for any
additional backporting work.




On Fri, Jan 15, 2021 at 10:39 AM Andrew Purtell  wrote:

> We do not have a single source of truth as to what are active branches and
> what was the most recent release by version number made from each branch.
> If we had such a resource, then committers could refer to it whenever
> merging PRs or cherry picking commits backward. Perhaps something can be
> created that takes the union of git branch listing and reporter.a.o release
> version history. The downside of course it this would be something new to
> maintain. Committers would also need to be educated about the existence of
> this tool and the requirement to use it, but that could be solved with an
> update to the How To Release section of the online book. I filed
> HBASE-25515 for this idea.
>
>
>
> On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell 
> wrote:
>
>> I have now had to sink two 2.4.1 RCs because of errors in the change log.
>>
>> I made a pass over git history and ensured every commit was included. I
>> had also made a pass over JIRA to move out any unresolved issues or
>> complete the resolution of same. What I did not do is check that every
>> resolved JIRA corresponded to an actual commit. This is not something RMs
>> have had to do in the past and it asks a lot of them.
>>
>> I know NOW that as RM I cannot currently trust committers to get fix
>> versions right or care about this.
>>
>> That's right... Commtters cannot be trusted to correctly maintain issue
>> metadata in JIRA.
>>
>> That is not a good situation for the project to be in. Up until now it
>> has not been the responsibility of the RM to check each and every JIRA
>> status. It has been the collective responsibility of committers to care
>> about the project's release tracking insofar as to correctly update fix
>> versions in JIRA. For releases containing relatively few changes, like
>> 2.4.1, with ~50 changes, I suppose it is possible for the RM to remove all
>> 2.4.1 fix versions, walk the commit history, and set back fix versions on
>> JIRA to actually correspond with what was truly committed. However, for
>> minor releases, with hundreds of commits, this will not be possible.
>>
>> I think the root cause is GitHub and JIRA are two separate change
>> tracking systems with only a minimal amount of integration. It requires
>> manual effort. More and more, new committers are familiar with GitHub and
>> PRs and are not familiar with JIRA and the Apache way of using JIRA to
>> build change logs. We need to better educate new and existing committers on
>> their responsibilities with regards to maintaining JIRA metadata correctly.
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>>- A23, Crosstalk
>>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[jira] [Created] (HBASE-25515) At a glance git branch release history tool

[Andrew Kyle Purtell (Jira)]

Andrew Kyle Purtell created HBASE-25515:
---

 Summary: At a glance git branch release history tool
 Key: HBASE-25515
 URL: https://issues.apache.org/jira/browse/HBASE-25515
 Project: HBase
  Issue Type: Task
Reporter: Andrew Kyle Purtell


We do not have a single source of truth as to what are active branches and what 
was the most recent release by version number made from each branch. If we had 
such a resource, then committers could refer to it whenever merging PRs or 
cherry picking commits backward. Perhaps something can be created that takes 
the union of git branch listing and reporter.a.o release version history. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Project management (JIRA fix version tracking) is in crisis

[Andrew Purtell]

We do not have a single source of truth as to what are active branches and
what was the most recent release by version number made from each branch.
If we had such a resource, then committers could refer to it whenever
merging PRs or cherry picking commits backward. Perhaps something can be
created that takes the union of git branch listing and reporter.a.o release
version history. The downside of course it this would be something new to
maintain. Committers would also need to be educated about the existence of
this tool and the requirement to use it, but that could be solved with an
update to the How To Release section of the online book. I filed
HBASE-25515 for this idea.



On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell  wrote:

> I have now had to sink two 2.4.1 RCs because of errors in the change log.
>
> I made a pass over git history and ensured every commit was included. I
> had also made a pass over JIRA to move out any unresolved issues or
> complete the resolution of same. What I did not do is check that every
> resolved JIRA corresponded to an actual commit. This is not something RMs
> have had to do in the past and it asks a lot of them.
>
> I know NOW that as RM I cannot currently trust committers to get fix
> versions right or care about this.
>
> That's right... Commtters cannot be trusted to correctly maintain issue
> metadata in JIRA.
>
> That is not a good situation for the project to be in. Up until now it has
> not been the responsibility of the RM to check each and every JIRA status.
> It has been the collective responsibility of committers to care about the
> project's release tracking insofar as to correctly update fix versions in
> JIRA. For releases containing relatively few changes, like 2.4.1, with ~50
> changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> versions, walk the commit history, and set back fix versions on JIRA to
> actually correspond with what was truly committed. However, for minor
> releases, with hundreds of commits, this will not be possible.
>
> I think the root cause is GitHub and JIRA are two separate change tracking
> systems with only a minimal amount of integration. It requires manual
> effort. More and more, new committers are familiar with GitHub and PRs and
> are not familiar with JIRA and the Apache way of using JIRA to build change
> logs. We need to better educate new and existing committers on their
> responsibilities with regards to maintaining JIRA metadata correctly.
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[WITHDRAWN] [VOTE] Second release candidate for HBase 2.4.1 (RC1) is available

[Andrew Purtell]

HBASE-25083 is not a big deal in my opinion. Although it is incorrect it is
"make sure the next hbase 1.y release has Hadoop 2.10 as a minimum
version". As documentation is generated from master, really, only master
branch is where we care about whether this is included or not. Still
annoying, though.

HBASE-25404 is a problem. The change log is still incorrect. This time,
including things that are not actually committed there. Last time, the
problem was NOT including things that were committed.

I wrote a separate email to dev@ calling out the decline in committer
attention to maintaining correct fix version metadata in JIRA and I expect
it to be resolved project wide. In the meantime I will sink this RC and
make another with an, again, corrected change log.

-1


On Fri, Jan 15, 2021 at 4:13 AM Viraj Jasani  wrote:

> HBASE-25404 and HBASE-25083 both are resolved with
> fixVersion 2.4.1 but corresponding commits are missing
> from branch-2.4  :(
>
>
> On 2021/01/14 23:10:41, Andrew Purtell  wrote:
> > Note the Maven temporary repository is hbase-1432 (not hbase-1430 as
> > cut-and-pasted)
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1432/
> >
> > On Thu, Jan 14, 2021 at 3:09 PM Andrew Purtell 
> wrote:
> >
> > > Please vote on this Apache hbase release candidate, hbase-2.4.1RC1.
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache hbase 2.4.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 2.4.1RC1:
> > >
> > > https://github.com/apache/hbase/tree/2.4.1RC1
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC1/
> > >
> > > Temporary Maven artifacts are available in the staging repository:
> > >
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1430/
> > >
> > > Artifacts were signed with the apurt...@apache.org key which can be
> found
> > > in:
> > >
> > > https://dist.apache.org/repos/dist/release/hbase/KEYS
> > >
> > > The API compatibility report for this RC can be found at:
> > >
> > >
> > >
> https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC1/api_compare_2.4.0_to_2.4.1RC1.html
> > >
> > > To learn more about Apache HBase, please see
> > >
> > > http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Project management (JIRA fix version tracking) is in crisis

[Andrew Purtell]

I have now had to sink two 2.4.1 RCs because of errors in the change log.

I made a pass over git history and ensured every commit was included. I had
also made a pass over JIRA to move out any unresolved issues or complete
the resolution of same. What I did not do is check that every resolved JIRA
corresponded to an actual commit. This is not something RMs have had to do
in the past and it asks a lot of them.

I know NOW that as RM I cannot currently trust committers to get fix
versions right or care about this.

That's right... Commtters cannot be trusted to correctly maintain issue
metadata in JIRA.

That is not a good situation for the project to be in. Up until now it has
not been the responsibility of the RM to check each and every JIRA status.
It has been the collective responsibility of committers to care about the
project's release tracking insofar as to correctly update fix versions in
JIRA. For releases containing relatively few changes, like 2.4.1, with ~50
changes, I suppose it is possible for the RM to remove all 2.4.1 fix
versions, walk the commit history, and set back fix versions on JIRA to
actually correspond with what was truly committed. However, for minor
releases, with hundreds of commits, this will not be possible.

I think the root cause is GitHub and JIRA are two separate change tracking
systems with only a minimal amount of integration. It requires manual
effort. More and more, new committers are familiar with GitHub and PRs and
are not familiar with JIRA and the Apache way of using JIRA to build change
logs. We need to better educate new and existing committers on their
responsibilities with regards to maintaining JIRA metadata correctly.

-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [VOTE] Second release candidate for HBase 2.4.1 (RC1) is available

[Andrew Purtell]

I made a pass over git history and ensured every commit was included. I had
also made a pass over JIRA to move out any unresolved issues or complete
the resolution of same. What I did not do is check that every resolved JIRA
corresponded to an actual commit. This is not something RMs have had to do
in the past and it asks a lot of them.

I know NOW that as RM I cannot currently trust committers to get fix
versions right or care about this. That is not a good situation for the
project to be in. Up until now it has not been the responsibility of the RM
to check each and every JIRA status. It has been the collective
responsibility of committers to care about the project's release tracking
insofar as to correctly update fix versions in JIRA.

This is very concerning, how far and fast fix version discipline has
fallen. I would go so far as to call it a crisis for the project. I think
the root cause is GitHub and JIRA are two separate change tracking systems
with only a minimal amount of integration. It requires manual effort. More
and more, new committers are familiar with GitHub and PRs and are not
familiar with JIRA and the Apache way of using JIRA to build change logs.
We need to better educate new and existing committers on their
responsibilities with regards to maintaining JIRA metadata correctly.


On Fri, Jan 15, 2021 at 4:13 AM Viraj Jasani  wrote:

> HBASE-25404 and HBASE-25083 both are resolved with
> fixVersion 2.4.1 but corresponding commits are missing
> from branch-2.4  :(
>
>
> On 2021/01/14 23:10:41, Andrew Purtell  wrote:
> > Note the Maven temporary repository is hbase-1432 (not hbase-1430 as
> > cut-and-pasted)
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1432/
> >
> > On Thu, Jan 14, 2021 at 3:09 PM Andrew Purtell 
> wrote:
> >
> > > Please vote on this Apache hbase release candidate, hbase-2.4.1RC1.
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache hbase 2.4.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 2.4.1RC1:
> > >
> > > https://github.com/apache/hbase/tree/2.4.1RC1
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC1/
> > >
> > > Temporary Maven artifacts are available in the staging repository:
> > >
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1430/
> > >
> > > Artifacts were signed with the apurt...@apache.org key which can be
> found
> > > in:
> > >
> > > https://dist.apache.org/repos/dist/release/hbase/KEYS
> > >
> > > The API compatibility report for this RC can be found at:
> > >
> > >
> > >
> https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC1/api_compare_2.4.0_to_2.4.1RC1.html
> > >
> > > To learn more about Apache HBase, please see
> > >
> > > http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[jira] [Resolved] (HBASE-25329) Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HBASE-25329.
--
Fix Version/s: 2.3.5
   2.2.7
   Resolution: Fixed

> Dump region hashes in logs for the regions that are stuck in transition for 
> more than a configured amount of time
> -
>
> Key: HBASE-25329
> URL: https://issues.apache.org/jira/browse/HBASE-25329
> Project: HBase
>  Issue Type: Improvement
>Reporter: Caroline
>Assignee: Caroline
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.1
>
> Attachments: HBASE-25329.branch-1.000.patch, 
> HBASE-25329.branch-2.000.patch, HBASE-25329.master.000.patch
>
>
> We have metrics for number of RITs as well as number of RITs above a certain 
> threshold, but we don't have any way of keeping track of the region hashes of 
> those RITs. It would be beneficial to emit those region hashes as a metric, 
> as well as log them, so that we don't accidentally lose this information for 
> debugging the RIT at a later tiime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-25329) Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani reopened HBASE-25329:
--

Reopening for branch-2.3 and 2.2 backports.

> Dump region hashes in logs for the regions that are stuck in transition for 
> more than a configured amount of time
> -
>
> Key: HBASE-25329
> URL: https://issues.apache.org/jira/browse/HBASE-25329
> Project: HBase
>  Issue Type: Improvement
>Reporter: Caroline
>Assignee: Caroline
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1
>
> Attachments: HBASE-25329.branch-1.000.patch, 
> HBASE-25329.branch-2.000.patch, HBASE-25329.master.000.patch
>
>
> We have metrics for number of RITs as well as number of RITs above a certain 
> threshold, but we don't have any way of keeping track of the region hashes of 
> those RITs. It would be beneficial to emit those region hashes as a metric, 
> as well as log them, so that we don't accidentally lose this information for 
> debugging the RIT at a later tiime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25514) Utility to identify git commit / Jira fixVersion discrepancies for RC preparation

[Viraj Jasani (Jira)]

Viraj Jasani created HBASE-25514:


 Summary: Utility to identify git commit / Jira fixVersion 
discrepancies for RC preparation
 Key: HBASE-25514
 URL: https://issues.apache.org/jira/browse/HBASE-25514
 Project: HBase
  Issue Type: Task
Reporter: Viraj Jasani


As part of RC preparation,  we need to identify all git commits that landed on 
release branch, however their corresponding Jira is either not resolved yet or 
does not contain expected fixVersions. Only when we have git commits and 
corresponding Jiras with expected fixVersion resolved, we get all such Jiras 
included in auto-generated CHANGES.md as per Yetus changelog generator.

Proposal of this Jira is to provide such script that can be useful for all 
upcoming RC preparations and list down all Jiras where we need manual 
intervention. This utility script should use Jira API to retrieve individual 
fields and use git log to loop through commit history.

The script should identify these issues:
 # commit is reverted as per commit message
 # commit does not contain Jira number format HBASE- in message
 # Jira does not have expected fixVersion
 # Jira has expected fixVersion, but it is not yet resolved

It can take inputs as:
 # First commit hash to start excluding commits from history
 # Fix Version
 # JIRA Project Name
 # Path of project's working dir
 # Jira server url

The script should also identify all Jiras that are marked resolved with given 
fixVersion but have no commit present with Jira number included.

RC preparation also needs to find this diff because we don't want to provide 
extra Jira in CHANGELOG without any corresponding commits present on git 
(unless it is svn checkin and fixVersion is relevant).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] HBase 2.3.4 release candidate (RC4) is available

[Viraj Jasani]

HBASE-25234 doesn't have corresponding commit
present on branch-2.3 but perhaps it is fine as it
is resolved with comment: "Fixed by HBASE-25238" ?

Are we find with HBASE-25234 showing up in CHANGELOG?

On 2021/01/15 03:03:47, Huaxiang Sun  wrote: 
> Please vote on this Apache hbase release candidate,
> 
> hbase-2.3.4RC4
> 
> 
> The VOTE will remain open for at least 72 hours.
> 
> 
> [ ] +1 Release this package as Apache hbase 2.3.4
> 
> [ ] -1 Do not release this package because ...
> 
> 
> The tag to be voted on is 2.3.4RC4:
> 
> 
>   https://github.com/apache/hbase/tree/2.3.4RC4
> 
> 
> The release files, including signatures, digests, as well as CHANGES.md
> 
> and RELEASENOTES.md included in this RC can be found at:
> 
> 
>   https://dist.apache.org/repos/dist/dev/hbase/2.3.4RC4/
> 
> 
> Maven artifacts are available in a staging repository at:
> 
> 
>   https://repository.apache.org/content/repositories/orgapachehbase-1433/
> 
> 
> Artifacts were signed with the huaxiang...@apache.org key which can be
> found in:
> 
> 
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
> 
> 
> To learn more about Apache hbase, please see
> 
> 
>   http://hbase.apache.org/
> 
> 
> Thanks,
> 
> Your HBase Release Manager
> 

Re: [VOTE] Second release candidate for HBase 2.4.1 (RC1) is available

[Viraj Jasani]

HBASE-25404 and HBASE-25083 both are resolved with
fixVersion 2.4.1 but corresponding commits are missing
from branch-2.4  :(


On 2021/01/14 23:10:41, Andrew Purtell  wrote: 
> Note the Maven temporary repository is hbase-1432 (not hbase-1430 as
> cut-and-pasted)
> 
> https://repository.apache.org/content/repositories/orgapachehbase-1432/
> 
> On Thu, Jan 14, 2021 at 3:09 PM Andrew Purtell  wrote:
> 
> > Please vote on this Apache hbase release candidate, hbase-2.4.1RC1.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache hbase 2.4.1
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 2.4.1RC1:
> >
> > https://github.com/apache/hbase/tree/2.4.1RC1
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC1/
> >
> > Temporary Maven artifacts are available in the staging repository:
> >
> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1430/
> >
> > Artifacts were signed with the apurt...@apache.org key which can be found
> > in:
> >
> > https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> > The API compatibility report for this RC can be found at:
> >
> >
> > https://dist.apache.org/repos/dist/dev/hbase/2.4.1RC1/api_compare_2.4.0_to_2.4.1RC1.html
> >
> > To learn more about Apache HBase, please see
> >
> > http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
> >
> 
> 
> -- 
> Best regards,
> Andrew
> 
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
> 

[jira] [Resolved] (HBASE-25442) Refactor MultiVersionConcurrencyControl: replace readWaiters.notifyAll() with WriteEntry.notify()

[Yulin Niu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulin Niu resolved HBASE-25442.
---
Resolution: Won't Fix

> Refactor MultiVersionConcurrencyControl: replace readWaiters.notifyAll() with 
> WriteEntry.notify()
> -
>
> Key: HBASE-25442
> URL: https://issues.apache.org/jira/browse/HBASE-25442
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yulin Niu
>Assignee: Yulin Niu
>Priority: Minor
> Attachments: 50M-performance.pdf
>
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java#L217]
>  
> here the notifyAll() wake up all thread, even the read point no big enough, 
> bring extra thread switch
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java#L143]
>  
> WriteEntry object can be thread local to reuse



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25513) When the table is turned on normalize, the first region may not be merged even the size is 0

[Baiqiang Zhao (Jira)]

Baiqiang Zhao created HBASE-25513:
-

 Summary: When the table is turned on normalize, the first region 
may not be merged even the size is 0
 Key: HBASE-25513
 URL: https://issues.apache.org/jira/browse/HBASE-25513
 Project: HBase
  Issue Type: Bug
  Components: Normalizer
Affects Versions: 2.4.1, 3.0.0-alpha-1
Reporter: Baiqiang Zhao
Assignee: Baiqiang Zhao


Suppose a table has 8 regions, the sizes are [0, 10, 1, 0, 9, 0, 12, 0], the 
average region size is 4, and split is disabled.

The current Normalizer can only get three merge plans (use size to represent 
region):

[1, 0], [9, 0],[12, 0]

It can not merge the first region, even it's size is 0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)