Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

From: "Junio C Hamano" 
Sent: Thursday, February 16, 2017 12:20 AM

"Philip Oakley"  writes:


It may even be worth 'splitting' the pu branch sequence into the
existing pu (with merges from series that are selected as reasonable),
and then a pr branch (public review?) on top of that holding the rest
of the series that have been submitted, so that the CI can do a full
test on the tips of them to support those devs with limited test
capability.


I won't stop you from publishing such a pr branch yourself.


But would others see it?... (rhetorical, better thoughts below))



For patches whose merit is not clear because the problem they try to
solve is under-explained, whose solution is ill-designed, etc., IOW,
with issues that makes me judge that they are not interesting enough
for 'pu',


It is reasonble that that a project's integrator is able to make these 
decisions. For some projects they may have a layered approach of descisions 
which does allow the gradations for the submitted feature series.


Some of this does fall into dscho's differentiation between those patch 
series that should pass the CI (continuous integration) testing, and those 
that are there for CT (continuous testing) feedback. This could either be an 
extra branch marking the transition, or a named commit similar to the 
"pu^{/^### match next}", etc. In some ways it is similar to my 'pr' 
suggestion, without the inclusion of the 'all and sundry' series.


For for integrators who are willing/want to recieve any/all contributions 
for public view (usually those for projects of a more lenient and less 
critical variety), then even the CT grouping could then have those 
additional pr submissions. For Git, you provide that that 'voice of reason' 
for gatekeeping the pu branch.




it is not worth my time to deal with whitespace brekages
in them to make them not even apply, to figure out what base the
patches are meant to apply to, or to resolve conflicts caused by
them with topics already in flight, etc.

If a centralised CT service was available to the project, maybe via the 
GitHub PR process (which isn't curently used by the project) then is may be 
a way of allowing the 'all and sundry' contributors to get their ideas upto 
a basic level before even bothering yourself (because PRs do not trouble 
you).


It may need an extra gatekeeper between the passing patches (PRs) and auto 
submission  (the Heroku script thingy) which could flood the list with with 
inane changes - one only has to look at the 
http://stackoverflow.com/questions/tagged/git stream to see that.


At least if there was a break point within pu that allowed differentiation 
between the series that should fit a CI view, and those that are still at 
the CT stage, then that may help.


Thanks

Philip.

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Junio C Hamano]

"Philip Oakley"  writes:

> For an integrator, or especially a CI tool, simply checking the second
> parents of each topic merge (post fail) should at least indicate if
> the basics of the feature actually passed the tests, though it doesn't
> check for interaction issues. This could give direct author feedback!

I think that is essentially what Avery advocated for in the old
thread Christian gave us earlier in the thread, and I agree that is
a useful way to help contributors.

While we are on the topic of testing efficiently and effectively,
there are a few other things worth taking advantage of.

One obvious thing is that not all topic branches get updated every
day, even though the tip of 'pu' will be different every day (or on
days when I run the integration cycle more than once, after every
pushout), because we constantly acquire new topics or updated ones.
The tips of topics that are merged in 'pu^{/^### match next}..pu'
can be enumerated by following the first-parent chain, and they can
be tested all one-by-one, excluding the ones that have been tested
already in the previous runs.  When I counted these merges the other
day I had 27 of them, and as of this writing I have 20 of them
(because some topics were merged to 'next' in the meantime).  Among
them, only 3 are new.  Everything else would have already been
tested if such a test was done daily.  So from that point of view,
testing them all could be less expensive than bisecting; in order to
bisect a first-parent chain of 20 merges, you'd need to test 5 or
so.

Another thing that may help to prevent breakages from seeping into
'next' is that "pu^{/^### match next}..pu" are rebuilt every day,
and the order of topics that are merged are updated.  The topics
that are closer to be merged to 'next' are moved down, so that
testing the merge result would give us closer result to what would
testing 'next' in the near future would give us [*1*].  So paying
closer attention to the merges (not tips of topics) above the commit
marked as "pu^{/^### match next}" when testing would catch potential
breakage about to happen in 'next' due to unexpected interactions of
topics when merged together.

There is another point in the history that may be worth paying
closer attention, which is the tip of 'jch'.  This always is ahead
of "pu^{/^### match next}", and almost always has a handful more
topics, many of which are considered to be merged to 'next' in near
future.  The branch is meant to be at least stable enough for my
personal use (hence its name) in helping me run everyday integration
cycles.


[Footnote]

*1* Suppose there are three topics A, B and C that are not yet in
'next', and 'pu' merges them in that order.  Further suppose
that C is a lot better cooked than others.  Merging C directly
to 'next' however can expose a hidden issue that changes C
introduces alone is not sufficient and depends on A or B to be
present.  To avoid such a surprise after merging C to 'next',
when 'pu' is rebuilt, I try to reorder them so that C is merged
first on top of an equivalent of 'next', and then A and B.  For
the same reason, topics in 'master..pu^{/^### match next}' are
also reordered so that ones that are planned to be merged to
'master' soon comes at the bottom.

One natural consequence of this is that 'pu' is rebuilt directly
on top of 'master' and 'next' does not fast-forwared to 'pu'.
This arrangement also helps to spot mismerges to 'next' and
avoid the same mismerge to affect 'master' when topics are
merged to it.

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Junio C Hamano]

"Philip Oakley"  writes:

> It may even be worth 'splitting' the pu branch sequence into the
> existing pu (with merges from series that are selected as reasonable),
> and then a pr branch (public review?) on top of that holding the rest
> of the series that have been submitted, so that the CI can do a full
> test on the tips of them to support those devs with limited test
> capability.

I won't stop you from publishing such a pr branch yourself.

For patches whose merit is not clear because the problem they try to
solve is under-explained, whose solution is ill-designed, etc., IOW,
with issues that makes me judge that they are not interesting enough
for 'pu', it is not worth my time to deal with whitespace brekages
in them to make them not even apply, to figure out what base the
patches are meant to apply to, or to resolve conflicts caused by
them with topics already in flight, etc.

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

From: "Johannes Schindelin" 

Hi Philip,

On Tue, 14 Feb 2017, Philip Oakley wrote:


From: "Christian Couder" 
> On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano 
> wrote:
> > Johannes Schindelin  writes:
> >
> > > On Mon, 13 Feb 2017, Junio C Hamano wrote:
> > >
> > > > Johannes Schindelin  writes:
> > > >
> > > > > That is why I taught the Git for Windows CI job that tests the
> > > > > four upstream Git integration branches to *also* bisect test
> > > > > breakages and then upload comments to the identified commit on
> > > > > GitHub
> > > >
> > > > Good.  I do not think it is useful to try 'pu' as an aggregate
> > > > and expect it to always build and work [*1*], but your "bisect
> > > > and pinpoint" approach makes it useful to identify individual
> > > > topic that brings in a breakage.
> > >
> > > Sadly the many different merge bases[*1*] between `next` and `pu`
> > > (which are the obvious good/bad points for bisecting
> > > automatically) bring my build agents to its knees. I may have to
> > > disable the bisecting feature as a consequence.
>
> Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC
> 2017 Ideas.

There are also a few ideas at the SO answers:
http://stackoverflow.com/a/5652323/717355


Thanks for that link!

However, my main aim was not to get distracted into yet another corner of
Git that needs to be fixed (I am on enough of those projects already).

I was merely surprised (and not in a good way) that a plenty ordinary
bisect between `next` and `pu` all of a sudden tested a *one year old*
commit whether it was good or not.

And I doubt that the strategy to mark all second parents of all merge
commits in pu..next as "good" would work well, as the merge bases *still*
would have to be tested.


I was expecting that if all the second parents were marked as good, then 
there would be no merge bases, as there shouldn't be a forked graph, just 
the linear string of pearls - if bisect doesn't do that then there's an 
failed optimisation to be had.


I don't see anything in the `git bisect --help` page that would indicate 
that the merges themselves are omitted from the bisection.




I guess what I have to resort to is this: if I know that `next` tests
fine, and that `pu` fails, I shall mark all merge bases as "good". I am
sure this has its own set of pitfalls, undoubtedly costing me more time on
that front...

But at least my cursory analysis of this idea seems to make sense: I use
`next` essentially as a catch-all to verify that the breakage has entered
`pu`, but not yet `next`. This reasoning makes sense, given that we know
the waterfall topology of pu/next/master/maint: patches enter from left to
right, i.e. anything that entered `pu` may later enter `next`, but not
vice versa.


It may even be worth 'splitting' the pu branch sequence into the existing pu 
(with merges from series that are selected as reasonable), and then a pr 
branch (public review?) on top of that holding the rest of the series that 
have been submitted, so that the CI can do a full test on the tips of them 
to support those devs with limited test capability.





Ciao,
Dscho

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

[sorry for the repeated emails - I'd prepared it off line, and then suffered 
a number of auto send actions]

From: "Junio C Hamano" 

"Philip Oakley"  writes:


In the next..pu case the abstraction is in the other direction, we
have potentially multiple points of infection (from feature branches),
and a broad test (the whole test suite). In this case I believe we
would like to investigate initially the --first-parent line with a
classic bisect for the first point of failure (obviously including
feature branch merges). This would identify which feature merge, or
regular commit, created the first breakage.


If you are going first-parent, you would limit the bisection to a
single-strand-of-pearls, and I agree that it is a good strategy to
find which topic branch merge broke the tip of 'pu'.

If we assume that there is no funny interaction among topics that
cancel a breakage brought in by one topic with another breakage by
another topic, then no matter how many broken topics there are, I
agree that we would get to the first broken topic.




A good thing that comes once we assume that topics are more-or-less
independent is that we could rebuild 'pu' minus the broken topic
identified by the above procedure and repeat it to find other broken
topics, still using the --first-parent bisection, because master..pu
is a linear sequence of merges of individual topics.



For an integrator, or especially a CI tool, simply checking the second 
parents of each topic merge (post fail) should at least indicate if the 
basics of the feature actually passed the tests, though it doesn't check for 
interaction issues. This could give direct author feedback!

--
Philip 

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

From: "Junio C Hamano" 

"Philip Oakley"  writes:


There are also a few ideas at the SO answers:
http://stackoverflow.com/a/5652323/717355


I vaguely recall that I saw somebody said the same "mark tips of
topics as good" on the list and answered with why it does not quite
work, though.


I think you may mean
https://public-inbox.org/git/7v8vyam5la@alter.siamese.dyndns.org/

I think we are thinking of opposite abstractions.

For regular bisect, the assumption (to a first order) is that there is a
single point of infection of a single persistent bug with a well defined
test, and that the goal is to find the point of first infection, as all
other incidents of the bug are in successor commits, which are all infected.
The fail-fix-break again sequence you mentioned in that thread is to my mind
a red herring as it contradicts the normal bisection assumptions (but see
below).

In the next..pu case the abstraction is in the other direction, we have
potentially multiple points of infection (from feature branches), and a
broad test (the whole test suite). In this case I believe we would like to
investigate initially the --first-parent line with a classic bisect for the
first point of failure (obviously including feature branch merges). This
would identify which feature merge, or regular commit, created the first
breakage.

Once the first point of failure has been identified, for the next..pu case,
each of the post-fail second parents of merge commits _could_ then also be
checked (which is a linear search, not a bisection), to identify any
additional feature branches that need attention. This second stage search
would probably be an option, but if the merging sequence onto pu is
generally from good to bad, then the search is likely to be short. At least
for a CI system this 2nd stage could provide useful feedback to the authors
of their mistakes...

I haven't looked back at the actual patches in that thread, so they may not
have followed my expectation of the --multi-bug (TM) search algorithm.
--

Philip

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

From: "Junio C Hamano" 

"Philip Oakley"  writes:


There are also a few ideas at the SO answers:
http://stackoverflow.com/a/5652323/717355


I vaguely recall that I saw somebody said the same "mark tips of
topics as good" on the list and answered with why it does not quite
work, though.


I think you may mean
https://public-inbox.org/git/7v8vyam5la@alter.siamese.dyndns.org/

I think we are thinking of opposite abstractions.

For regular bisect, the assumption (to a first order) is that there is a
single point of infection of a single persistent bug with a well defined
test, and that the goal is to find the point of first infection, as all
other incidents of the bug are in successor commits, which are all infected.
The fail-fix-break again sequence you mentioned in that thread is to my mind
a red herring as it contradicts the normal bisection assumptions (but see
below).

In the next..pu case the abstraction is in the other direction, we have
potentially multiple points of infection (from feature branches), and a
broad test (the whole test suite). In this case I believe we would like to
investigate initially the --first-parent line with a classic bisect for the
first point of failure (obviously including feature branch merges). This
would identify which feature merge, or regular commit, created the first
breakage.

Once the first point of failure has been identified, for the next..pu case,
each of the post-fail second parents of merge commits _could_ then also be
checked (which is a linear search, not a bisection), to identify any
additional feature branches that need attention. This second stage search
would probably be an option, but if the merging sequence onto pu is
generally from good to bad, then the search is likely to be short. At least
for a CI system this 2nd stage could provide useful feedback to the authors
of their mistakes...

I haven't looked back at the actual patches in that thread, so they may not
have followed my expectation of the --multi-bug (TM) search algorithm.
--

Philip

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Junio C Hamano]

"Philip Oakley"  writes:

> In the next..pu case the abstraction is in the other direction, we
> have potentially multiple points of infection (from feature branches),
> and a broad test (the whole test suite). In this case I believe we
> would like to investigate initially the --first-parent line with a
> classic bisect for the first point of failure (obviously including
> feature branch merges). This would identify which feature merge, or
> regular commit, created the first breakage.

If you are going first-parent, you would limit the bisection to a
single-strand-of-pearls, and I agree that it is a good strategy to
find which topic branch merge broke the tip of 'pu'.

If we assume that there is no funny interaction among topics that
cancel a breakage brought in by one topic with another breakage by
another topic, then no matter how many broken topics there are, I
agree that we would get to the first broken topic.

A good thing that comes once we assume that topics are more-or-less
independent is that we could rebuild 'pu' minus the broken topic
identified by the above procedure and repeat it to find other broken
topics, still using the --first-parent bisection, because master..pu
is a linear sequence of merges of individual topics.

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

From: "Junio C Hamano" 

"Philip Oakley"  writes:


There are also a few ideas at the SO answers:
http://stackoverflow.com/a/5652323/717355


I vaguely recall that I saw somebody said the same "mark tips of
topics as good" on the list and answered with why it does not quite
work, though.

I think you may mean 
https://public-inbox.org/git/7v8vyam5la@alter.siamese.dyndns.org/


I think we are thinking of opposite abstractions.

For regular bisect, the assumption (to a first order) is that there is a 
single point of infection of a single persistent bug with a well defined 
test, and that the goal is to find the point of first infection, as all 
other incidents of the bug are in successor commits, which are all infected. 
The fail-fix-break again sequence you mentioned in that thread is to my mind 
a red herring as it contradicts the normal bisection assumptions (but see 
below).


In the next..pu case the abstraction is in the other direction, we have 
potentially multiple points of infection (from feature branches), and a 
broad test (the whole test suite). In this case I believe we would like to 
investigate initially the --first-parent line with a classic bisect for the 
first point of failure (obviously including feature branch merges). This 
would identify which feature merge, or regular commit, created the first 
breakage.


Once the first point of failure has been identified, for the next..pu case, 
each of the post-fail second parents of merge commits _could_ then also be 
checked (which is a linear search, not a bisection), to identify any 
additional feature branches that need attention. This second stage search 
would probably be an option, but if the merging sequence onto pu is 
generally from good to bad, then the search is likely to be short. At least 
for a CI system this 2nd stage could provide useful feedback to the authors 
of their mistakes...


I haven't looked back at the actual patches in that thread, so they may not 
have followed my expectation of the --multi-bug (TM) search algorithm.

--

Philip

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Johannes Schindelin]

Hi Philip,

On Tue, 14 Feb 2017, Philip Oakley wrote:

> From: "Christian Couder" 
> > On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano 
> > wrote:
> > > Johannes Schindelin  writes:
> > >
> > > > On Mon, 13 Feb 2017, Junio C Hamano wrote:
> > > >
> > > > > Johannes Schindelin  writes:
> > > > >
> > > > > > That is why I taught the Git for Windows CI job that tests the
> > > > > > four upstream Git integration branches to *also* bisect test
> > > > > > breakages and then upload comments to the identified commit on
> > > > > > GitHub
> > > > >
> > > > > Good.  I do not think it is useful to try 'pu' as an aggregate
> > > > > and expect it to always build and work [*1*], but your "bisect
> > > > > and pinpoint" approach makes it useful to identify individual
> > > > > topic that brings in a breakage.
> > > >
> > > > Sadly the many different merge bases[*1*] between `next` and `pu`
> > > > (which are the obvious good/bad points for bisecting
> > > > automatically) bring my build agents to its knees. I may have to
> > > > disable the bisecting feature as a consequence.
> >
> > Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC
> > 2017 Ideas.
> 
> There are also a few ideas at the SO answers:
> http://stackoverflow.com/a/5652323/717355

Thanks for that link!

However, my main aim was not to get distracted into yet another corner of
Git that needs to be fixed (I am on enough of those projects already).

I was merely surprised (and not in a good way) that a plenty ordinary
bisect between `next` and `pu` all of a sudden tested a *one year old*
commit whether it was good or not.

And I doubt that the strategy to mark all second parents of all merge
commits in pu..next as "good" would work well, as the merge bases *still*
would have to be tested.

I guess what I have to resort to is this: if I know that `next` tests
fine, and that `pu` fails, I shall mark all merge bases as "good". I am
sure this has its own set of pitfalls, undoubtedly costing me more time on
that front...

But at least my cursory analysis of this idea seems to make sense: I use
`next` essentially as a catch-all to verify that the breakage has entered
`pu`, but not yet `next`. This reasoning makes sense, given that we know
the waterfall topology of pu/next/master/maint: patches enter from left to
right, i.e. anything that entered `pu` may later enter `next`, but not
vice versa.

Ciao,
Dscho

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Johannes Schindelin]

Hi Christian,

On Wed, 15 Feb 2017, Christian Couder wrote:

> On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano 
> wrote:
> 
> > Probably a less resource intensive approach is to find the tips of the
> > topics not in 'next' but in 'pu' and test them.  That would give you
> > which topic(s) are problematic, which is a better starting point than
> > "Oh, 'pu' does not build".  After identifying which branch is
> > problematic, bisection of individual topic would be of more manageable
> > size.
> 
> It is still probably more resource intensive than it couls be.

Indeed.

Ciao,
Dscho

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Junio C Hamano]

"Philip Oakley"  writes:

> There are also a few ideas at the SO answers:
> http://stackoverflow.com/a/5652323/717355

I vaguely recall that I saw somebody said the same "mark tips of
topics as good" on the list and answered with why it does not quite
work, though.

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Philip Oakley]

From: "Christian Couder" 
On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano  
wrote:

Johannes Schindelin  writes:


On Mon, 13 Feb 2017, Junio C Hamano wrote:


Johannes Schindelin  writes:

> That is why I taught the Git for Windows CI job that tests the four
> upstream Git integration branches to *also* bisect test breakages and
> then upload comments to the identified commit on GitHub

Good.  I do not think it is useful to try 'pu' as an aggregate and
expect it to always build and work [*1*], but your "bisect and
pinpoint" approach makes it useful to identify individual topic that
brings in a breakage.


Sadly the many different merge bases[*1*] between `next` and `pu` (which
are the obvious good/bad points for bisecting automatically) bring my
build agents to its knees. I may have to disable the bisecting feature 
as

a consequence.


Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC
2017 Ideas.


There are also a few ideas at the SO answers: 
http://stackoverflow.com/a/5652323/717355





Probably a less resource intensive approach is to find the tips of
the topics not in 'next' but in 'pu' and test them.  That would give
you which topic(s) are problematic, which is a better starting point
than "Oh, 'pu' does not build".  After identifying which branch is
problematic, bisection of individual topic would be of more manageable
size.


It is still probably more resource intensive than it couls be.

[...]


This is one of these times I wish "git bisect --first-parent" were
available.


Implementing "git bisect --first-parent" is also part of the GSoC 2017 
Ideas.


By the way it should not be very difficult as a patch to do this and
more was proposed a long time ago:

https://public-inbox.org/git/4d3cddf9.6080...@intel.com/


The above "log" gives me 27 merges right now, which
should be bisectable within 5 rounds to identify a single broken
topic (if there is only one breakage, that is).



--
Philip 

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Junio C Hamano]

Christian Couder  writes:

> By the way it should not be very difficult as a patch to do this and
> more was proposed a long time ago:
>
> https://public-inbox.org/git/4d3cddf9.6080...@intel.com/

Thanks for a link.  The one I found most interesting in the thread
is by Avery [*1*], where he explains why "first-parent" bisection
makes sense in "many people develop topics of their own, and they
are aggregated into an integration branch" environment:

Basically, we push/fetch *all* the branches from *everybody* into a
single repo, and build all of them as frequently as we can.  If you
think about it, if you have all the branches that someone might have
pulled/merged from, then you don't have to think of the git history
as a whole complicated DAG; you can just think of it as a whole
bunch of separate chunks of linear history.  Moreover, as long as
people are careful to only pull from a branch when that branch is
passing all tests - which you can easily see by looking at the
gitbuilder console - then playing inside each of these chunks of
linear history can help you figure out where particular bugs were
introduced during "messy" branches.

It also allows you a nice separation of concerns.  The owner of the
mainline branch (the "integration manager" person) only really cares
about which branch they merged that caused a problem, because that
person doesn't want to fix bugs, he/she simply wants to know who
owns the failing branch, so that person can fix *their* bug and
their branch will merge without breaking things.

[Reference]

*1* 
https://public-inbox.org/git/aanlktinwbm9gczhgeqcboepov0_xv7ujyqvc7j13q...@mail.gmail.com/

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Christian Couder]

On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano  wrote:
> Johannes Schindelin  writes:
>
>> On Mon, 13 Feb 2017, Junio C Hamano wrote:
>>
>>> Johannes Schindelin  writes:
>>>
>>> > That is why I taught the Git for Windows CI job that tests the four
>>> > upstream Git integration branches to *also* bisect test breakages and
>>> > then upload comments to the identified commit on GitHub
>>>
>>> Good.  I do not think it is useful to try 'pu' as an aggregate and
>>> expect it to always build and work [*1*], but your "bisect and
>>> pinpoint" approach makes it useful to identify individual topic that
>>> brings in a breakage.
>>
>> Sadly the many different merge bases[*1*] between `next` and `pu` (which
>> are the obvious good/bad points for bisecting automatically) bring my
>> build agents to its knees. I may have to disable the bisecting feature as
>> a consequence.

Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC
2017 Ideas.

> Probably a less resource intensive approach is to find the tips of
> the topics not in 'next' but in 'pu' and test them.  That would give
> you which topic(s) are problematic, which is a better starting point
> than "Oh, 'pu' does not build".  After identifying which branch is
> problematic, bisection of individual topic would be of more manageable
> size.

It is still probably more resource intensive than it couls be.

[...]

> This is one of these times I wish "git bisect --first-parent" were
> available.

Implementing "git bisect --first-parent" is also part of the GSoC 2017 Ideas.

By the way it should not be very difficult as a patch to do this and
more was proposed a long time ago:

https://public-inbox.org/git/4d3cddf9.6080...@intel.com/

> The above "log" gives me 27 merges right now, which
> should be bisectable within 5 rounds to identify a single broken
> topic (if there is only one breakage, that is).

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Junio C Hamano]

Johannes Schindelin  writes:

> On Mon, 13 Feb 2017, Junio C Hamano wrote:
>
>> Johannes Schindelin  writes:
>> 
>> > That is why I taught the Git for Windows CI job that tests the four
>> > upstream Git integration branches to *also* bisect test breakages and
>> > then upload comments to the identified commit on GitHub
>> 
>> Good.  I do not think it is useful to try 'pu' as an aggregate and
>> expect it to always build and work [*1*], but your "bisect and
>> pinpoint" approach makes it useful to identify individual topic that
>> brings in a breakage.
>
> Sadly the many different merge bases[*1*] between `next` and `pu` (which
> are the obvious good/bad points for bisecting automatically) bring my
> build agents to its knees. I may have to disable the bisecting feature as
> a consequence.

Probably a less resource intensive approach is to find the tips of
the topics not in 'next' but in 'pu' and test them.  That would give
you which topic(s) are problematic, which is a better starting point
than "Oh, 'pu' does not build".  After identifying which branch is
problematic, bisection of individual topic would be of more manageable
size.

  $ git log --first-parent --oneline 'pu^{/^### match next}..pu'

will you the merges of topics left outside 'next'.  I often reorder
to make the ones that look more OK than others closer to the bottom,
and if the breakages caused by them are caught earlier than they hit
'next', that would be ideal.

This is one of these times I wish "git bisect --first-parent" were
available.  The above "log" gives me 27 merges right now, which
should be bisectable within 5 rounds to identify a single broken
topic (if there is only one breakage, that is).

Re: [git-for-windows] Re: Continuous Testing of Git on Windows

[Johannes Schindelin]

Hi,

On Mon, 13 Feb 2017, Junio C Hamano wrote:

> Johannes Schindelin  writes:
> 
> > That is why I taught the Git for Windows CI job that tests the four
> > upstream Git integration branches to *also* bisect test breakages and
> > then upload comments to the identified commit on GitHub
> 
> Good.  I do not think it is useful to try 'pu' as an aggregate and
> expect it to always build and work [*1*], but your "bisect and
> pinpoint" approach makes it useful to identify individual topic that
> brings in a breakage.

Sadly the many different merge bases[*1*] between `next` and `pu` (which
are the obvious good/bad points for bisecting automatically) bring my
build agents to its knees. I may have to disable the bisecting feature as
a consequence.

Ciao,
Johannes

Footnote *1*: There are currently 21, some of which stemming back from a
year ago. For bisecting, they all have to be tested individually, putting
a major dent into bisect's otherwise speedy process.