Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Ulf Hermann]

Hi,

I'd like to remind everyone to the fact that the current dependencies 
model was intentionally chosen for faster iteration during early Qt 6 
development. We _wanted_ to allow breaking things in other modules so 
that we could move on. The breakage was to be fixed later, possibly 
after multiple other changes were in.


Before this, we had a "private API revision" in qtdeclarative, and every 
time we changed something about the private API, the revision had to be 
incremented, then all dependent modules had to receive some #if'ery to 
support both, old and new API revisions, and only then the change could 
be fully integrated. Afterwards, all the #if'ery could be removed again. 
This is obviously not a very efficient way of managing changes.


We may consider the phase of rapid iteration to be over and revert to 
the old model, but that should be a choice, too, not just a side effect 
of some test strategy.


Ulf
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Edward Welbourne]

Leaving aside the question of what to test when integrating changes to
each given module, I pause to consider how we could sensibly implement
the necessary information for telling automated systems to do that
testing.  That may place constraints on what we can do, that might bound
the discussion of what we want to do / what it makes sense to do.

Presumably we could add a new stanza to dependencies.yaml, a top level
peer of the dependencies stanza (which I'd rather call prerequisites,
but I guess it's too late for that), named dependants, with suitable
meta-data to say what version of it to test against and what to test -
build, test, example - in the dependant, as a prerequisite of accepting
a change in the upstream module in Coin.

PreCheck shall, naturally, test everything that integration shall test.
An option to consider is to have PreCheck perform a broader set of tests
of dependant modules - e.g. the new stanza could have a required:
option, as in the dependencies stanza, whose false means Coin doesn't
reject based on the dependant failing, where PreCheck maybe would.
Whether Coin runs those tests (reporting, but not failing on, any
failures) is a choice we can make based on practicalities like load.

The committer of the new version of the upstream module would then be
able to get changes to the downstream through integration, that prepare
it for the upstream change, and update the upstream commit that would
have broken the downstream's tests otherwise, to show the new sha1 of
the downstream module as the basis for the dependant-testing.

Relevant module owners can then haggle over how far upstream each module
gets protected by being identified as dependant.  That'll be a balancing
act between the upstream not wanting to be slowed too much in Coin by
the building and testing of dependants, vs the dependants wanting that
protection.  That haggling would surely happen in Gerrit changes to the
upstream's dependencies.yaml file, which seems like a suitable enough
forum in which to, at least, open such a discussion.

Which is all just a sketch, but hopefully gives some shape to what's
practically implementable.  Can anyone spot any obvious holes in this
approach ?  Or other constraints it'd impose on what we can do ?

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Mitch Curtis]

From: Development  On Behalf Of Shawn 
Rutledge
Sent: Tuesday, 15 June 2021 10:57 PM
To: development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in upstream 
modules are tested with downstream modules before merging


On 2021 Jun 7, at 15:30, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

Some I can think of:

- Slower git operations.
- Slower build times for qtdeclarative devs. You can use e.g. ninja targets, 
but you would have to do this every time you build. Ideally we’d add a 
configure option for this, like -no-gui in qtbase, so that they don’t need to 
build qtquickcontrols2.

FWIW, I’m routinely building all of Qt together (the modules that I have 
anyway) with cmake and ninja.  (It’s around 15k steps, building all autotests 
but not examples.  A Zen3 processor and ccache make it bearable.  Before that, 
I was using icecc.)  So the cases where I have broken something in controls by 
making a change in qtdeclarative are usually only test failures (behavior 
changes), not build failures, right?  (Or at least not build failures that I 
didn’t know about.)  I never run all autotests in all modules locally, because 
it takes too long.  Even running all tests in qtdeclarative takes too long, 
except when the change is so invasive that I know I really need to.  If I’m 
worried about a specific change breaking controls, I run those tests.  (The 
script to run tests in xephyr is handy, so I can keep using the computer while 
they are running.)

“only” test failures, yes. :D Build failures would be preferable because 
they’re more obvious and easily fixed.

I rarely run all tests for a module when making a change. I’d never get any 
work done if I did.

- Creator would probably be quite a bit slower to work with having both of 
these in one project.

Qtdeclarative is already a rather big module.  And Creator is slow, for sure.  
It’s been a couple of years since I could use it at all on my older laptops 
without enough memory.


This why I like the idea of just running qtquickcontrols2 tests for each 
qtdeclarative change: it doesn’t waste developer time (and again, CI 
integrations don’t count since they shouldn’t be wasting developer time).

Hopefully that will work, yeah.


Though then you do lose the ability to use controls in examples, which does 
seem like a huge loss.

It depends whether we care about being able to automatically test examples 
without building controls first, right?  Otherwise I thought the existing 
advice lately is to go ahead and use controls in examples?

I’m not sure to be honest. I know the consensus is that we should use controls 
in examples but I don’t know how that’s supposed to work in terms of module 
dependencies.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Shawn Rutledge]

On 2021 Jun 7, at 15:30, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

Some I can think of:

- Slower git operations.
- Slower build times for qtdeclarative devs. You can use e.g. ninja targets, 
but you would have to do this every time you build. Ideally we’d add a 
configure option for this, like -no-gui in qtbase, so that they don’t need to 
build qtquickcontrols2.

FWIW, I’m routinely building all of Qt together (the modules that I have 
anyway) with cmake and ninja.  (It’s around 15k steps, building all autotests 
but not examples.  A Zen3 processor and ccache make it bearable.  Before that, 
I was using icecc.)  So the cases where I have broken something in controls by 
making a change in qtdeclarative are usually only test failures (behavior 
changes), not build failures, right?  (Or at least not build failures that I 
didn’t know about.)  I never run all autotests in all modules locally, because 
it takes too long.  Even running all tests in qtdeclarative takes too long, 
except when the change is so invasive that I know I really need to.  If I’m 
worried about a specific change breaking controls, I run those tests.  (The 
script to run tests in xephyr is handy, so I can keep using the computer while 
they are running.)

- Creator would probably be quite a bit slower to work with having both of 
these in one project.

Qtdeclarative is already a rather big module.  And Creator is slow, for sure.  
It’s been a couple of years since I could use it at all on my older laptops 
without enough memory.

This why I like the idea of just running qtquickcontrols2 tests for each 
qtdeclarative change: it doesn’t waste developer time (and again, CI 
integrations don’t count since they shouldn’t be wasting developer time).

Hopefully that will work, yeah.

Though then you do lose the ability to use controls in examples, which does 
seem like a huge loss.

It depends whether we care about being able to automatically test examples 
without building controls first, right?  Otherwise I thought the existing 
advice lately is to go ahead and use controls in examples?

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Mitch Curtis]

Some I can think of:

- Slower git operations.
- Slower build times for qtdeclarative devs. You can use e.g. ninja targets, 
but you would have to do this every time you build. Ideally we’d add a 
configure option for this, like -no-gui in qtbase, so that they don’t need to 
build qtquickcontrols2.
- Creator would probably be quite a bit slower to work with having both of 
these in one project.

This why I like the idea of just running qtquickcontrols2 tests for each 
qtdeclarative change: it doesn’t waste developer time (and again, CI 
integrations don’t count since they shouldn’t be wasting developer time). 
Though then you do lose the ability to use controls in examples, which does 
seem like a huge loss.

From: David Skoland 
Sent: Monday, 7 June 2021 1:30 PM
To: Mitch Curtis 
Cc: Volker Hilsheimer ; development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in upstream 
modules are tested with downstream modules before merging

I would really be interested if there are other leaf modules that have this 
issue. If it's just qtquickcontrols2 then perhaps merging into qtdeclarative 
would be the easiest solution after all.

Taking a quick look through the qtquickcontrols2’s changes, it seems a large 
number(half?) of them are just Qt Submodule Update Bot and Qt Cherry-pick Bot 
changes. I would think that merging qqc2 into qtdeclarative could make things 
simpler, but I would like to hear if anyone knows about any potential downsides 
this may have. Is it a realistic move to make at this point?

On 7 Jun 2021, at 10:22, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

-Original Message-
From: Volker Hilsheimer 
mailto:volker.hilshei...@qt.io>>
Sent: Friday, 4 June 2021 4:51 PM
To: Mitch Curtis mailto:mitch.cur...@qt.io>>
Cc: development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in upstream
modules are tested with downstream modules before merging


On 4 Jun 2021, at 16:10, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:


-Original Message-
From: Volker Hilsheimer 
mailto:volker.hilshei...@qt.io>>
Sent: Friday, 4 June 2021 2:45 PM
To: Mitch Curtis mailto:mitch.cur...@qt.io>>
Cc: development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in
upstream modules are tested with downstream modules before merging


On 4 Jun 2021, at 13:56, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

Hi.

A common problem I see is that a change in say qtbase or
qtdeclarative can
cause test failures in modules that depend on them after that change
is merged. As a result, dependency updates for the downstream modules
can be blocked, requiring the module maintainer to look into the
failure, only to discover that it is caused by an upstream module.
This causes a lot of time and effort to be diverted away from regular
work. Both upstream and downstream developers would benefit from
having more immediate CI feedback on these kinds of changes.


https://bugreports.qt.io/browse/QTBUG-79454 was created to track
this
problem. So far the ideas suggested have been:


- Run tests of dependent modules when testing a change, in addition
to
the regular tests for that module

- Merge repositories of more tightly coupled modules (e.g. move
qtquickcontrols2 into qtdeclarative)


It would be beneficial to discuss the advantages and disadvantages
of
these and other solutions so that we can address this problem.


Cheers.


Excellent topic for the upcoming Qt Contributors Summit, Mitch, great
if you can add it to the list of topics:

https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program

Which doesn’t mean that we shouldn't start the discussion on the
list, and maybe we can then conclude it at the summit.


Cheers,
Volker

OK, added it to the list.

Thanks.


As for the topic:

I do think that we have cases where too many modules are lumped into a
single repo, and cases where we have too much granularity.

For the former: I see no immediate reason why QtNetwork needs to be in
qtbase (network tests failing is a major cause of unrelated integrations
failing), or why the Qt Positioning module needs to be in qtlocation.

For the latter: Qt Quick Controls 2 might be better off sharing a commit
history with QtQuick

However, while we can make adjustments, I also think that there’s no
perfect solution. It’s always a tradeoff between speed and autonomy on the
one, and integrity across dependencies on the other hand.

I agree.


Running more tests is a similar tradeoff, and I don’t think blocking an
integration into e.g. qtbase because it breaks a leaf module is a good solution
to the problem. We should aim for fast feedback, and short meantime to
recover such breakages, and making CI runs take even longer helps with
neither. In particular if we then can’t make any such changes even though a
follow up change in a leaf module is already available and just needs to wait

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[David Skoland]

I would really be interested if there are other leaf modules that have this 
issue. If it's just qtquickcontrols2 then perhaps merging into qtdeclarative 
would be the easiest solution after all.

Taking a quick look through the qtquickcontrols2’s changes, it seems a large 
number(half?) of them are just Qt Submodule Update Bot and Qt Cherry-pick Bot 
changes. I would think that merging qqc2 into qtdeclarative could make things 
simpler, but I would like to hear if anyone knows about any potential downsides 
this may have. Is it a realistic move to make at this point?

On 7 Jun 2021, at 10:22, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

-Original Message-
From: Volker Hilsheimer 
mailto:volker.hilshei...@qt.io>>
Sent: Friday, 4 June 2021 4:51 PM
To: Mitch Curtis mailto:mitch.cur...@qt.io>>
Cc: development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in upstream
modules are tested with downstream modules before merging

On 4 Jun 2021, at 16:10, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

-Original Message-
From: Volker Hilsheimer 
mailto:volker.hilshei...@qt.io>>
Sent: Friday, 4 June 2021 2:45 PM
To: Mitch Curtis mailto:mitch.cur...@qt.io>>
Cc: development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in
upstream modules are tested with downstream modules before merging

On 4 Jun 2021, at 13:56, Mitch Curtis 
mailto:mitch.cur...@qt.io>> wrote:

Hi.

A common problem I see is that a change in say qtbase or
qtdeclarative can
cause test failures in modules that depend on them after that change
is merged. As a result, dependency updates for the downstream modules
can be blocked, requiring the module maintainer to look into the
failure, only to discover that it is caused by an upstream module.
This causes a lot of time and effort to be diverted away from regular
work. Both upstream and downstream developers would benefit from
having more immediate CI feedback on these kinds of changes.

https://bugreports.qt.io/browse/QTBUG-79454 was created to track
this
problem. So far the ideas suggested have been:

- Run tests of dependent modules when testing a change, in addition
to
the regular tests for that module
- Merge repositories of more tightly coupled modules (e.g. move
qtquickcontrols2 into qtdeclarative)

It would be beneficial to discuss the advantages and disadvantages
of
these and other solutions so that we can address this problem.

Cheers.


Excellent topic for the upcoming Qt Contributors Summit, Mitch, great
if you can add it to the list of topics:

https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program

Which doesn’t mean that we shouldn't start the discussion on the
list, and maybe we can then conclude it at the summit.


Cheers,
Volker

OK, added it to the list.


Thanks.


As for the topic:

I do think that we have cases where too many modules are lumped into a
single repo, and cases where we have too much granularity.

For the former: I see no immediate reason why QtNetwork needs to be in
qtbase (network tests failing is a major cause of unrelated integrations
failing), or why the Qt Positioning module needs to be in qtlocation.

For the latter: Qt Quick Controls 2 might be better off sharing a commit
history with QtQuick

However, while we can make adjustments, I also think that there’s no
perfect solution. It’s always a tradeoff between speed and autonomy on the
one, and integrity across dependencies on the other hand.

I agree.

Running more tests is a similar tradeoff, and I don’t think blocking an
integration into e.g. qtbase because it breaks a leaf module is a good solution
to the problem. We should aim for fast feedback, and short meantime to
recover such breakages, and making CI runs take even longer helps with
neither. In particular if we then can’t make any such changes even though a
follow up change in a leaf module is already available and just needs to wait
for the dependency update. Then we are in a deadlock, unless we keep the
code in qtbase working for both versions of e.g. qtdeclarative, which is not
always practical.

I could be wrong, but perhaps this is what Ossi's 
https://bugreports.qt.io/browse/COIN-133 was targeting. In any case, I can't 
really speak for qtbase as I rarely work on it.

The problem is perhaps not that we break things, but that we don’t notice
for too long that we broke things? And once we do notice, we often seem to
be unwilling to revert the change that did cause the breakage, if things don’t
get fixed quickly.

This might be the case for qtbase and its leaf modules, but not for 
qtdeclarative and qtquickcontrols2. The typical situation there is: 
qtquickcontrols2 dependency update fails. I immediately cannot think of any 
relevant changes that would cause such a failure, so I quickly look through git 
log. If I can't find anything there, I need to start bisecting with 
qtdeclarative (and this 

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Mitch Curtis]

Good point. I guess you implied this in "all the relevant examples", but just 
to spell it out: this would also allow other leaf repositories that currently 
depend on qtdeclarative.git, but might not want to depend on 
qtquickcontrols2.git for whatever reason, to use controls in their examples.

From: Tuukka Turunen 
Sent: Friday, 4 June 2021 9:35 PM
To: Volker Hilsheimer ; Mitch Curtis 

Cc: development@qt-project.org
Subject: Re: [Development] Solutions for ensuring that changes in upstream 
modules are tested with downstream modules before merging


Hi,

Having the controls within declarative and also using the controls more in all 
the relevant examples would be beneficial for the users. So I think this might 
be one dependency we could add, not only for testing.

Yours,

Tuukka


Lähettäjä: Development 
mailto:development-boun...@qt-project.org>> 
käyttäjän Volker Hilsheimer 
mailto:volker.hilshei...@qt.io>> puolesta
Lähetetty: perjantaina, kesäkuuta 4, 2021 5:55 ip.
Vastaanottaja: Mitch Curtis
Kopio: development@qt-project.org
Aihe: Re: [Development] Solutions for ensuring that changes in upstream modules 
are tested with downstream modules before merging

> On 4 Jun 2021, at 16:10, Mitch Curtis 
> mailto:mitch.cur...@qt.io>> wrote:
>
>> -Original Message-
>> From: Volker Hilsheimer 
>> mailto:volker.hilshei...@qt.io>>
>> Sent: Friday, 4 June 2021 2:45 PM
>> To: Mitch Curtis mailto:mitch.cur...@qt.io>>
>> Cc: development@qt-project.org
>> Subject: Re: [Development] Solutions for ensuring that changes in upstream
>> modules are tested with downstream modules before merging
>>
>>> On 4 Jun 2021, at 13:56, Mitch Curtis 
>>> mailto:mitch.cur...@qt.io>> wrote:
>>>
>>> Hi.
>>>
>>> A common problem I see is that a change in say qtbase or qtdeclarative can
>> cause test failures in modules that depend on them after that change is
>> merged. As a result, dependency updates for the downstream modules can
>> be blocked, requiring the module maintainer to look into the failure, only to
>> discover that it is caused by an upstream module. This causes a lot of time
>> and effort to be diverted away from regular work. Both upstream and
>> downstream developers would benefit from having more immediate CI
>> feedback on these kinds of changes.
>>>
>>> https://bugreports.qt.io/browse/QTBUG-79454 was created to track this
>> problem. So far the ideas suggested have been:
>>>
>>> - Run tests of dependent modules when testing a change, in addition to
>> the regular tests for that module
>>> - Merge repositories of more tightly coupled modules (e.g. move
>> qtquickcontrols2 into qtdeclarative)
>>>
>>> It would be beneficial to discuss the advantages and disadvantages of
>> these and other solutions so that we can address this problem.
>>>
>>> Cheers.
>>
>>
>> Excellent topic for the upcoming Qt Contributors Summit, Mitch, great if you
>> can add it to the list of topics:
>>
>> https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program
>>
>> Which doesn't mean that we shouldn't start the discussion on the list, and
>> maybe we can then conclude it at the summit.
>>
>>
>> Cheers,
>> Volker
>
> OK, added it to the list.
>

Thanks.


As for the topic:

I do think that we have cases where too many modules are lumped into a single 
repo, and cases where we have too much granularity.

For the former: I see no immediate reason why QtNetwork needs to be in qtbase 
(network tests failing is a major cause of unrelated integrations failing), or 
why the Qt Positioning module needs to be in qtlocation.

For the latter: Qt Quick Controls 2 might be better off sharing a commit 
history with QtQuick

However, while we can make adjustments, I also think that there's no perfect 
solution. It's always a tradeoff between speed and autonomy on the one, and 
integrity across dependencies on the other hand.


Running more tests is a similar tradeoff, and I don't think blocking an 
integration into e.g. qtbase because it breaks a leaf module is a good solution 
to the problem. We should aim for fast feedback, and short meantime to recover 
such breakages, and making CI runs take even longer helps with neither. In 
particular if we then can't make any such changes even though a follow up 
change in a leaf module is already available and just needs to wait for the 
dependency update. Then we are in a deadlock, unless we keep the code in qtbase 
working for both versions of e.g. qtdeclarative, which is not always practical.


The problem is perhaps not that we break things, but that we don't notice for 
too long that we broke things? And once we do notice, we often seem to be 
unwilling to revert the change that did cause the breakage, if things don't get 
fixed quickly.


To know about breakage quicker, a nighly run of all tests against "HEAD of 
everything" could help. Our dependency updates give us some of that, and we 
usually respond 

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Mitch Curtis]

> -Original Message-
> From: Volker Hilsheimer 
> Sent: Friday, 4 June 2021 4:51 PM
> To: Mitch Curtis 
> Cc: development@qt-project.org
> Subject: Re: [Development] Solutions for ensuring that changes in upstream
> modules are tested with downstream modules before merging
> 
> > On 4 Jun 2021, at 16:10, Mitch Curtis  wrote:
> >
> >> -Original Message-
> >> From: Volker Hilsheimer 
> >> Sent: Friday, 4 June 2021 2:45 PM
> >> To: Mitch Curtis 
> >> Cc: development@qt-project.org
> >> Subject: Re: [Development] Solutions for ensuring that changes in
> >> upstream modules are tested with downstream modules before merging
> >>
> >>> On 4 Jun 2021, at 13:56, Mitch Curtis  wrote:
> >>>
> >>> Hi.
> >>>
> >>> A common problem I see is that a change in say qtbase or
> >>> qtdeclarative can
> >> cause test failures in modules that depend on them after that change
> >> is merged. As a result, dependency updates for the downstream modules
> >> can be blocked, requiring the module maintainer to look into the
> >> failure, only to discover that it is caused by an upstream module.
> >> This causes a lot of time and effort to be diverted away from regular
> >> work. Both upstream and downstream developers would benefit from
> >> having more immediate CI feedback on these kinds of changes.
> >>>
> >>> https://bugreports.qt.io/browse/QTBUG-79454 was created to track
> >>> this
> >> problem. So far the ideas suggested have been:
> >>>
> >>> - Run tests of dependent modules when testing a change, in addition
> >>> to
> >> the regular tests for that module
> >>> - Merge repositories of more tightly coupled modules (e.g. move
> >> qtquickcontrols2 into qtdeclarative)
> >>>
> >>> It would be beneficial to discuss the advantages and disadvantages
> >>> of
> >> these and other solutions so that we can address this problem.
> >>>
> >>> Cheers.
> >>
> >>
> >> Excellent topic for the upcoming Qt Contributors Summit, Mitch, great
> >> if you can add it to the list of topics:
> >>
> >> https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program
> >>
> >> Which doesn’t mean that we shouldn't start the discussion on the
> >> list, and maybe we can then conclude it at the summit.
> >>
> >>
> >> Cheers,
> >> Volker
> >
> > OK, added it to the list.
> >
> 
> Thanks.
> 
> 
> As for the topic:
> 
> I do think that we have cases where too many modules are lumped into a
> single repo, and cases where we have too much granularity.
> 
> For the former: I see no immediate reason why QtNetwork needs to be in
> qtbase (network tests failing is a major cause of unrelated integrations
> failing), or why the Qt Positioning module needs to be in qtlocation.
> 
> For the latter: Qt Quick Controls 2 might be better off sharing a commit
> history with QtQuick
> 
> However, while we can make adjustments, I also think that there’s no
> perfect solution. It’s always a tradeoff between speed and autonomy on the
> one, and integrity across dependencies on the other hand.

I agree.

> Running more tests is a similar tradeoff, and I don’t think blocking an
> integration into e.g. qtbase because it breaks a leaf module is a good 
> solution
> to the problem. We should aim for fast feedback, and short meantime to
> recover such breakages, and making CI runs take even longer helps with
> neither. In particular if we then can’t make any such changes even though a
> follow up change in a leaf module is already available and just needs to wait
> for the dependency update. Then we are in a deadlock, unless we keep the
> code in qtbase working for both versions of e.g. qtdeclarative, which is not
> always practical.

I could be wrong, but perhaps this is what Ossi's 
https://bugreports.qt.io/browse/COIN-133 was targeting. In any case, I can't 
really speak for qtbase as I rarely work on it.

> The problem is perhaps not that we break things, but that we don’t notice
> for too long that we broke things? And once we do notice, we often seem to
> be unwilling to revert the change that did cause the breakage, if things don’t
> get fixed quickly.

This might be the case for qtbase and its leaf modules, but not for 
qtdeclarative and qtquickcontrols2. The typical situation there is: 
qtquickcontrols2 dependency update fails. I immediately cannot think of any 
relevant changes that would cause such a failure, so I quickly look through git 
log. If I can't find anything there, I need to start bisecting with 
qtdeclarative (and this assumes that the bisect doesn't result in build 
errors). It's not a productive use of time.

On the other hand, a CI run, while taking extra time if having to run the tests 
of two or more modules, shouldn't block someone from doing work. There is 
always another task to work on while waiting for CI. Flakiness will of course 
make the feedback loop take way longer, but that has always been the case.

> To know about breakage quicker, a nighly run of all tests against “HEAD of
> everything” could help. Our dependency updates give us some 

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Tuukka Turunen]

Hi,

Having the controls within declarative and also using the controls more in all 
the relevant examples would be beneficial for the users. So I think this might 
be one dependency we could add, not only for testing.

Yours,

Tuukka


Lähettäjä: Development  käyttäjän Volker 
Hilsheimer  puolesta
Lähetetty: perjantaina, kesäkuuta 4, 2021 5:55 ip.
Vastaanottaja: Mitch Curtis
Kopio: development@qt-project.org
Aihe: Re: [Development] Solutions for ensuring that changes in upstream modules 
are tested with downstream modules before merging

> On 4 Jun 2021, at 16:10, Mitch Curtis  wrote:
>
>> -Original Message-
>> From: Volker Hilsheimer 
>> Sent: Friday, 4 June 2021 2:45 PM
>> To: Mitch Curtis 
>> Cc: development@qt-project.org
>> Subject: Re: [Development] Solutions for ensuring that changes in upstream
>> modules are tested with downstream modules before merging
>>
>>> On 4 Jun 2021, at 13:56, Mitch Curtis  wrote:
>>>
>>> Hi.
>>>
>>> A common problem I see is that a change in say qtbase or qtdeclarative can
>> cause test failures in modules that depend on them after that change is
>> merged. As a result, dependency updates for the downstream modules can
>> be blocked, requiring the module maintainer to look into the failure, only to
>> discover that it is caused by an upstream module. This causes a lot of time
>> and effort to be diverted away from regular work. Both upstream and
>> downstream developers would benefit from having more immediate CI
>> feedback on these kinds of changes.
>>>
>>> https://bugreports.qt.io/browse/QTBUG-79454 was created to track this
>> problem. So far the ideas suggested have been:
>>>
>>> - Run tests of dependent modules when testing a change, in addition to
>> the regular tests for that module
>>> - Merge repositories of more tightly coupled modules (e.g. move
>> qtquickcontrols2 into qtdeclarative)
>>>
>>> It would be beneficial to discuss the advantages and disadvantages of
>> these and other solutions so that we can address this problem.
>>>
>>> Cheers.
>>
>>
>> Excellent topic for the upcoming Qt Contributors Summit, Mitch, great if you
>> can add it to the list of topics:
>>
>> https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program
>>
>> Which doesn’t mean that we shouldn't start the discussion on the list, and
>> maybe we can then conclude it at the summit.
>>
>>
>> Cheers,
>> Volker
>
> OK, added it to the list.
>

Thanks.


As for the topic:

I do think that we have cases where too many modules are lumped into a single 
repo, and cases where we have too much granularity.

For the former: I see no immediate reason why QtNetwork needs to be in qtbase 
(network tests failing is a major cause of unrelated integrations failing), or 
why the Qt Positioning module needs to be in qtlocation.

For the latter: Qt Quick Controls 2 might be better off sharing a commit 
history with QtQuick

However, while we can make adjustments, I also think that there’s no perfect 
solution. It’s always a tradeoff between speed and autonomy on the one, and 
integrity across dependencies on the other hand.


Running more tests is a similar tradeoff, and I don’t think blocking an 
integration into e.g. qtbase because it breaks a leaf module is a good solution 
to the problem. We should aim for fast feedback, and short meantime to recover 
such breakages, and making CI runs take even longer helps with neither. In 
particular if we then can’t make any such changes even though a follow up 
change in a leaf module is already available and just needs to wait for the 
dependency update. Then we are in a deadlock, unless we keep the code in qtbase 
working for both versions of e.g. qtdeclarative, which is not always practical.


The problem is perhaps not that we break things, but that we don’t notice for 
too long that we broke things? And once we do notice, we often seem to be 
unwilling to revert the change that did cause the breakage, if things don’t get 
fixed quickly.


To know about breakage quicker, a nighly run of all tests against “HEAD of 
everything” could help. Our dependency updates give us some of that, and we 
usually respond swiftly when an update fails. It shouldn’t be on the leaf 
module maintainer alone to follow up on things, and I do have the impression 
that people do take ownership, at least once they see that their base module 
change broke stuff. I’m sure we can improve there as well.
But now that Qt 6 is out, I’d expect that a “everything at HEAD” toplevel build 
should work most of the time. My local worktrees are usually at HEAD of 
everything, and it’s rare that there are build problems at least.


Lastly, I do think that we could do more in encoding assumptions made across 
modules in unit tests. If a change in qtdeclarative breaks an assumption made 
in qqc2, then ideally there’s a unit test in qtdeclarative that would fail. We 
use private APIs a lot across some modules, but we don’t have unit tests for 
them in the base 

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Volker Hilsheimer]

> On 4 Jun 2021, at 16:10, Mitch Curtis  wrote:
> 
>> -Original Message-
>> From: Volker Hilsheimer 
>> Sent: Friday, 4 June 2021 2:45 PM
>> To: Mitch Curtis 
>> Cc: development@qt-project.org
>> Subject: Re: [Development] Solutions for ensuring that changes in upstream
>> modules are tested with downstream modules before merging
>> 
>>> On 4 Jun 2021, at 13:56, Mitch Curtis  wrote:
>>> 
>>> Hi.
>>> 
>>> A common problem I see is that a change in say qtbase or qtdeclarative can
>> cause test failures in modules that depend on them after that change is
>> merged. As a result, dependency updates for the downstream modules can
>> be blocked, requiring the module maintainer to look into the failure, only to
>> discover that it is caused by an upstream module. This causes a lot of time
>> and effort to be diverted away from regular work. Both upstream and
>> downstream developers would benefit from having more immediate CI
>> feedback on these kinds of changes.
>>> 
>>> https://bugreports.qt.io/browse/QTBUG-79454 was created to track this
>> problem. So far the ideas suggested have been:
>>> 
>>> - Run tests of dependent modules when testing a change, in addition to
>> the regular tests for that module
>>> - Merge repositories of more tightly coupled modules (e.g. move
>> qtquickcontrols2 into qtdeclarative)
>>> 
>>> It would be beneficial to discuss the advantages and disadvantages of
>> these and other solutions so that we can address this problem.
>>> 
>>> Cheers.
>> 
>> 
>> Excellent topic for the upcoming Qt Contributors Summit, Mitch, great if you
>> can add it to the list of topics:
>> 
>> https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program
>> 
>> Which doesn’t mean that we shouldn't start the discussion on the list, and
>> maybe we can then conclude it at the summit.
>> 
>> 
>> Cheers,
>> Volker
> 
> OK, added it to the list.
> 

Thanks.


As for the topic:

I do think that we have cases where too many modules are lumped into a single 
repo, and cases where we have too much granularity.

For the former: I see no immediate reason why QtNetwork needs to be in qtbase 
(network tests failing is a major cause of unrelated integrations failing), or 
why the Qt Positioning module needs to be in qtlocation.

For the latter: Qt Quick Controls 2 might be better off sharing a commit 
history with QtQuick

However, while we can make adjustments, I also think that there’s no perfect 
solution. It’s always a tradeoff between speed and autonomy on the one, and 
integrity across dependencies on the other hand.


Running more tests is a similar tradeoff, and I don’t think blocking an 
integration into e.g. qtbase because it breaks a leaf module is a good solution 
to the problem. We should aim for fast feedback, and short meantime to recover 
such breakages, and making CI runs take even longer helps with neither. In 
particular if we then can’t make any such changes even though a follow up 
change in a leaf module is already available and just needs to wait for the 
dependency update. Then we are in a deadlock, unless we keep the code in qtbase 
working for both versions of e.g. qtdeclarative, which is not always practical.


The problem is perhaps not that we break things, but that we don’t notice for 
too long that we broke things? And once we do notice, we often seem to be 
unwilling to revert the change that did cause the breakage, if things don’t get 
fixed quickly.


To know about breakage quicker, a nighly run of all tests against “HEAD of 
everything” could help. Our dependency updates give us some of that, and we 
usually respond swiftly when an update fails. It shouldn’t be on the leaf 
module maintainer alone to follow up on things, and I do have the impression 
that people do take ownership, at least once they see that their base module 
change broke stuff. I’m sure we can improve there as well.
But now that Qt 6 is out, I’d expect that a “everything at HEAD” toplevel build 
should work most of the time. My local worktrees are usually at HEAD of 
everything, and it’s rare that there are build problems at least.


Lastly, I do think that we could do more in encoding assumptions made across 
modules in unit tests. If a change in qtdeclarative breaks an assumption made 
in qqc2, then ideally there’s a unit test in qtdeclarative that would fail. We 
use private APIs a lot across some modules, but we don’t have unit tests for 
them in the base module.


Volker


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Mitch Curtis]

> -Original Message-
> From: Volker Hilsheimer 
> Sent: Friday, 4 June 2021 2:45 PM
> To: Mitch Curtis 
> Cc: development@qt-project.org
> Subject: Re: [Development] Solutions for ensuring that changes in upstream
> modules are tested with downstream modules before merging
> 
> > On 4 Jun 2021, at 13:56, Mitch Curtis  wrote:
> >
> > Hi.
> >
> > A common problem I see is that a change in say qtbase or qtdeclarative can
> cause test failures in modules that depend on them after that change is
> merged. As a result, dependency updates for the downstream modules can
> be blocked, requiring the module maintainer to look into the failure, only to
> discover that it is caused by an upstream module. This causes a lot of time
> and effort to be diverted away from regular work. Both upstream and
> downstream developers would benefit from having more immediate CI
> feedback on these kinds of changes.
> >
> > https://bugreports.qt.io/browse/QTBUG-79454 was created to track this
> problem. So far the ideas suggested have been:
> >
> > - Run tests of dependent modules when testing a change, in addition to
> the regular tests for that module
> > - Merge repositories of more tightly coupled modules (e.g. move
> qtquickcontrols2 into qtdeclarative)
> >
> > It would be beneficial to discuss the advantages and disadvantages of
> these and other solutions so that we can address this problem.
> >
> > Cheers.
> 
> 
> Excellent topic for the upcoming Qt Contributors Summit, Mitch, great if you
> can add it to the list of topics:
> 
> https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program
> 
> Which doesn’t mean that we shouldn't start the discussion on the list, and
> maybe we can then conclude it at the summit.
> 
> 
> Cheers,
> Volker

OK, added it to the list.

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Volker Hilsheimer]

> On 4 Jun 2021, at 13:56, Mitch Curtis  wrote:
> 
> Hi.
>  
> A common problem I see is that a change in say qtbase or qtdeclarative can 
> cause test failures in modules that depend on them after that change is 
> merged. As a result, dependency updates for the downstream modules can be 
> blocked, requiring the module maintainer to look into the failure, only to 
> discover that it is caused by an upstream module. This causes a lot of time 
> and effort to be diverted away from regular work. Both upstream and 
> downstream developers would benefit from having more immediate CI feedback on 
> these kinds of changes.
>  
> https://bugreports.qt.io/browse/QTBUG-79454 was created to track this 
> problem. So far the ideas suggested have been:
>  
> - Run tests of dependent modules when testing a change, in addition to the 
> regular tests for that module
> - Merge repositories of more tightly coupled modules (e.g. move 
> qtquickcontrols2 into qtdeclarative)
>  
> It would be beneficial to discuss the advantages and disadvantages of these 
> and other solutions so that we can address this problem.
>  
> Cheers.


Excellent topic for the upcoming Qt Contributors Summit, Mitch, great if you 
can add it to the list of topics:

https://wiki.qt.io/Qt_Contributors_Summit_2021_-_Program

Which doesn’t mean that we shouldn't start the discussion on the list, and 
maybe we can then conclude it at the summit.


Cheers,
Volker


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Solutions for ensuring that changes in upstream modules are tested with downstream modules before merging

[Mitch Curtis]

Hi.

A common problem I see is that a change in say qtbase or qtdeclarative can 
cause test failures in modules that depend on them after that change is merged. 
As a result, dependency updates for the downstream modules can be blocked, 
requiring the module maintainer to look into the failure, only to discover that 
it is caused by an upstream module. This causes a lot of time and effort to be 
diverted away from regular work. Both upstream and downstream developers would 
benefit from having more immediate CI feedback on these kinds of changes.

https://bugreports.qt.io/browse/QTBUG-79454 was created to track this problem. 
So far the ideas suggested have been:

- Run tests of dependent modules when testing a change, in addition to the 
regular tests for that module
- Merge repositories of more tightly coupled modules (e.g. move 
qtquickcontrols2 into qtdeclarative)

It would be beneficial to discuss the advantages and disadvantages of these and 
other solutions so that we can address this problem.

Cheers.


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development