Re: [Development] Qt6: Adding UTF-8 storage support to QString

[Thiago Macieira]

On Thursday, 24 January 2019 05:06:58 PST Konstantin Tokarev wrote:
> I will be officially pissed off if possibility to access raw data of QString
> without extra copy is gone  It would be better if there is a way to figure
> out internal storage encoding (e.g. isUtf16()) and access raw data

How often do you need that, oustide of QString itself? And maybe a few 
efficient QtCore classes? (QCborValue comes to mind)

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Qt6: Adding UTF-8 storage support to QString

[Thiago Macieira]

On Wednesday, 23 January 2019 23:32:28 PST Olivier Goffart wrote:
>   - Introduce some iterator that iterates over unicode code points.

I wrote that about a decade ago. It's called QStringIterator and it's inside 
our sources, but in a private header.

But we may want to make it iterate over grapheme clusters instead of Unicode 
codepoints. That is, make it use QTextBoundaryFinder to iterate, instead of 
decode the storage to UTF-32.

>   - Deprecate utf16()  and other API that assume that QString is UTF-16
>   - Replace them by a toUtf16 which returns a QVector.  I believe
> that it is possible to make the cotent implicitly shared with the QString,
> avoiding copies. (since it is just a QTypedArrayData internally)

QVector.

Sharing QVector and QString is possible, but we need to fix a few 
discrepancies, especially that of QVector not being allowed to be raw data, 
while QString can be (QVector::fromRawData was proposed for Qt 5.0 [Andreas 
Hartmetz, if I'm not mistaken] but we never added it). So this is fixable for 
Qt 6, but not before Qt 6.

I think I tried even in my branch and ran into a lot of trouble. It was a non-
obvious change. So I abandoned it.

Still, we're not going to switch away from UTF-16 in Qt 6. The best we can do 
is pave the way for switching in Qt 7, if we add the methods you're talking 
about, change ALL the Windows, Cocoa and Android code that calls .data() and 
assumes it to be UTF-16 to toUtf16(). We may want to have some #defines like 
the QStringView stirng level or the ASCII-cast ones, so we catch those.

But we WILL NOT change from UTF-16 in the next 2 years. 

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Ville Voutilainen]

On Thu, 24 Jan 2019 at 20:26, Simon Hausmann  wrote:
>
>
> I would see the biggest long term impact with the massive amount of cherry 
> picks from dev to qt6 over a long period of time.
>
> Git rerere works locally, so it doesn’t help in this setup I think.

Completely seriously, without any preference to either branching
approach: aren't we going to be in some sort of trouble
with the qt6 branch anyway, no matter what we do? Following on a bit:

Pardon me if I missed some important part of the motivation of all of
this, but I *CAN'T* see how this should, could,
or needs to be an approach that helps with the qt6 branch
merge/cherry-pick/rebase. I don't think that's going to
be a pleasant operation whatever we do.

I would like a "push to trunk, backport to release-branches" approach
going forward. As in, once we have the usual
umpteen X.y branches, it's a simple approach.

But I don't see how going from merge-forward (except also
merge-backward sometimes) to cherry-pick-backward
(except also cherry-pick forward, or kinda sideways to qt6, and maybe
sometimes merge in some direction) is going
to help us with qt6. These matters seem orthogonal.

Qt6 and dev are going to diverge. Drastically, eventually. I don't
know how to solve that. A new branching policy
is not going to help with that.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Sergio Ahumada]

On 24.01.19 14:10, Edward Welbourne wrote:


Automated cherry-picking implies various complications that we haven't
fully explored; whereas merges have some well-established reliable
properties that avoid many of those complications.  Engineers prefer
what's known, from experience, to work over things that sound like they
might.  OTOH, as Ville points out, other projects do use cherry-picking;
so perhaps we need more information from such projects,



I have seen this implemented, similar to what Jedrek proposed, but the 
decision where to cherry pick is in JIRA, not via git annotations/marks.


This requires the Release Manager to keep an eye on which changes have 
not being propagated to all branches, though.


fully automated process, the fix is pushed to the oldest branch (5.12) 
and then propagated to newer versions (5.13, dev, etc). Change is 
automatically cherry-picked, updated the commit message, uploaded to 
gerrit, automatic Code-Review+2, automatic stage, automatically merged 
if CI passes.


propagation rate failure is around 20% of, mostly due to:
- merge conflicts (will happen, for sure)
-- genuine merge conflicts
-- change was already cherry-picked manually
-- dependencies missing
- ci failures
-- mostly due to infrastructure issues
-- ci does not know how to handle dependencies

advantages:
- when it works, everybody is happy
- pass rate is very high (around 80%)
- one can run git-cherry-pick -n and add additional info in the commit 
message
- can be tuned up to exclude some changes from being propagated (the 
bot, not git-cherry-pick itself)


disadvantages:
- most developers don't know how to handle the propagation failures
-- I already moved on and am working on something else, do you want me 
to switch to '5.19' to fix a change I did in '5.12' ? oh, and it failed 
in '5.18' as well? why don't you do merges instead and leave alone? 
[true story :-)]

- change can pass the CI and found to be problematic later on
-- oh, I need to fix my change in 5 branches now? [Gerrit 2.14 has a 
cherry-pick button in the UI now, so this is not a problem with that 
version]

- automated cherry-pick process/bot/script has to be maintained
-- one change not merged might lead to lots of changes to fail with 
merge conflicts, staying open in gerrit forever
-- hey, can you add intelligence to the bot so it know how to merge the 
open changes once the dependency is merged?
-- hey, can you add intelligence to the bot so it can cherry-pick them 
in the right order?)
- yes, git-cherry-pick does weird things. I have seen it cherry-picking 
the same change twice without problems (same line twice in the file, I 
think I was an include guard that compiled but behaved weirdly, or similar)


my 2€
--
Sergio Ahumada
sahum...@texla.cl


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Simon Hausmann]

I would see the biggest long term impact with the massive amount of cherry 
picks from dev to qt6 over a long period of time.

Git rerere works locally, so it doesn’t help in this setup I think.

Simon

> On 24. Jan 2019, at 15:28, Jedrzej Nowacki  wrote:
> 
> Dnia środa, 23 stycznia 2019 23:05:17 CET Allan Sandfeld Jensen pisze:
>> More than that. Once you have had cherry-pick only for a while git will be
>> unable to find useful common ancestors for the changes, and will be unable
>> to do smart three-way merging of cherry-picks, increasing the number of
>> conflicts that needs to be resolved manually while decreasing the useful
>> information git can give you (no more useful three-way diff).
>> 
>> 'Allan
> 
> Good point, that is a risk. Question is how big it is. Does it really affect 
> us, as old branches over time should be modified less and less. Are there 
> tooling like git rerere able to help us?
> 
> Cheers,
>  Jędrek
> 
> 
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia czwartek, 24 stycznia 2019 13:33:28 CET Allan Sandfeld Jensen pisze:
> On Donnerstag, 24. Januar 2019 13:27:41 CET Jedrzej Nowacki wrote:
> > Dnia środa, 23 stycznia 2019 22:04:16 CET Allan Sandfeld Jensen pisze:
> > > On Mittwoch, 23. Januar 2019 16:51:10 CET Jedrzej Nowacki wrote:
> > > >   Proposal in short: let's use cherry-pick mode everywhere.
> > > >   
> > > >   All(**) changes would go to dev. From which they would be
> > > >   automatically
> > > > 
> > > > cherry-picked by a bot to other branches. The decision to which branch
> > > > cherry-
> > >  
> > >  pick, would be taken based on a tag in the commit message. We
> > >  
> > > > could add a footer that marks the change risk level as in quip-5
> > > > (http://quips-qt-io.herokuapp.com/quip-0005.html), so for example
> > > > "dev",
> > > > "stable", "LTS". By default everything would be cherry-picked to qt6
> > > > branch
> > > > unless "no-future" tag would be given. Of course we can bike-shed
> > > > about
> > > > the
> > > > tag names.
> > > 
> > > I don't see any advantage to this what so ever. The same amount of work
> > > and
> 
>  refactoring needs to be done, all you have done is made development
> 
> > > more prone to human error, and fixes less likely to reach their intended
> > > target, and made getting point releases out on time harder as they need
> > > to go through more steps before they have all their patches in.
> > > 
> > > 'Allan
> > 
> > It is hard to answer this because you have forgotten to write why you
> > think
> 
>  that development would be more error prone or fixes less likely to reach
> 
> > the destination or more steps would be involved in releasing. So I will
> > try
> > shooting in the dark :-)
> > 
> > The current merging strategy involves couple of people, one merge master
> > and a
> 
>  random reviewer that looks at conflicts if there are any. Usually it is
> 
> > not even close to quality level we have for single patches in which we
> > have
> > more reviewers and gerrit shows the real diff. In practice we assume that
> > no conflicts == good merge.
> > 
> > Fixes are less likely to hit target branches, true, but only if nobody
> > cares.
> 
>  Mark that a fix can magically disappear during broken mere, nobody
> 
> > will see it. The proposed solution allows at least track such cases.
> > 
> > Releases branches already use cherry-pick mode so from that perspective
> > there
> 
>  is no big changes, the only one is the origin branch from which the
> 
> > cherry- pick should happen.
> 
> Yeah, this was no the best of comments on the issue. But I believe I wrote
> my objections elsewhere:
> - Cherry-picks are more error prone in git than merges and requires manual
> intervention more frequently

Yes, they are in some ways, I think I managed to not hide them in the 
proposal. I believe that the load distribution, the fact that more people can 
handle issues, reduction in bus factor, ability to solve problems in parallel 
with a significantly smaller context/scope and transparency of the process 
overweight the risks.

In addition we may write tooling, that helps us to detect problems, like for 
example gerrit change hanging in a limbo for too long, because all of the data 
would be there. 

> - Even if we accept the always and instantly cherry-pick model, there is no
> reason to push to dev, continuing to push to the same branches as now could
> achieve all the same things with less room for error (because where it is
> supposed to be merged to is both implicit and unambigious)

I mentioned that before. Cherry-picking from stable to dev means that in case 
of an error we have a regression. In my opinion it is worse, then accidentally 
not having a fix for something that was broken forever in an older branch.

> - By having all commits go to the same branch you make that branch
> overloaded, and will break our CI even more than it already is.

I will answer that in other part of thread, as I believe it deserve a subtopic 
on it's own.

> - I don't see making it harder to fix bugs in the stable branch as a
> benefit.

Why harder? Conflicts rate is more or less the same, they will be just resolved 
by a different person. So I wouldn't call it harder just the maintenance cost 
would be more visible / moved to the author.

Cheers, 
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia czwartek, 24 stycznia 2019 14:23:21 CET Kari Oikarinen pisze:
> We're also using cherry-picking ourselves in LTS branches. At least older
> ones,
 5.12 is not at that mode yet. As far as I know it has been working
> well. 
> Since the problems with cherry-picking come up as the changes pile up and
> concurrent development happens, those are largely avoided in the older
> branches.
 
> A less ambitious change to our current process could be moving branches
> into
 cherry-picking mode earlier. That would limit the count of currently
> active branches that we need to merge between. Of course the effects of
> such a change would be much smaller also. But just to mention it because I
> don't think anyone has mentioned it yet.
> 
> -- 

Good point, we could move branches to cherry-pick mode earlier. So for example 
if we have qt6, dev and stable (whatever the number is) we would have "only" 
two merges from stable to qt6. Or we could keep merges from dev to qt6 for a 
little longer. Personally I would go all for cherry-pick mode instead to 
simplify the model, but I agree that it is still better then what we have now.

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

On Wednesday, 23 January 2019 11:01:53 PST Volker Hilsheimer wrote:
 I think that’s fine. What’s much worse is having a fix in 5.12 and
 not knowing how to deal with the merge conflicts into dev. That
 means dev might regress, unless whoever authored the change is
 willing to spend time on making it work. In the end, if
 contributors can’t own their changes for all various branches of
 Qt, then I much prefer for them to own the changes at least for
 dev. And with Qt 6, this will become a much bigger problem.

Thiago Macieira (23 January 2019 20:10)
>>> The problem is I can turn this around and say that we introduce
>>> regressions into the older branches due to an improper cherry-pick
>>> that didn't conflict.

Dnia środa, 23 stycznia 2019 21:47:16 CET Edward Welbourne pisze:
>> and *that* is a concern that does bother me.
>> Of course, it's got to pass integration, as well as not conflict, but
>> that doesn't guarantee it hasn't broken something.

Jedrzej Nowacki (24 January 2019 15:11)
> Of course, but it is not a regression to the current system, we have
> the problem currently, right?

Only for LTS branches, to which relatively few changes get picked.

> We can introduce a regression in a release branch

Just to be clear: while some cherry-picks do happen to release branches,
that is not their normal model.  Changes for an imminent release should
be sent to its release branch in the first instance.  The only thing
that's different there is that *staging* gets to be reserved to the
release team.

The only branches to which we currently cherry-pick by default are the
LTS ones.  Here the problem Thiago points out is *a bit* mitigated by
attentive review and the fact that it's a past branch we know well and
reviewers likely have some clue about what changes have happened since
that might not mix well with the change being cherry-picked; but this
only mitigates the problem a bit (we forget details) and I have my
doubts about how well that would work if we had more past branches to
remember that much about (I routinely get confused about which branch
first got a given change, even of mine, much less by anyone else), so
I'm not sure the mitigation would scale to twice as many destination
branches for cherry-picking.

> by merging and cherry-picking without conflicts. On the other hand
> logic that doesn't apply cleanly has a higher chances of introducing
> bugs and therefore higher chance of causing release problems.

well, breakage on the next LTS branch, that'll show up in its next
release, since (non-LTS) release branches don't get cherry-picks (much).

> These changes are more visible in gerrit as opposite to somehow opaque
> merges. So in my opinion it is an improvement.

That doesn't help the folk doing the LTS release, if the cherry-pick
went in without conflict and without any integration test failing.
The first they know about it is RTA or, worse, later.
So the change is more visible, yes, but how does that help anyone ?

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 23:05:17 CET Allan Sandfeld Jensen pisze:
> More than that. Once you have had cherry-pick only for a while git will be
> unable to find useful common ancestors for the changes, and will be unable
> to do smart three-way merging of cherry-picks, increasing the number of
> conflicts that needs to be resolved manually while decreasing the useful
> information git can give you (no more useful three-way diff).
> 
> 'Allan

Good point, that is a risk. Question is how big it is. Does it really affect 
us, as old branches over time should be modified less and less. Are there 
tooling like git rerere able to help us?

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model (part: new contributors)

[Ville Voutilainen]

On Thu, 24 Jan 2019 at 16:14, Konstantin Tokarev  wrote:
>
>
>
> 24.01.2019, 17:09, "Ville Voutilainen" :
> > With the proposed model I push into trunk like in every other project.
>
> Why every other? Quite a few projects have different policies, e.g. in
> some of them master is always in more-or-less ready for release state
> and new changes go to staging branch like "dev" or "next"

Well, sure, my comment is anecdotal. Then there are frozen-trunk
project, likes GCC, where I need
to sit on my adventurous patches while trunk is about to branch. In
such projects, I still push into
trunk first, always. In Qt, I need to spend time thinking where to
push, need to look at commit policy,
and verify my understanding by asking for help. Which is also
anecdotal, maybe it's just me.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 24.1.2019 16.15, Edward Welbourne wrote:
> Kari Oikarinen (24 January 2019 15:02)
>> The rest of the paragraph talks about a situation where we will have two 
>> stable
>> branches alive at the same time. Typically we don't, because once 5.x+1 is
>> created, 5.x is closed.
> 
> Not quite: once 5.x+1 is *released*, 5.x (if not LTS) is closed
> (unless we have a pressing reason to release another 5.x.y).
> So, in the interval between 5.x+1 branching and releasing,
> we have three branches.

Right, so typically we indeed have two stable branches open. Thanks for 
correcting me.

> 
>> But 5.12 is an LTS version, so it will still stay open.
> 
> Indeed.
> 
>> But at what point (under current process) would be switch it to cherry-pick 
>> only
>> mode? I don't remember when it happened for 5.9. It could be when 5.13 is
>> created and then there would be no release blocked by waiting for a merge.
> 
> We switch to cherry-picking into 5.12 when 5.14 is created.
> See QUIP-5,
> * https://quips-qt-io.herokuapp.com/quip-0005.html
> 
> So creation of 5.14 switches our merge pattern from 5.12->5.13->dev
> to 5.13->5.14->dev and the 5.14 release (probably) closes 5.13.
> Again, we could of course change that.

Yes, but I was attempting to describe the current approach and messed it up.

-- 
Kari
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Konstantin Tokarev]

24.01.2019, 17:13, "Jedrzej Nowacki" :
> Dnia środa, 23 stycznia 2019 21:47:16 CET Edward Welbourne pisze:
>>  On Wednesday, 23 January 2019 11:01:53 PST Volker Hilsheimer wrote:
>>  >> I think that’s fine. What’s much worse is having a fix in 5.12 and not
>>  >> knowing how to deal with the merge conflicts into dev. That means dev
>>  >> might
>>  >> regress, unless whoever authored the change is willing to spend time on
>>  >> making it work. In the end, if contributors can’t own their changes for
>>  >> all
>>  >> various branches of Qt, then I much prefer for them to own the changes at
>>  >> least for dev. And with Qt 6, this will become a much bigger problem.
>>
>>  Thiago Macieira (23 January 2019 20:10)
>>
>>  > The problem is I can turn this around and say that we introduce
>>  > regressions
>>  > into the older branches due to an improper cherry-pick that didn't
>>  > conflict.
>>  and *that* is a concern that does bother me.
>>  Of course, it's got to pass integration, as well as not conflict,
>>  but that doesn't guarantee it hasn't broken something.
>
> Of course, but it is not a regression to the current system, we have the
> problem currently, right? We can introduce a regression in a release branch by
> merging and cherry-picking without conflicts. On the other hand logic that
> doesn't apply cleanly has a higher chances of introducing bugs and therefore
> higher chance of causing release problems. These changes are more visible in
> gerrit as opposite to somehow opaque merges. So in my opinion it is an
> improvement.

However, regression in stable branch costs more, as in worst case it can result 
in
brown paper bug being released

>
> Cheers,
>   Jędrek
>
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development

-- 
Regards,
Konstantin

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

Kari Oikarinen (24 January 2019 15:02)
> The rest of the paragraph talks about a situation where we will have two 
> stable
> branches alive at the same time. Typically we don't, because once 5.x+1 is
> created, 5.x is closed.

Not quite: once 5.x+1 is *released*, 5.x (if not LTS) is closed
(unless we have a pressing reason to release another 5.x.y).
So, in the interval between 5.x+1 branching and releasing,
we have three branches.

> But 5.12 is an LTS version, so it will still stay open.

Indeed.

> But at what point (under current process) would be switch it to cherry-pick 
> only
> mode? I don't remember when it happened for 5.9. It could be when 5.13 is
> created and then there would be no release blocked by waiting for a merge.

We switch to cherry-picking into 5.12 when 5.14 is created.
See QUIP-5,
* https://quips-qt-io.herokuapp.com/quip-0005.html

So creation of 5.14 switches our merge pattern from 5.12->5.13->dev
to 5.13->5.14->dev and the 5.14 release (probably) closes 5.13.
Again, we could of course change that.

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Ville Voutilainen]

On Thu, 24 Jan 2019 at 15:39, Shawn Rutledge  wrote:
>
>
>
> > On 24 Jan 2019, at 14:15, Jedrzej Nowacki  wrote:
> >
> > Typical use case I have with bug fixing in dev is when I develop a feature 
> > and
> > I see let's say a memory leak in the code that I'm modifying or somewhere
> > around. With the merge model I have four options now:
> > 1. Pretend that I haven't seen it
> > 2. Make a fix in LTS and develop my feature separately, counting on the 
> > merge
> > master to resolve conflicts
> > 3. Do the fix only in dev
> > 4. Make a fix in LTS and wait for all merges before I continue with the 
> > feature
> > If it is a memory leak then it is not as bad as memory corruption and 
> > crashes,
> > as at that point I can not use approach 2. Option 1 and 3 are not great from
> > QA perspective. Option 4 is slow and kills motivation for development.
>
> 5. Ask Liang to do a merge ASAP, as soon as the LTS fix is integrated
>
> 6. Do the merge yourself on dev, push it, add reviewers, and in the meantime, 
> develop your feature on top (but so far we have a cultural limitation on who 
> is allowed to do this)
>
> 7. Cherry-pick the fix patch on dev so that you can keep working.  After that:
> 7a. Periodically run git pull —rebase locally until the rebase pulls in the 
> merged fix, and your new changes are now on the tip; by then, nobody can see 
> that you did the cherry-pick; or,
> 7b. Push both patches, with the cherry-picked fix marked WIP and with a note 
> that it’s a temporary cherry-pick, so that nobody will try to review or 
> integrate it (or write redundant comments reminding you that you shouldn’t do 
> that).  This way you can get your new work reviewed: you just can’t stage it 
> until the merge is done (and then you can abandon the cherry-picked fix).

Right. As opposed to

in-backporting-cherry-pick-mode) fix the bug in dev, backport to older
branches, life on all branches goes on with the fix applied

Having seven options is not necessarily better.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model (part: new contributors)

[Konstantin Tokarev]

24.01.2019, 17:09, "Ville Voutilainen" :
> With the proposed model I push into trunk like in every other project.

Why every other? Quite a few projects have different policies, e.g. in
some of them master is always in more-or-less ready for release state
and new changes go to staging branch like "dev" or "next"

-- 
Regards,
Konstantin

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model (part: new contributors)

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 18:09:31 CET Alex Blasche pisze:
> > - simpler for new contributors, always push to dev
> 
> Really? Me being the new guy wanting to fix a bug in 5.12 need to magically
> know that I have to push to dev and know about a magic cherry-pick logic
> and a magic tag in the commit log. Right now I need to know I want to fix
> in 512 and push to it. Also the current model does not bother the new guy
> with myriads of potentially following cherry-picks which may require a
> larger commitment than he is willing to give. 

Really, at least that is my latest experience, when I contributed a fix to 
python standard library. I was trying to understand the branching model just 
to read that I should not bother and push stuff to master, there it was the 
reviewer responsibility to cherry-pick the change to other branches based on 
the content. I have to admit that I was impressed how easy is to make the first 
time contribution to the project. In our case, gerrit allows you to change 
commit message online in the browser, that makes reviewing / re-targeting  
task easier and less prone to errors (case of accidental re-push to a wrong 
branch after bot moving it). You are assuming that the contributor knows that 
the fix has to be and can be in 5.12. Well, that may be the case, some will 
have no clue and be fine with any future release. 

If a new contributor pushes to a wrong branch sanity bot would warn and allow 
you to move it to dev.

> The entire bot logic section below is another non-implicit logic.

What do you mean? As you do not target 5.X.Z, I would call it quite implicit 
:-).

Cheers,
  Jedrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Ville Voutilainen]

On Thu, 24 Jan 2019 at 15:47, Edward Welbourne  wrote:
>
> Allan Sandfeld Jensen (24 January 2019 14:35)
> > We can't integrate multiple changes to the same branch in parellel.
>
> I was under the impression that was exactly what we do.
>
> When I press Stage, my change gets cherry-picked to a little branch that
> gerrit is building, along with (i.e. in parallel with) various other
> changes staged around the same time.  When Coin next has its hands free,
> it starts a new integration of that staging branch (and a fresh staging
> branch is started).  If the integration fails, the message at the end of
> the comment on my review tells me the list of other commits that were in
> the same integration branch.
>
> Or have I misunderstood what you're saying ?

That's not "in parallel". :)
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 24.1.2019 15.15, Jedrzej Nowacki wrote:
> Last, but not least. The release branches are already in cherry-pick mode, so
> again it is not different, amount of conflicts is the same, true one needs to
> solve them earlier,  but on the other hand you can prepare the conflict
> resolution in parallel, while waiting for the integration. In your example you
> picked an easy target, but try the same with 5.13 and case of a fix targeting
> LTS. With the current model you need one merge (5.12 -> 5.13) and then one
> cherry-pick (5.13 -> 5.13.x). Of course you could cherry-pick from LTS
> directly to the release branch, but then you are acknowledge that merges are
> hurting us in terms of release speed.

Our release branches aren't in cherry-pick mode. If you have a blocker fix for
5.12.1 right now, it should be submitted into 5.12.1. The changes will reach
5.12 by a merge that will be done later. Based on a quick look qtbase 5.12.1 had
only one cherry-pick since the downmerge, so cherry-picking into release
branches is an exception, not the rule.

The rest of the paragraph talks about a situation where we will have two stable
branches alive at the same time. Typically we don't, because once 5.x+1 is
created, 5.x is closed. But 5.12 is an LTS version, so it will still stay open.
But at what point (under current process) would be switch it to cherry-pick only
mode? I don't remember when it happened for 5.9. It could be when 5.13 is
created and then there would be no release blocked by waiting for a merge.

-- 
Kari
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 21:47:16 CET Edward Welbourne pisze:
> On Wednesday, 23 January 2019 11:01:53 PST Volker Hilsheimer wrote:
> >> I think that’s fine. What’s much worse is having a fix in 5.12 and not
> >> knowing how to deal with the merge conflicts into dev. That means dev
> >> might
> >> regress, unless whoever authored the change is willing to spend time on
> >> making it work. In the end, if contributors can’t own their changes for
> >> all
> >> various branches of Qt, then I much prefer for them to own the changes at
> >> least for dev. And with Qt 6, this will become a much bigger problem.
> 
> Thiago Macieira (23 January 2019 20:10)
> 
> > The problem is I can turn this around and say that we introduce
> > regressions
> > into the older branches due to an improper cherry-pick that didn't
> > conflict.
> and *that* is a concern that does bother me.
> Of course, it's got to pass integration, as well as not conflict,
> but that doesn't guarantee it hasn't broken something.

Of course, but it is not a regression to the current system, we have the 
problem currently, right? We can introduce a regression in a release branch by 
merging and cherry-picking without conflicts. On the other hand logic that 
doesn't apply cleanly has a higher chances of introducing bugs and therefore 
higher chance of causing release problems. These changes are more visible in 
gerrit as opposite to somehow opaque merges. So in my opinion it is an 
improvement.

Cheers,
  Jędrek




___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model (part: new contributors)

[Ville Voutilainen]

On Thu, 24 Jan 2019 at 15:56, Jedrzej Nowacki  wrote:
>
> Dnia środa, 23 stycznia 2019 18:09:31 CET Alex Blasche pisze:
> > > - simpler for new contributors, always push to dev
> >
> > Really? Me being the new guy wanting to fix a bug in 5.12 need to magically
> > know that I have to push to dev and know about a magic cherry-pick logic
> > and a magic tag in the commit log. Right now I need to know I want to fix
> > in 512 and push to it. Also the current model does not bother the new guy
> > with myriads of potentially following cherry-picks which may require a
> > larger commitment than he is willing to give.
>
> Really, at least that is my latest experience, when I contributed a fix to
> python standard library. I was trying to understand the branching model just
> to read that I should not bother and push stuff to master, there it was the
> reviewer responsibility to cherry-pick the change to other branches based on
> the content. I have to admit that I was impressed how easy is to make the 
> first
> time contribution to the project. In our case, gerrit allows you to change
> commit message online in the browser, that makes reviewing / re-targeting
> task easier and less prone to errors (case of accidental re-push to a wrong
> branch after bot moving it). You are assuming that the contributor knows that
> the fix has to be and can be in 5.12. Well, that may be the case, some will
> have no clue and be fine with any future release.

Well, me being a new guy wanting to fix a bug in 5.14 need to
magically know that I need
to push to 5.12 and then wait for a forward merge. With the proposed
model I push into
trunk like in every other project. As far as following cherry-picks
go, they don't necessarily
need to bother me, they could be done by other people, some called maintainers.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

Allan Sandfeld Jensen (24 January 2019 14:35)
> We can't integrate multiple changes to the same branch in parellel.

I was under the impression that was exactly what we do.

When I press Stage, my change gets cherry-picked to a little branch that
gerrit is building, along with (i.e. in parallel with) various other
changes staged around the same time.  When Coin next has its hands free,
it starts a new integration of that staging branch (and a fresh staging
branch is started).  If the integration fails, the message at the end of
the comment on my review tells me the list of other commits that were in
the same integration branch.

Or have I misunderstood what you're saying ?

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Shawn Rutledge]

> On 24 Jan 2019, at 14:15, Jedrzej Nowacki  wrote:
> 
> Typical use case I have with bug fixing in dev is when I develop a feature 
> and 
> I see let's say a memory leak in the code that I'm modifying or somewhere 
> around. With the merge model I have four options now:
> 1. Pretend that I haven't seen it
> 2. Make a fix in LTS and develop my feature separately, counting on the merge 
> master to resolve conflicts
> 3. Do the fix only in dev
> 4. Make a fix in LTS and wait for all merges before I continue with the 
> feature
> If it is a memory leak then it is not as bad as memory corruption and 
> crashes, 
> as at that point I can not use approach 2. Option 1 and 3 are not great from 
> QA perspective. Option 4 is slow and kills motivation for development.

5. Ask Liang to do a merge ASAP, as soon as the LTS fix is integrated

6. Do the merge yourself on dev, push it, add reviewers, and in the meantime, 
develop your feature on top (but so far we have a cultural limitation on who is 
allowed to do this)

7. Cherry-pick the fix patch on dev so that you can keep working.  After that:
7a. Periodically run git pull —rebase locally until the rebase pulls in the 
merged fix, and your new changes are now on the tip; by then, nobody can see 
that you did the cherry-pick; or, 
7b. Push both patches, with the cherry-picked fix marked WIP and with a note 
that it’s a temporary cherry-pick, so that nobody will try to review or 
integrate it (or write redundant comments reminding you that you shouldn’t do 
that).  This way you can get your new work reviewed: you just can’t stage it 
until the merge is done (and then you can abandon the cherry-picked fix).

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Allan Sandfeld Jensen]

On Donnerstag, 24. Januar 2019 14:31:16 CET Jedrzej Nowacki wrote:
> Dnia czwartek, 24 stycznia 2019 10:24:57 CET Allan Sandfeld Jensen pisze:
> 
> > On Donnerstag, 24. Januar 2019 10:11:35 CET Volker Hilsheimer wrote:
> > 
> > > More people working and building against dev helps keep dev more stable
> > > (by
> > > virtue of discovering breakage sooner), and the proposal encourages
> > > more
> > > people to work on dev. That can be a good thing.
> > 
> > 
> > No,  it will overload the CI by having all integrations happen on the
> > same
> > branch, instead being able to integrate multiple branches at once.
> 
> 
> It will overload in what way? Yes, in total we would have more integrations.
> 
 If you look into the statistics (https://testresults.qt.io/grafana/d/
> 00024/telegraf-host-metrics) you can see that we have some free
> resources. In general we should not really cross load of 60, but most of
> the hosts are on significantly lower load. I would be more concerned about
> flakiness, as it means that we run more tests, but that is something that
> we need to fix no matter what.
> 
We can't integrate multiple changes to the same branch in parellel. So you 
can't start using more resources to speed things up. (9 women to have a child 
in 1 month) The only way to speed up CI integration is to be merging to 
different branches, that way we CAN have multiple integrations running in 
parallel.

'Allan


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia czwartek, 24 stycznia 2019 10:24:57 CET Allan Sandfeld Jensen pisze:
> On Donnerstag, 24. Januar 2019 10:11:35 CET Volker Hilsheimer wrote:
> > More people working and building against dev helps keep dev more stable
> > (by
> > virtue of discovering breakage sooner), and the proposal encourages more
> > people to work on dev. That can be a good thing.
> 
> No,  it will overload the CI by having all integrations happen on the same
> branch, instead being able to integrate multiple branches at once.

It will overload in what way? Yes, in total we would have more integrations. 
If you look into the statistics (https://testresults.qt.io/grafana/d/
00024/telegraf-host-metrics) you can see that we have some free resources. 
In general we should not really cross load of 60, but most of the hosts are on 
significantly lower load. I would be more concerned about flakiness, as it 
means 
that we run more tests, but that is something that we need to fix no matter 
what.

> > Either way, nothing in the proposal prevents us from developing and push a
> > fix for 5.12, and then develop a different fix for dev that we don’t
> > cherry
> > pick into 5.12 or other stable branches. This should be an exception, used
> > in urgent situations where indeed we can’t wait. For those cases, there’s
> > no merge conflict in the first place in the new model (because there’s no
> > merge).
> 
> It is still doing merging the wrong way for no good reason. Even if cherry-
> picking itself wasnt a bad idea. You could just as well commit to the stable
> branch first and then cherry-pick it to the other branches. That way there
> would be no need for cryptic annotations, the branches to pick to would be
> implicit from where the patch first landed. Or even better, you could
> perform a merge to dev on every single commit to the stable branch, or even
> better, only create a merge if there are conflicts, so we don't overload
> the CI on dev.
> 
> 'Allan

If you cherry-pick the opposite way, the risk of regression is; higher, 
because of one time contributions (I'm using 5.9 I do not care about other 
releases, so I will not solve the merge conflicts or baby sit the changes) or 
because of lack of interest (oh, that is simply re-write I will not do it) or 
because just a mistake (oh it is still in my gerrit dashboard). In my opinion 
regressions are much worse then, a missing fix in an older branch.

Cheers,
  Jędrek



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Shawn Rutledge]

> On 24 Jan 2019, at 14:10, Edward Welbourne  wrote:
> 
> Dnia czwartek, 24 stycznia 2019 09:08:29 CET Liang Qi pisze:
>>> My concern is about "cherry-pick" a series of changes for a big
>>> feature, especially during the period to have "dev" branches for both
>>> 5 and 6. I don't have solution for this issue yet.
> 
> Jedrzej Nowacki (24 January 2019 11:53)
>> My assumption is that bot would cherry-pick them in the same order in
>> which patches got integrated to dev. That way we could reduce amount
>> of issues. Of course if the first patch from a series causes conflicts
>> then all other would also be in conflict.
> 
> Well, the *initial* cherry-pick-and-send-to-Coin can be made to happen
> in the same order; but after that, any that didn't get through on the
> first pass are going to be apparently-unrelated changes waiting to go
> into the destination branch.  The script might be able to look at the
> upstream dev versions of its cherry-picks and reproduce the dependencies
> among them, but that's going to get tricky when some of those upstream
> changes got integrated out of order (because some of them were merely
> later in the branch, without a strict dependence on those earlier) or
> some of them succeed on the first pass while others, before and after
> them on the original branch, fail.  Either you'll end up losing some
> dependency relations or you risk having things on my branch depend on
> other folks' unrelated changes that were just upstream of my branch,
> that haven't yet made it through cherry-picking.  This isn't fatal, as
> the developer taking care of the post-failure fix-up and/or retries can
> stage despite a dependency missing, but it'll cause some pain at times.

Either way it can fail though: if the bot tries too hard to keep the patches in 
order, we’ll all waste time whenever a lot of patches “depend” on one that is 
failing to be cherry-picked to a release branch.  (So then isn’t it similar to 
merging change sets from feature branches, in that the whole branch is an 
all-or-nothing deal?  That’s what I dislike about how GitHub does it.)  But if 
we don’t keep them in order, it will be the same as now when trying to stage 
any given patch: individual patches will need rebasing sometimes, and that’s 
also extra hassle.

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 24.1.2019 15.10, Edward Welbourne wrote:
> Dnia czwartek, 24 stycznia 2019 09:08:29 CET Liang Qi pisze:
>>> My concern is about "cherry-pick" a series of changes for a big
>>> feature, especially during the period to have "dev" branches for both
>>> 5 and 6. I don't have solution for this issue yet.
> 
> Jedrzej Nowacki (24 January 2019 11:53)
>> My assumption is that bot would cherry-pick them in the same order in
>> which patches got integrated to dev. That way we could reduce amount
>> of issues. Of course if the first patch from a series causes conflicts
>> then all other would also be in conflict.
> 
> Well, the *initial* cherry-pick-and-send-to-Coin can be made to happen
> in the same order; but after that, any that didn't get through on the
> first pass are going to be apparently-unrelated changes waiting to go
> into the destination branch.  The script might be able to look at the
> upstream dev versions of its cherry-picks and reproduce the dependencies
> among them, but that's going to get tricky when some of those upstream
> changes got integrated out of order (because some of them were merely
> later in the branch, without a strict dependence on those earlier) or
> some of them succeed on the first pass while others, before and after
> them on the original branch, fail.  Either you'll end up losing some
> dependency relations or you risk having things on my branch depend on
> other folks' unrelated changes that were just upstream of my branch,
> that haven't yet made it through cherry-picking.  This isn't fatal, as
> the developer taking care of the post-failure fix-up and/or retries can
> stage despite a dependency missing, but it'll cause some pain at times.
> 
> There are also potential problems where I push to dev a fix for some
> issue, followed by a change that undoes a kludge-around whose root cause
> is the issue I've just fixed, but which was put there to fix some
> seemingly unrelated issue.  Even if the first fix's cherry-pick hits a
> conflict or fails to integrate, it's entirely possible that the removal
> of the kludge-around gets through on the first pass; and that could be a
> problem.  With luck there's a test for what it kludged round, that'll
> prevent it integrating; but, if there isn't, that issue resurfaces in
> the target branch until the real fix's cherry-pick lands.
> 
> Automated cherry-picking implies various complications that we haven't
> fully explored; whereas merges have some well-established reliable
> properties that avoid many of those complications.  Engineers prefer
> what's known, from experience, to work over things that sound like they
> might.  OTOH, as Ville points out, other projects do use cherry-picking;
> so perhaps we need more information from such projects,

We're also using cherry-picking ourselves in LTS branches. At least older ones,
5.12 is not at that mode yet. As far as I know it has been working well.

Since the problems with cherry-picking come up as the changes pile up and
concurrent development happens, those are largely avoided in the older branches.

A less ambitious change to our current process could be moving branches into
cherry-picking mode earlier. That would limit the count of currently active
branches that we need to merge between. Of course the effects of such a change
would be much smaller also. But just to mention it because I don't think anyone
has mentioned it yet.

-- 
Kari
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 22:09:30 CET Allan Sandfeld Jensen pisze:
> On Mittwoch, 23. Januar 2019 21:42:35 CET Edward Welbourne wrote:
> > Jedrzej Nowacki wrote:
> > >>  Advantages:
> > >>  - no waiting for merges, a fix can be used right away after
> > >>  
> > >>integration
> > >>  
> > >>  - faster propagation of fixes targeting all branches, as there are
> > >>  
> > >>no merges of merges
> > 
> > Alex Blasche (23 January 2019 18:09)
> > 
> > > This is pure speculation because you potentially triple (or worse) the
> > > amount of individual patches requiring a merge in gerrit when you
> > > consider that you want to at least merge to 5.9, 512, dev and qt6. I
> > > don't see this prediction come true.
> > 
> > Well, as soon as it hits dev, the patch is cherry-picked to every branch
> > that its footer says it belongs in.  Those branches where all goes well
> > see it one integration later.  Each branch that has a conflict needs
> > that resolved before we get to that second integrtion.  Contrast this
> > with a 5.12 -> 5.13 -> dev chain of merges, where dev doesn't get the
> > change that landed in 5.12 (even if that change could land in dev
> > without conflict) until
> > 
> >  * there's been some delay between their change being accepted in 5.12
> >  
> >and the next merge-bot run
> >  
> >  * everyone who made change to 5.12 that conflicted on merging to 5.13
> >  
> >has advised Liang on how to resolve their conflicts
> >  
> >  * we've got the result through integration into 5.13
> >  * everyone who's made changes to 5.13 or (as possibly just amended in
> >  
> >merging) 5.12 that conflicts with anything in dev has again advised
> >how to resolve their conflicts
> >  
> >  * and we've got the result through a second integration, into dev.
> > 
> > When nothing but the change being considered has a conflict along
> > the way, that's twice as long; and any change to an upstream branch,
> > that does have a conflict, introduces delay for all the other changes
> > that landed in that branch, even if they don't have conflicts.  In the
> > middle of summer, when lots of folk are away on holiday, getting help
> > with resolving conflicts isn't easy - the folk who know those commits
> > won't be back for a month - and all changes, no matter how urgent, get
> > stuck behind any conflict we can't work out how to resolve.
> > 
> > So no, Jedrzej's claim is not *pure* speculation; it's at least quite a
> > lot adulterated with reasons to believe that many changes would, under
> > his scheme, propagate to all branches they're destined for sooner than
> > happens with our present scheme.
> 
> No, it is speculation, and it optimizing the least important case: bug-fixes
> in dev. Dev is the branch that can wait the longest to get a bug-fix, the
> stable branch is the branch that need it the most urgent. And fixing a bug
> in 5.12 now means you first fix it where you need it (5.12), then rewrite
> it for dev, then resolve the inevitable conflict back so it can be merged,
> all waiting for bots and release teams to stumple into the issues and
> delaying the next 5.12.x release.
> 

First, I do not agree that dev is less important. Dev is the base for all 
future releases. Even if a current release is broken in some way, the next one 
should not be. We can not focus on short term only.

Typical use case I have with bug fixing in dev is when I develop a feature and 
I see let's say a memory leak in the code that I'm modifying or somewhere 
around. With the merge model I have four options now:
1. Pretend that I haven't seen it
2. Make a fix in LTS and develop my feature separately, counting on the merge 
master to resolve conflicts
3. Do the fix only in dev
4. Make a fix in LTS and wait for all merges before I continue with the feature
If it is a memory leak then it is not as bad as memory corruption and crashes, 
as at that point I can not use approach 2. Option 1 and 3 are not great from 
QA perspective. Option 4 is slow and kills motivation for development.

Last, but not least. The release branches are already in cherry-pick mode, so 
again it is not different, amount of conflicts is the same, true one needs to 
solve them earlier,  but on the other hand you can prepare the conflict 
resolution in parallel, while waiting for the integration. In your example you 
picked an easy target, but try the same with 5.13 and case of a fix targeting 
LTS. With the current model you need one merge (5.12 -> 5.13) and then one 
cherry-pick (5.13 -> 5.13.x). Of course you could cherry-pick from LTS 
directly to the release branch, but then you are acknowledge that merges are 
hurting us in terms of release speed. 

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

Dnia czwartek, 24 stycznia 2019 09:08:29 CET Liang Qi pisze:
>> My concern is about "cherry-pick" a series of changes for a big
>> feature, especially during the period to have "dev" branches for both
>> 5 and 6. I don't have solution for this issue yet.

Jedrzej Nowacki (24 January 2019 11:53)
> My assumption is that bot would cherry-pick them in the same order in
> which patches got integrated to dev. That way we could reduce amount
> of issues. Of course if the first patch from a series causes conflicts
> then all other would also be in conflict.

Well, the *initial* cherry-pick-and-send-to-Coin can be made to happen
in the same order; but after that, any that didn't get through on the
first pass are going to be apparently-unrelated changes waiting to go
into the destination branch.  The script might be able to look at the
upstream dev versions of its cherry-picks and reproduce the dependencies
among them, but that's going to get tricky when some of those upstream
changes got integrated out of order (because some of them were merely
later in the branch, without a strict dependence on those earlier) or
some of them succeed on the first pass while others, before and after
them on the original branch, fail.  Either you'll end up losing some
dependency relations or you risk having things on my branch depend on
other folks' unrelated changes that were just upstream of my branch,
that haven't yet made it through cherry-picking.  This isn't fatal, as
the developer taking care of the post-failure fix-up and/or retries can
stage despite a dependency missing, but it'll cause some pain at times.

There are also potential problems where I push to dev a fix for some
issue, followed by a change that undoes a kludge-around whose root cause
is the issue I've just fixed, but which was put there to fix some
seemingly unrelated issue.  Even if the first fix's cherry-pick hits a
conflict or fails to integrate, it's entirely possible that the removal
of the kludge-around gets through on the first pass; and that could be a
problem.  With luck there's a test for what it kludged round, that'll
prevent it integrating; but, if there isn't, that issue resurfaces in
the target branch until the real fix's cherry-pick lands.

Automated cherry-picking implies various complications that we haven't
fully explored; whereas merges have some well-established reliable
properties that avoid many of those complications.  Engineers prefer
what's known, from experience, to work over things that sound like they
might.  OTOH, as Ville points out, other projects do use cherry-picking;
so perhaps we need more information from such projects,

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Qt6: Adding UTF-8 storage support to QString

[Konstantin Tokarev]

24.01.2019, 10:34, "Olivier Goffart" :
> On 23.01.19 23:15, André Pönitz wrote:
>>  On Wed, Jan 23, 2019 at 05:40:33PM +0300, Konstantin Tokarev wrote:
>>>  23.01.2019, 16:55, "Edward Welbourne" :
  All of this discussion ignores a major elephant: QString's indexing is
  by 16-bit UTF-16 tokens, not by Unicode characters. We've had Unicode
  for a couple of decades now.

  We *should* have a string type (I don't care what you call it) that acts
  on strings indexed by Unicode characters, not in terms of a
  representation. Whether that string type internally uses UTF-16 or
  UTF-8 should be invisible to its user. Ideally it would be capable of
  carrying its data internally in either form (so as to avoid needless
  conversion when both producer and consumer use the same form) and of
  converting between the two (e.g. so as to append efficiently) as needed.
>>>
>>>  I think this is excessive. Most common operations with strings in 
>>> application
>>>  code are:
>>>
>>>  * Pass the string around or compare as an opaque token
>>>  * Draw the string on screen e.g. with QPainter (while technically it
>>> falls in the previous category, I think it's important enough to
>>> deserve separate item)
>>>  * Find substring or pattern (regex) inside the string
>>>  * Split the string by character, pattern, or index boundaries found by 
>>> means
>>> of previous item
>>>
>>>  I think the only common cases when dealing with Unicode grapheme clusters
>>>  is required are
>>>
>>>  * Handling of text cursor movement
>>>  * Implementation of text shaping, i.e. what Harfbuzz is doing
>>>
>>>  I think having special iterator would be quite enough for cursor case. Such
>>>  iterator could abstract away underlying encoding, instead of forcing 
>>> everyone
>>>  to convert to UTF-16 first.
>>
>>  All of that is scarily close to my opinion on the topic.
>
> Same here. I think Konstantin is spot on.
>
> Another example of good string design, I think, is the Rust's String. Their
> string is encoded in valid UTF-8, indexed by bytes, and splitting the string 
> in
> the middle of a code point is a programmer error.
>
> As already mentioned before, UTF-16 is quite a bad choice, if it weren't for
> legacy.
>
> The argument of that developper wrongly using indexes cause more problem with
> utf-8 than with utf-16 ("it would happen for a lot more characters") actually
> means that the developper will see and fix their bugs quickly.
>
> I understand changing QString to UTF-8 is a difficult task if we want to do it
> in a compatible way. However, I think there is a way:
> In Qt5.x:
>   - Introduce some iterator that iterates over unicode code points.
>   - Deprecate utf16() and other API that assume that QString is UTF-16
>   - Replace them by a toUtf16 which returns a QVector. I believe that
> it is possible to make the cotent implicitly shared with the QString, avoiding
> copies. (since it is just a QTypedArrayData internally)

I will be officially pissed off if possibility to access raw data of QString 
without
extra copy is gone :( It would be better if there is a way to figure out 
internal
storage encoding (e.g. isUtf16()) and access raw data

>
> Then in Qt6 one can simply change the representation without breaking
> compatibility with non-deprecated functions.
>
> --
> Olivier
>
> Woboq - Qt services and support - https://woboq.com - https://code.woboq.org
>
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development

-- 
Regards,
Konstantin

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Allan Sandfeld Jensen]

On Donnerstag, 24. Januar 2019 13:27:41 CET Jedrzej Nowacki wrote:
> Dnia środa, 23 stycznia 2019 22:04:16 CET Allan Sandfeld Jensen pisze:
> 
> > On Mittwoch, 23. Januar 2019 16:51:10 CET Jedrzej Nowacki wrote:
> > 
> > >   Proposal in short: let's use cherry-pick mode everywhere.
> > >   
> > >   All(**) changes would go to dev. From which they would be
> > >   automatically
> > > 
> > > 
> > > cherry-picked by a bot to other branches. The decision to which branch
> > > cherry-
> > 
> > 
> > 
> >  pick, would be taken based on a tag in the commit message. We
> > 
> > 
> > 
> > > could add a footer that marks the change risk level as in quip-5
> > > (http://quips-qt-io.herokuapp.com/quip-0005.html), so for example
> > > "dev",
> > > "stable", "LTS". By default everything would be cherry-picked to qt6
> > > branch
> > > unless "no-future" tag would be given. Of course we can bike-shed about
> > > the
> > > tag names.
> > 
> > 
> > I don't see any advantage to this what so ever. The same amount of work
> > and
 refactoring needs to be done, all you have done is made development
> > more prone to human error, and fixes less likely to reach their intended
> > target, and made getting point releases out on time harder as they need
> > to go through more steps before they have all their patches in.
> > 
> > 'Allan
> 
> 
> 
> It is hard to answer this because you have forgotten to write why you think
> 
 that development would be more error prone or fixes less likely to reach
> the destination or more steps would be involved in releasing. So I will try
> shooting in the dark :-)
> 
> The current merging strategy involves couple of people, one merge master and
> a 
 random reviewer that looks at conflicts if there are any. Usually it is
> not even close to quality level we have for single patches in which we have
> more reviewers and gerrit shows the real diff. In practice we assume that
> no conflicts == good merge.
> 
> Fixes are less likely to hit target branches, true, but only if nobody
> cares. 
 Mark that a fix can magically disappear during broken mere, nobody
> will see it. The proposed solution allows at least track such cases.
> 
> Releases branches already use cherry-pick mode so from that perspective
> there 
 is no big changes, the only one is the origin branch from which the
> cherry- pick should happen.
> 
Yeah, this was no the best of comments on the issue. But I believe I wrote my 
objections elsewhere:
- Cherry-picks are more error prone in git than merges and requires manual 
intervention more frequently
- Even if we accept the always and instantly cherry-pick model, there is no 
reason to push to dev, continuing to push to the same branches as now could 
achieve all the same things with less room for error (because where it is 
supposed to be merged to is both implicit and unambigious)
- By having all commits go to the same branch you make that branch overloaded, 
and will break our CI even more than it already is.
- I don't see making it harder to fix bugs in the stable branch as a benefit.

Best regards
'Allan




___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 22:04:16 CET Allan Sandfeld Jensen pisze:
> On Mittwoch, 23. Januar 2019 16:51:10 CET Jedrzej Nowacki wrote:
> >   Proposal in short: let's use cherry-pick mode everywhere.
> >   
> >   All(**) changes would go to dev. From which they would be automatically
> > 
> > cherry-picked by a bot to other branches. The decision to which branch
> > cherry-
> 
>  pick, would be taken based on a tag in the commit message. We
> 
> > could add a footer that marks the change risk level as in quip-5
> > (http://quips-qt-io.herokuapp.com/quip-0005.html), so for example "dev",
> > "stable", "LTS". By default everything would be cherry-picked to qt6
> > branch
> > unless "no-future" tag would be given. Of course we can bike-shed about
> > the
> > tag names.
> 
> I don't see any advantage to this what so ever. The same amount of work and
> refactoring needs to be done, all you have done is made development more
> prone to human error, and fixes less likely to reach their intended target,
> and made getting point releases out on time harder as they need to go
> through more steps before they have all their patches in.
> 
> 'Allan


It is hard to answer this because you have forgotten to write why you think 
that development would be more error prone or fixes less likely to reach the 
destination or more steps would be involved in releasing. So I will try 
shooting in the dark :-)

The current merging strategy involves couple of people, one merge master and a 
random reviewer that looks at conflicts if there are any. Usually it is not 
even close to quality level we have for single patches in which we have more 
reviewers and gerrit shows the real diff. In practice we assume that no 
conflicts == good merge.

Fixes are less likely to hit target branches, true, but only if nobody cares. 
Mark that a fix can magically disappear during broken mere, nobody will see it. 
The proposed solution allows at least track such cases.

Releases branches already use cherry-pick mode so from that perspective there 
is no big changes, the only one is the origin branch from which the cherry-
pick should happen.

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] HEADS-UP: Branching from '5.12' to '5.12.1' started

[ekke]

Am 23.01.19 um 17:03 schrieb Kai Koehne:
>> -Original Message-
>> From: Development  On Behalf Of ekke
>> Sent: Wednesday, January 23, 2019 9:45 AM
>> To: development@qt-project.org
>> Subject: Re: [Development] HEADS-UP: Branching from '5.12' to '5.12.1' 
>> started
>>
>> seems this time it's a rocky path to release Qt 5.12.1
> We're still struggling with a change in configure that made system library 
> paths absolute - good for local installations, but problematic for the binary 
> packages, which are supposed to work in different setups:
>
>  https://bugreports.qt.io/browse/QTBUG-72903
>
> Kai
>
I'm waiting for 5.12.1 because 5.12 is blocked on Android because of
https://bugreports.qt.io/browse/QTBUG-72101


wish you luck to figure out the correct configure to be able to release
5.12.1

thx for your hard work

ekke

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Qt6: Adding UTF-8 storage support to QString

[Konstantin Ritt]

>  - Introduce some iterator that iterates over unicode code points.

QStringIterator


> We *should* have a string type (I don't care what you call it) that acts
> on strings indexed by Unicode characters, not in terms of a
> representation.  Whether that string type internally uses UTF-16 or
> UTF-8 should be invisible to its user.  Ideally it would be capable of
> carrying its data internally in either form (so as to avoid needless
> conversion when both producer and consumer use the same form) and of
> converting between the two (e.g. so as to append efficiently) as needed.

That's what I'd support with both hands.
However, I don't think we could do that on QString without breaking most of
the existing code.

P.S. \note Unicode operates on "code points" not "characters". And
moreover, there is no such thing like "glyph" in Unicode string.
And looking for grapheme or glyph boundary is clearly not a string
storage's or a string view's responsibility.

Regards,
Konstantin


чт, 24 янв. 2019 г. в 10:33, Olivier Goffart :

> On 23.01.19 23:15, André Pönitz wrote:
> > On Wed, Jan 23, 2019 at 05:40:33PM +0300, Konstantin Tokarev wrote:
> >> 23.01.2019, 16:55, "Edward Welbourne" :
> >>> All of this discussion ignores a major elephant: QString's indexing is
> >>> by 16-bit UTF-16 tokens, not by Unicode characters. We've had Unicode
> >>> for a couple of decades now.
> >>>
> >>> We *should* have a string type (I don't care what you call it) that
> acts
> >>> on strings indexed by Unicode characters, not in terms of a
> >>> representation. Whether that string type internally uses UTF-16 or
> >>> UTF-8 should be invisible to its user. Ideally it would be capable of
> >>> carrying its data internally in either form (so as to avoid needless
> >>> conversion when both producer and consumer use the same form) and of
> >>> converting between the two (e.g. so as to append efficiently) as
> needed.
> >>
> >> I think this is excessive. Most common operations with strings in
> application
> >> code are:
> >>
> >> * Pass the string around or compare as an opaque token
> >> * Draw the string on screen e.g. with QPainter (while technically it
> >>falls in the previous category, I think it's important enough to
> >>deserve separate item)
> >> * Find substring or pattern (regex) inside the string
> >> * Split the string by character, pattern, or index boundaries found by
> means
> >>of previous item
> >>
> >> I think the only common cases when dealing with Unicode grapheme
> clusters
> >> is required are
> >>
> >> * Handling of text cursor movement
> >> * Implementation of text shaping, i.e. what Harfbuzz is doing
> >>
> >> I think having special iterator would be quite enough for cursor case.
> Such
> >> iterator could abstract away underlying encoding, instead of forcing
> everyone
> >> to convert to UTF-16 first.
> >
> > All of that is scarily close to my opinion on the topic.
>
> Same here. I think Konstantin is spot on.
>
> Another example of good string design, I think, is the Rust's String.
> Their
> string is encoded in valid UTF-8, indexed by bytes, and splitting the
> string in
> the middle of a code point is a programmer error.
>
> As already mentioned before, UTF-16 is quite a bad choice, if it weren't
> for
> legacy.
>
> The argument of that developper wrongly using indexes cause more problem
> with
> utf-8 than with utf-16 ("it would happen for a lot more characters")
> actually
> means that the developper will see and fix their bugs quickly.
>
> I understand changing QString to UTF-8 is a difficult task if we want to
> do it
> in a compatible way. However, I think there is a way:
> In Qt5.x:
>   - Introduce some iterator that iterates over unicode code points.
>   - Deprecate utf16()  and other API that assume that QString is UTF-16
>   - Replace them by a toUtf16 which returns a QVector.  I believe
> that
> it is possible to make the cotent implicitly shared with the QString,
> avoiding
> copies. (since it is just a QTypedArrayData internally)
>
> Then in Qt6 one can simply change the representation without breaking
> compatibility with non-deprecated functions.
>
> --
> Olivier
>
> Woboq - Qt services and support - https://woboq.com -
> https://code.woboq.org
>
>
>
>
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
>
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia czwartek, 24 stycznia 2019 11:24:41 CET Vitaly Fanaskov pisze:
> > Why not X instead?
> > --
> > 
> > - GitFlow, GitHub <= both are based on feature branches, that doesn't work
> > well with gerrit.
> 
> So, the only problem here is gerrit (aside from personal preferences and
> habits of course). The question is shouldn't we abandon gerrit as this
> tool doesn't feet with the needs well? Inventing new approaches is of
> course super cool... But not always necessary.
> 
> What's wrong with using, say, gitflow + gitlab + upsource? Upsource is
> optional, but it's the way better for code review than gerrit.

That is a completely different discussion. If we ever want to change the 
tooling we can re-consider branching model too.

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Aleksey Kontsevich]

Good point. Also thinking this way every time need to deal with Gerrit. Any 
chance to replace it to something more modern and convenient and less time 
wasting like GitLab, etc?

-- 
Best regards,
Aleksey
Linked in  https://www.linkedin.com/in/alekseykontsevich


24.01.2019, 12:28, "Vitaly Fanaskov" :
>> Why not X instead?
>> --
>>
>> - GitFlow, GitHub <= both are based on feature branches, that doesn't 
>> work
>>  well with gerrit.
>
> So, the only problem here is gerrit (aside from personal preferences and
> habits of course). The question is shouldn't we abandon gerrit as this
> tool doesn't feet with the needs well? Inventing new approaches is of
> course super cool... But not always necessary.
>
> What's wrong with using, say, gitflow + gitlab + upsource? Upsource is
> optional, but it's the way better for code review than gerrit.
>
> On 1/23/19 4:51 PM, Jedrzej Nowacki wrote:
>>  Hi,
>>
>> It is time to rethink our branch model. We are approaching Qt6 
>> development
>>  and I'm worried that what we have now, will simply not scale. As you know, 
>> our
>>  branch model is mainly(*) based on merging from stable up to development
>>  branches. In general, it is a very good model, which make sure that release
>>  branches are not getting obsolete too quickly, that most of the fixes are in
>>  the right places, that every commit is only once in the git history. It is a
>>  very clean model. It is also a very slow model, with a single point of
>>  failure.
>>
>> It is hard to maintain:
>> My impression is that the current model works great for a single release
>>  branch, but we have: 5.6 5.9 5.12 and soon we will have 5.13, that is 
>> without
>>  counting temporary patch level branches. Merging through them is hard, we
>>  already had to use "cherry-pick mode" in some places. When we add to the
>>  picture dev and qt6 branches, merging will be very, very hard. It is
>>  practically a full time job to update qt5 repository and coordinate all the
>>  merges now (thank you Liang!), shortly after qt6 branch opening amount of 
>> work
>>  will be much bigger.
>> It is slow:
>> The merges take time. We produce a lot of code, we have a lot of tests 
>> that
>>  needs to pass. Even single failure delays merge propagation by at least one
>>  day. If by bad luck, the merge contains some API incompatible changes an
>>  intermediate jump through Qt5 integration is required, that adds at least 3
>>  days of delay.
>>
>> --
>>
>> Proposal in short: let's use cherry-pick mode everywhere.
>>
>> All(**) changes would go to dev. From which they would be automatically
>>  cherry-picked by a bot to other branches. The decision to which branch 
>> cherry-
>>  pick, would be taken based on a tag in the commit message. We could add a
>>  footer that marks the change risk level as in quip-5 
>> (http://quips-qt-io.herokuapp.com/quip-0005.html), so for example "dev", 
>> "stable", "LTS". By
>>  default everything would be cherry-picked to qt6 branch unless "no-future" 
>> tag
>>  would be given. Of course we can bike-shed about the tag names.
>>
>> Advantages:
>> - no waiting for merges, a fix can be used right away after integration
>> - faster propagation of fixes targeting all branches, as there are no 
>> merges
>>  of merges
>> - simpler for new contributors, always push to dev
>> - in most cases a reviewer does no need to know what the current version
>>  number is
>> - conflict resolution is distributed and affects mostly the author of the
>>  change
>> - documents a change intent, which may be useful for people keeping own
>>  forks
>> - over time with increased amount of conflicts old branches, in natural 
>> way,
>>  stay untouched
>>
>> Disadvantages:
>> - git history would be a bit wilder, "git branch --contains" would not 
>> work
>> - commit messages in some branches would have kind of ugly footer as an
>>  effect of "cherry-pick -x"
>> - there is a chance, that some cherry-picked commits may be left 
>> forgotten
>>  in gerrit after a failed integration
>>
>> Bot details:
>>
>> The bot would listen only for changes in dev, in some unusual cases one
>>  could target an other branch directly, but bot would not trigger automatic
>>  cherry-pick(***). The bot would wait for a successful dev integration before
>>  creating cherry-picked changes. The bot would use cherry-pick -x to annotate
>>  the origin patch. After the cherry-pick creation, it would push it to 
>> gerrit,
>>  +2 and stage once. It would be up to the author to re-stage in case of
>>  flakiness. In case of a cherry-pick conflict it should push unresolved 
>> conflict
>>  to gerrit and add all reviewers and author to handle the issue.
>>
>>  The list below shows branch targets for automatic cherry-pick based on given
>>  tag.
>>
>>  dev (qt6)
>>  stable (qt6, 5.13)
>>  

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia czwartek, 24 stycznia 2019 09:08:29 CET Liang Qi pisze:
> My concern is about "cherry-pick" a series of changes for a big
> feature, especially during the period to have "dev" branches for both
> 5 and 6. I don't have solution for this issue yet.

My assumption is that bot would cherry-pick them in the same order in which 
patches got integrated to dev. That way we could reduce amount of issues. Of 
course if the first patch from a series causes conflicts then all other would 
also be in conflict.

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Ville Voutilainen]

On Wed, 23 Jan 2019 at 17:53, Jedrzej Nowacki  wrote:
>   Proposal in short: let's use cherry-pick mode everywhere.
>
>   All(**) changes would go to dev. From which they would be automatically
> cherry-picked by a bot to other branches. The decision to which branch cherry-
> pick, would be taken based on a tag in the commit message. We could add a
> footer that marks the change risk level as in quip-5 
> (http://quips-qt-io.herokuapp.com/quip-0005.html), so for example "dev", 
> "stable", "LTS". By
> default everything would be cherry-picked to qt6 branch unless "no-future" tag
> would be given. Of course we can bike-shed about the tag names.
>
>   Advantages:
>   - no waiting for merges, a fix can be used right away after integration
>   - faster propagation of fixes targeting all branches, as there are no merges
> of merges
>   - simpler for new contributors, always push to dev
>   - in most cases a reviewer does no need to know what the current version
> number is
>   - conflict resolution is distributed and affects mostly the author of the
> change
>   - documents a change intent, which may be useful for people keeping own
> forks
>   - over time with increased amount of conflicts old branches, in natural way,
> stay untouched
>
>   Disadvantages:
>   - git history would be a bit wilder, "git branch --contains" would not work
>   - commit messages in some branches would have kind of ugly footer as an
> effect of "cherry-pick -x"
>   - there is a chance, that some cherry-picked commits may be left forgotten
> in gerrit after a failed integration

+1 Strongly in Favor of this branching model. I use it in other
projects like GCC, I understand how it works, my
fixes land instantly on branches instead of waiting for a dedicated
person responsible for a merge to do it,
I never run into a situation where I fix a compilation problem in
5.12, someone else fixes it in 5.11, we all scratch
our heads about why we're duplicating work and also scratch our heads
how to resolve the merge conflict.
I can answer the questions about whether my fix will or will not land
in other branches and when, and I can make
sure it does land on the right branches because I can do those
cherry-picks myself, and I will, because I never
expect a magical merge to do it for me.

I consider this proposal a proposal to make our branching model
normal, regular, and unsurprising.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

On 23.1.2019 17.51, Jedrzej Nowacki wrote:
>>Can we use annotate instead of cherry-pick -x?
>>--
>>
>>No, but we should use it in addition. Annotations are mutable,
>> therefore are not 100% trust worthy. If we have both, we could
>> validate annotations against commit message, sadly it would not
>> protect against annotation removal.

Kari Oikarinen (24 January 2019 11:37)
> What are these annotations? I tried to google them but only found
> documentation for git-annotate, which is just git-blame with a
> different output format.

They're actually called git "notes" (or "object notes");
see man git-notes.

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 17:37:57 CET Konstantin Tokarev pisze:
> Note that backporting changes from dev should also be a full-time job for
> someone, otherwise amount of fixes going to stable branches will likely
> drop

Yes, depending who you ask it is good or bad. It means that only fixes that 
someone really cares about are really integrated to the stable branches. If 
somebody cares about a fix it means that it is more important. More over 
cherry-pick mode distributes the load of conflict resolution.

> >   - there is a chance, that some cherry-picked commits may be left
> > forgotten in gerrit after a failed integration
> 
> Or not submitted at all because of failed cherry-pick. Also, notorious
> property of cherry-pick strategy is explosion of conflicts: if there is a
> patch series modifying some code, and first of them requires adjustment to
> resolve conflict, it's likely that all fol
> lowing patches will have
> conflicts as well in the same places.

Assumption is that a bot would do the cherry-picking for you. If a cherry-pick 
cause conflicts, then the bot would push unresolved conflict to gerrit. So no, 
all changes would be in gerrit. Some may be to hard to resolve or one would 
lack a motivation. Again, main point is that it is up to author to resolve the 
conflicts.

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jedrzej Nowacki]

Dnia środa, 23 stycznia 2019 18:49:46 CET Thiago Macieira pisze:
> On Wednesday, 23 January 2019 08:37:57 PST Konstantin Tokarev wrote:
> > >   Disadvantages:
> > >   - git history would be a bit wilder, "git branch --contains" would not
> > >   work
> > >   - commit messages in some branches would have kind of ugly footer as
> > >   an
> > > 
> > > effect of "cherry-pick -x"
> > 
> > Gerrit's Change-Id can be used to track presence of patch in branches of
> > interest
> 
> Yes, but not as easily, since the git branch --contains and git tag
> --contains are pure DAG operations. The search you're talking about is a
> text search (usually implemented by a regexp search) on the commit message,
> with no DAG boundaries. You have to scan all valid branches for a given
> string.
> 
> And then you still have to run git branch --contains on each entry you found
> to figure out which branches contain those commits.
> 
> So this is scriptable. It's going to be something like 100x slower than
> today, but it should still finish within 10 seconds, even on slow machines.

As we would not be the only one using it, I bet the script could be 
upstream'ed to git.

Cheers,
  Jędrek


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 23.1.2019 17.51, Jedrzej Nowacki wrote:
>Can we use annotate instead of cherry-pick -x?
>--
> 
>No, but we should use it in addition. Annotations are mutable, therefore 
> are
> not 100% trust worthy. If we have both, we could validate annotations against
> commit message, sadly it would not protect against annotation removal.

What are these annotations? I tried to google them but only found documentation
for git-annotate, which is just git-blame with a different output format.

-- 
Kari
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Vitaly Fanaskov]

>Why not X instead?
>--
>
>- GitFlow, GitHub <= both are based on feature branches, that doesn't work
> well with gerrit.
So, the only problem here is gerrit (aside from personal preferences and 
habits of course). The question is shouldn't we abandon gerrit as this 
tool doesn't feet with the needs well? Inventing new approaches is of 
course super cool... But not always necessary.

What's wrong with using, say, gitflow + gitlab + upsource? Upsource is 
optional, but it's the way better for code review than gerrit.

On 1/23/19 4:51 PM, Jedrzej Nowacki wrote:
> Hi,
>
>It is time to rethink our branch model. We are approaching Qt6 development
> and I'm worried that what we have now, will simply not scale. As you know, our
> branch model is mainly(*) based on merging from stable up to development
> branches. In general, it is a very good model, which make sure that release
> branches are not getting obsolete too quickly, that most of the fixes are in
> the right places, that every commit is only once in the git history. It is a
> very clean model. It is also a very slow model, with a single point of
> failure.
>
>It is hard to maintain:
>My impression is that the current model works great for a single release
> branch, but we have: 5.6 5.9 5.12 and soon we will have 5.13, that is without
> counting temporary patch level branches. Merging through them is hard, we
> already had to use "cherry-pick mode" in some places. When we add to the
> picture dev and qt6 branches, merging will be very, very hard. It is
> practically a full time job to update qt5 repository and coordinate all the
> merges now (thank you Liang!), shortly after qt6 branch opening amount of work
> will be much bigger.
>It is slow:
>The merges take time. We produce a lot of code, we have a lot of tests that
> needs to pass. Even single failure delays merge propagation by at least one
> day. If by bad luck, the merge contains some API incompatible changes an
> intermediate jump through Qt5 integration is required, that adds at least 3
> days of delay.
>
>--
>
>Proposal in short: let's use cherry-pick mode everywhere.
>
>All(**) changes would go to dev. From which they would be automatically
> cherry-picked by a bot to other branches. The decision to which branch cherry-
> pick, would be taken based on a tag in the commit message. We could add a
> footer that marks the change risk level as in quip-5 
> (http://quips-qt-io.herokuapp.com/quip-0005.html), so for example "dev", 
> "stable", "LTS". By
> default everything would be cherry-picked to qt6 branch unless "no-future" tag
> would be given. Of course we can bike-shed about the tag names.
>
>Advantages:
>- no waiting for merges, a fix can be used right away after integration
>- faster propagation of fixes targeting all branches, as there are no 
> merges
> of merges
>- simpler for new contributors, always push to dev
>- in most cases a reviewer does no need to know what the current version
> number is
>- conflict resolution is distributed and affects mostly the author of the
> change
>- documents a change intent, which may be useful for people keeping own
> forks
>- over time with increased amount of conflicts old branches, in natural 
> way,
> stay untouched
>
>Disadvantages:
>- git history would be a bit wilder, "git branch --contains" would not work
>- commit messages in some branches would have kind of ugly footer as an
> effect of "cherry-pick -x"
>- there is a chance, that some cherry-picked commits may be left forgotten
> in gerrit after a failed integration
>
>Bot details:
>
>The bot would listen only for changes in dev, in some unusual cases one
> could target an other branch directly, but bot would not trigger automatic
> cherry-pick(***). The bot would wait for a successful dev integration before
> creating cherry-picked changes. The bot would use cherry-pick -x to annotate
> the origin patch. After the cherry-pick creation, it would push it to gerrit,
> +2 and stage once. It would be up to the author to re-stage in case of
> flakiness. In case of a cherry-pick conflict it should push unresolved 
> conflict
> to gerrit and add all reviewers and author to handle the issue.
>
> The list below shows branch targets for automatic cherry-pick based on given
> tag.
>
> dev (qt6)
> stable (qt6, 5.13)
> stable, no-future (5.13)
> LTS (qt6, 5.13, 5.12)
> LTS-strict (qt6, 5.13, 5.12, 5.9)
> LTS, no-future (5.13, 5.12)
>
> That is assuming that we have branches: qt6  dev  5.13  5.13.q  5.12  5.12.w
> 5.9  5.9.e  5.6  5.6.r
> I think we should assume that patch level branches, as well LTS in very strict
> state, are handled manually.
>
>
>Creation of new branches:
> We would branch more or less as usual. The only difference from the current
> system is that we would not need the back merge / soft branching 

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 24.1.2019 11.49, Edward Welbourne wrote:
> Olivier Goffart (24 January 2019 08:03)
>> Let's start by looking at the "problem"
> 
> Always a good place to start.
> 
> On 23.01.19 16:51, Jedrzej Nowacki wrote:
>>> My impression is that the current model works great for a single
>>> release branch, but we have: 5.6 5.9 5.12 and soon we will have 5.13,
>>> that is without counting temporary patch level branches.
> 
>> But the LTS and patch level branches are already in cherry-pick mode!
> 
> Indeed: Jedrzej does here over-state the problem.
> 
>> So we have only ever have one release branch. Therefore the current
>> model works great (your own words).
> 
> We routinely have two branches other than dev that are merged up to dev,
> one via the other.  Thus, until 5.11 closed, we were merging 5.11 up
> through 5.12 to dev; once 5.13 branches, we'll be merging 5.12 through
> 5.13 up to dev.  Once 5.13 closes, we'll put 5.12 in cherry-pick mode
> and be back to just one (as we are right now, temporarily); but wne 5.15
> branches we'll be back to 5.14 -> 5.15 -> dev again.  Then, of course,
> we also have the 5.x.y branches transiently, which does complicate life,
> but I'm going to leave it out of my analysis.
> 
> So some of the time we have three branches, some of the time two,
> between which we merge; plus we have an LTS or two to which we
> cherry-pick.  (Which means we mix models.)
> 
> However, merge-mode always implies more delay for a change's propagation
> compared to the cherry-pick model, at least when the developer whose
> change is considered is eager to get it propagated.

In general it leads to a longer delay until the change is in all relevant
branches. It does however lead to a change getting into the branch closest to
release faster, because it doesn't need to get into dev first. Branches close to
release are an important case.

Yes, you can deviate from the process in urgent cases. But even if the release
is not right at the door to justify the deviation, the change landing sooner
gives more time to find any possible issues in it.

> 
>>> Merging through them is hard, we already had to use "cherry-pick
>>> mode" in some places. When we add to the picture dev and qt6
>>> branches, merging will be very, very hard. It is practically a full
>>> time job to update qt5 repository and coordinate all the merges now
>>> (thank you Liang!),
> 
>> But as other have already said, the total amount of work would still be the
>> same. Actually worse since one would have to manually stage the patch and 
>> fail
>> the tests and resolve the conflicts that happen between them as they may 
>> occurs
>> in different order.  Overall, i'd say this is much more burden for branches
>> which still are supposed to get lots of patches.
> 
> There would, indeed, be extra staging with the proposed model; and
> patches may land in a different order.  The latter bothers me more.

Add to that the fact that a long-lived branch that is cherry-picked to may lead
to having to resolve the same conflict multiple times. (As discussed elsewhere
in the thread.)

Especially so for qt6 branch, which would be getting almost every change made in
dev + API breaking changes preparing for Qt 6. How many changes will actually
have the no-future tag?

>>> shortly after qt6 branch opening amount of work will be much bigger.
> 
>> But I fail to see how the proposed model helps with that in any way.
> 
> ... and we'll eventually have to merge the qt6 branch with dev, which is
> going to be "fun" if they've had cherry-picks between them that have had
> conflicts (as we can expect) that have been resolved.  Whoever does that
> merge is going to wish we'd been using merges from dev to qt6 earlier.

If I understood the proposal right, there wouldn't be a merge from dev into the
qt6 branch. Rather the dev branch would be deleted and the qt6 branch renamed to
dev. It already has all the changes, except those with the tag no-future

-- 
Kari
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

On 24.1.2019 11.13, Edward Welbourne wrote:
>> A cherry-pick takes the diff involved in one commit and patches
>> another check-out with it.  A merge uses the digraph of commits in
>> sophisticated ways; a cherry-pick does not.

Kari Oikarinen (24 January 2019 10:33)
> Quoting man page for git-cherry-pick: https://git-scm.com/docs/git-cherry-pick
>
> "For conflicting paths, the index file records up to three versions,
> as described in the "TRUE MERGE" section of git-merge[1]. The working
> tree files will include a description of the conflict bracketed by the
> usual conflict markers <<< and >>>."
>
> It is similar enough that it actually refers to the same section of
> documentation!

I stand corrected.
Time to re-read the cherry-pick docs.
My apologies to Allan for my misunderstanding.

> But it doesn't actually create a new common ancestor for the
> commits. So every further cherry-pick can still hit the same
> conflict. Instead of being resolved by the merge, there's an unbounded
> amount of time that different people will hit it and need to resolve
> it.

That's a good point.

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 24.1.2019 11.11, Volker Hilsheimer wrote:
> 
> 
>> On 23 Jan 2019, at 22:09, Allan Sandfeld Jensen > > wrote:
>>
>> On Mittwoch, 23. Januar 2019 21:42:35 CET Edward Welbourne wrote:
>>> Jedrzej Nowacki wrote:
> Advantages:
> - no waiting for merges, a fix can be used right away after
>
>   integration
>
> - faster propagation of fixes targeting all branches, as there are
>
>   no merges of merges
>>>
>>> Alex Blasche (23 January 2019 18:09)
>>>
 This is pure speculation because you potentially triple (or worse) the
 amount of individual patches requiring a merge in gerrit when you
 consider that you want to at least merge to 5.9, 512, dev and qt6. I
 don't see this prediction come true.
>>>
>>> Well, as soon as it hits dev, the patch is cherry-picked to every branch
>>> that its footer says it belongs in.  Those branches where all goes well
>>> see it one integration later.  Each branch that has a conflict needs
>>> that resolved before we get to that second integrtion.  Contrast this
>>> with a 5.12 -> 5.13 -> dev chain of merges, where dev doesn't get the
>>> change that landed in 5.12 (even if that change could land in dev
>>> without conflict) until
>>> * there's been some delay between their change being accepted in 5.12
>>>   and the next merge-bot run
>>> * everyone who made change to 5.12 that conflicted on merging to 5.13
>>>   has advised Liang on how to resolve their conflicts
>>> * we've got the result through integration into 5.13
>>> * everyone who's made changes to 5.13 or (as possibly just amended in
>>>   merging) 5.12 that conflicts with anything in dev has again advised
>>>   how to resolve their conflicts
>>> * and we've got the result through a second integration, into dev.
>>>
>>> When nothing but the change being considered has a conflict along
>>> the way, that's twice as long; and any change to an upstream branch,
>>> that does have a conflict, introduces delay for all the other changes
>>> that landed in that branch, even if they don't have conflicts.  In the
>>> middle of summer, when lots of folk are away on holiday, getting help
>>> with resolving conflicts isn't easy - the folk who know those commits
>>> won't be back for a month - and all changes, no matter how urgent, get
>>> stuck behind any conflict we can't work out how to resolve.
>>>
>>> So no, Jedrzej's claim is not *pure* speculation; it's at least quite a
>>> lot adulterated with reasons to believe that many changes would, under
>>> his scheme, propagate to all branches they're destined for sooner than
>>> happens with our present scheme.
>>>
>> No, it is speculation, and it optimizing the least important case: bug-fixes
>> in dev. Dev is the branch that can wait the longest to get a bug-fix, the
>> stable branch is the branch that need it the most urgent. And fixing a bug in
>> 5.12 now means you first fix it where you need it (5.12), then rewrite it for
>> dev, then resolve the inevitable conflict back so it can be merged, all
>> waiting for bots and release teams to stumple into the issues and delaying 
>> the
>> next 5.12.x release.
>>
>> ‘Allan
> 
> 
> More people working and building against dev helps keep dev more stable (by 
> virtue of discovering breakage sooner), and the proposal encourages more 
> people 
> to work on dev. That can be a good thing.

Our users use our releases. Our development process should aim at producing good
releases (at scheduled times). This proposal is putting more focus on dev, but
will it lead to better quality releases? Users won't care about a dev branch
that gets fixes sooner, if they have to wait for those important fixes longer
than in the current situation.

> 
> Urgency to get fixes into a stable branch is an argument that has come up a 
> few 
> times. The current 5.12.1 release seems to be delayed because some changes 
> made 
> it into 5.12 that shouldn’t have been there in the first place (given the 
> definition of “stable”). It would be interesting to see some data about how 
> many 
> point releases we had to delay because “highly desirable fix for old bug was 
> stuck in the process” vs “regression was introduced in a stable branch and 
> only 
> discovered during release testing”.

The absolute path changes got into 5.12.1 due to an explicit wish to have them
in a fresh LTS branch early. Reviewers accepted that and could have done so
under both models as easily.

The argument that these changes help by reducing changes in stable branches is
that currently people are putting in changes that don't belong there. Making
getting changes into stable branches more painful with the new model will lead
to less changes getting in. This will happen due to people avoiding work they
can avoid.

Importantly this is not the same thing as more careful consideration about
whether a change *should* go to the stable branch. The changes will be selected
based on how persistent people are, not on how 

Re: [Development] Proposal: New branch model

[Edward Welbourne]

Olivier Goffart (24 January 2019 08:03)
> Let's start by looking at the "problem"

Always a good place to start.

On 23.01.19 16:51, Jedrzej Nowacki wrote:
>>My impression is that the current model works great for a single
>> release branch, but we have: 5.6 5.9 5.12 and soon we will have 5.13,
>> that is without counting temporary patch level branches.

> But the LTS and patch level branches are already in cherry-pick mode!

Indeed: Jedrzej does here over-state the problem.

> So we have only ever have one release branch. Therefore the current
> model works great (your own words).

We routinely have two branches other than dev that are merged up to dev,
one via the other.  Thus, until 5.11 closed, we were merging 5.11 up
through 5.12 to dev; once 5.13 branches, we'll be merging 5.12 through
5.13 up to dev.  Once 5.13 closes, we'll put 5.12 in cherry-pick mode
and be back to just one (as we are right now, temporarily); but wne 5.15
branches we'll be back to 5.14 -> 5.15 -> dev again.  Then, of course,
we also have the 5.x.y branches transiently, which does complicate life,
but I'm going to leave it out of my analysis.

So some of the time we have three branches, some of the time two,
between which we merge; plus we have an LTS or two to which we
cherry-pick.  (Which means we mix models.)

However, merge-mode always implies more delay for a change's propagation
compared to the cherry-pick model, at least when the developer whose
change is considered is eager to get it propagated.

>> Merging through them is hard, we already had to use "cherry-pick
>> mode" in some places. When we add to the picture dev and qt6
>> branches, merging will be very, very hard. It is practically a full
>> time job to update qt5 repository and coordinate all the merges now
>> (thank you Liang!),

> But as other have already said, the total amount of work would still be the
> same. Actually worse since one would have to manually stage the patch and fail
> the tests and resolve the conflicts that happen between them as they may 
> occurs
> in different order.  Overall, i'd say this is much more burden for branches
> which still are supposed to get lots of patches.

There would, indeed, be extra staging with the proposed model; and
patches may land in a different order.  The latter bothers me more.

>> shortly after qt6 branch opening amount of work will be much bigger.

> But I fail to see how the proposed model helps with that in any way.

... and we'll eventually have to merge the qt6 branch with dev, which is
going to be "fun" if they've had cherry-picks between them that have had
conflicts (as we can expect) that have been resolved.  Whoever does that
merge is going to wish we'd been using merges from dev to qt6 earlier.

>>It is slow:
>>The merges take time. We produce a lot of code, we have a lot of tests 
>> that
>> needs to pass. Even single failure delays merge propagation by at least one
>> day. If by bad luck, the merge contains some API incompatible changes an
>> intermediate jump through Qt5 integration is required, that adds at least 3
>> days of delay.

> Incompatile changes should not happen in a release branch normally. (Or very
> seldomly, as fixes on new API)

No, but they do happen in dev, which complicates merging things up to
dev - which, I suppose, was Jedrzej's point.

> And as it has been said already, there is no priority to get changes quick in
> the dev branch. On the other hand, haing the change faster in the stable 
> branch
> are more important, for it is going to be released first. So your model would
> delay the change in the branch where it matters more.

The delay would only happen to things that got conflicts or on which CI
failed; and, as long as someone's eager to get the change into the
target branch, it's likely to be delayed by at most about a day.  In
contrast, when we do have three branches, the fix in a release branch
takes several days to reach the stable branch and about a week to reach
dev.

When the fix is for breakage that's causing Coin failures, that's
holding up everyone for a week.  We get this not so infrequently.  With
the cherry-pick model, the fix would be done first in dev and would
reach all other affected branches probably within a day.

>[...]
>>- Stay with the current solution <= the merge effort is too big and qt6 is
>> expected to cause conflicts that really should not be solved by one person

> Again, I don't see how the proposed model reduce the amount of
> conflicts.

It doesn't.  It does, however, ensure that conflicts are dealt with by
the people who made the change on one side of the conflict, while the
change is still fresh in their minds.  That is probably better than
having the conflict resolved by someone who knows neither half of the
change.  OTOH, it runs the risk that the other side of the conflict
won't be properly understood by those resolving it, creating an
asymmetry that's absent when the resolver is equally ignorant of both
sides.

> 

Re: [Development] Proposal: New branch model

[Volker Hilsheimer]

> On 24 Jan 2019, at 09:22, Jaroslaw Kobus  wrote:
>>> "All(**) changes would go to dev. From which they would be automatically
>>> cherry-picked by a bot to other branches. The decision to which branch 
>>> cherry-
>>> pick, would be taken based on a tag in the commit message. We could add a
>>> footer that marks the change risk level as in quip-5"
>>> 
>>> No sure I understand the above correctly. Let's say in dev branch some 
>>> source file got refactored completely, so that no single line match the old 
>>> version anymore, e.g. Qt 5.9. Now you need to fix the old code, which is in 
>>> 5.9 branch - how in this case you may try to push your fix to dev?
>>> 
>>> Jarek
>>> 
>> That’s where the “with exception of branch specific changes” clause (which 
>> the ** points at) kicks in.
>> 
>> Is the fix needed in dev (or is the bug fixed by the refactoring)?
>> 
>> If yes, then fix it in dev, and then make a separate fix in the relevant LTS 
>> branches (perhaps starting with the cherry-pick’ed change). Or just accept 
>> that this bug won’t/can’t be fixed in the pre-refactoring codebase.
>> 
>> If no, then push the fix for the newest branch where it’s needed, from where 
>> it can be cherry picked further; don’t do anything in dev (including “don’t 
>> expect someone that knows nothing about your change to deal with the merge 
>> conflict”).
>> 
> In this case there is additional step then. You would be forced now to check 
> first both 5.9 and dev branches and detect if a fix would be "applyable" to 
> the dev or not.


Testing whether the bug that I’m fixing exists in dev or not is part of the 
drill of fixing bug, isn’t it? Why would you spend time on fixing something in 
5.12 without checking whether the issue is still present in the latest 
codebase? Perhaps the issue has been fixed already, just without the author 
considering it relevant for 5.12 (perhaps for good reasons).


> Besides, if you decide that your fix goes only to the 5.9 branch - in this 
> case you follow the current model anyway, right? So, having two models isn't 
> better than having just one I think.


With the proposed model there are no more automatic merges from more stable to 
less stable, so once my change made it into 5.9, that is it.


Volker


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Oswald Buddenhagen]

On Thu, Jan 24, 2019 at 09:13:32AM +, Edward Welbourne wrote:
> A cherry-pick takes the diff involved in one commit and patches another
> check-out with it.  A merge uses the digraph of commits in sophisticated
> ways; a cherry-pick does not.
> 
uhhh, actually, it does. cherry-pick even has the --strategy option from
merge.
compare also git am -3.

it seems plausible that a longer series of cherry-picks to the same area
would increasingly defeat the merge logic.

while at it, i'll also note that last time i checked git was somewhat
weak for a cherry-picking workflow compared to subversion and a bunch of
other VCSs aimed at exclusively linear histories, because it doesn't
record where commits were picked _to_ (though that would be actually
implementable on top of notes).
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Allan Sandfeld Jensen]

On Donnerstag, 24. Januar 2019 10:13:32 CET Edward Welbourne wrote:
> OK, now you're just engaging in ill-informed FUD.
> Cherry-picks do not involve any three-way anything.
> You clearly do not understand the difference between merging and
> cherry-picking.
> 
> A cherry-pick takes the diff involved in one commit and patches another
> check-out with it.  A merge uses the digraph of commits in sophisticated
> ways; a cherry-pick does not.
> 
They do on my machine when I do git git cherry-picks. I do that about a 
hundred times every month when maintaining qtwebengine, and I use the three 
way diff to resolve conflicts. They way I avoid the three way diff getting 
useless is by resetting the chromium fork regularly, so there is always a 
nearest common ancestor at the last fork point.

Git performs merges and cherry-picks the same way, the main difference is how 
they are recorded in the history, where merges provide a better history to do 
future merges and cherry-picks on.

'Allan





___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Kari Oikarinen]

On 24.1.2019 11.13, Edward Welbourne wrote:
> 23.01.2019, 21:38, "Alex Blasche" :
 At the end of the day each cherry-pick is a merge too
> 
> Merges and cherry-picks have a certain amount in common, but they are
> not the same thing at all.
> 
 and they can conflict too. The conflict resolution process is still
 the same. if everything is conflict free then a git merge would be
 no more difficult than a cherry-pick.
> 
> This is true.
> 
> On Mittwoch, 23. Januar 2019 19:43:06 CET Konstantin Tokarev wrote:
>>> And when conflicts are present, cherry-picking N patches may result in N
>>> times more work than merge in worst case (and same amount of work in the
>>> best case)
> 
> Allan Sandfeld Jensen (23 January 2019 23:05)
>> More than that. Once you have had cherry-pick only for a while git will be
>> unable to find useful common ancestors for the changes, and will be unable to
>> do smart three-way merging of cherry-picks,
> 
> OK, now you're just engaging in ill-informed FUD.
> Cherry-picks do not involve any three-way anything.
> You clearly do not understand the difference between merging and
> cherry-picking.
> 
> A cherry-pick takes the diff involved in one commit and patches another
> check-out with it.  A merge uses the digraph of commits in sophisticated
> ways; a cherry-pick does not.
> 

Quoting man page for git-cherry-pick: https://git-scm.com/docs/git-cherry-pick

"For conflicting paths, the index file records up to three versions, as 
described in the "TRUE MERGE" section of git-merge[1]. The working tree files 
will include a description of the conflict bracketed by the usual conflict 
markers <<< and >>>."

It is similar enough that it actually refers to the same section of
documentation!

But it doesn't actually create a new common ancestor for the commits. So every
further cherry-pick can still hit the same conflict. Instead of being resolved
by the merge, there's an unbounded amount of time that different people will hit
it and need to resolve it.

-- 
Kari
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Allan Sandfeld Jensen]

On Donnerstag, 24. Januar 2019 10:11:35 CET Volker Hilsheimer wrote:
> More people working and building against dev helps keep dev more stable (by
> virtue of discovering breakage sooner), and the proposal encourages more
> people to work on dev. That can be a good thing.
 
No,  it will overload the CI by having all integrations happen on the same 
branch, instead being able to integrate multiple branches at once.

> Urgency to get fixes into a stable branch is an argument that has come up a
> few times. The current 5.12.1 release seems to be delayed because some
> changes made it into 5.12 that shouldn’t have been there in the first place
> (given the definition of “stable”). It would be interesting to see some
> data about how many point releases we had to delay because “highly
> desirable fix for old bug was stuck in the process” vs “regression was
> introduced in a stable branch and only discovered during release testing”.
> 
I dont see that. 5.12.1 branched a long time ago and very early, patches in 
5.12 shouldnt affect it. It is stalled because certain things needs to be 
fixed that arent fixed.

> Without data, I would speculate that perhaps a model that leads to fewer
> changes made to stable branches because it requires an explicit decision
> (and perhaps extra work) to get changes there is not such a terrible idea.
> And making the work to make changes to a stable branch, ie the cost of
> maintenance, very visible is also a good thing.
 
> Either way, nothing in the proposal prevents us from developing and push a
> fix for 5.12, and then develop a different fix for dev that we don’t cherry
> pick into 5.12 or other stable branches. This should be an exception, used
> in urgent situations where indeed we can’t wait. For those cases, there’s
> no merge conflict in the first place in the new model (because there’s no
> merge).
 
It is still doing merging the wrong way for no good reason. Even if cherry-
picking itself wasnt a bad idea. You could just as well commit to the stable 
branch first and then cherry-pick it to the other branches. That way there 
would be no need for cryptic annotations, the branches to pick to would be 
implicit from where the patch first landed. Or even better, you could perform 
a merge to dev on every single commit to the stable branch, or even better, 
only create a merge if there are conflicts, so we don't overload the CI on 
dev.

'Allan



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Volker Hilsheimer]

> On 24 Jan 2019, at 08:03, Olivier Goffart  wrote:
> [...]>- Stay with the current solution <= the merge effort is too big and 
> qt6 is
>> expected to cause conflicts that really should not be solved by one person
> 
> Again, I don't see how the proposed model reduce the amount of conflicts.
> If the "one person" is the problem, then nothing prevents you to assign more 
> people to the job. One easy way is already to share the different modules 
> (repositories). But with some cooperation it is also possible to share the 
> work accross directories, or by number of commits. One can also be pragmatic 
> and revert most problematic commit (that fails tests) in dev or stable, then 
> let the author work at it again.

Having to wait for someone else to trigger the merge and resolve the various 
conflicts before I can continue to base my work on dev on a fix that I already 
made in 5.12 breaks flow. Distributing this work to more people doesn’t solve 
the problem.

The whole notion that my change has to become someone else’s problem by design 
of the merge process is more than just a little crazy to me. I want to own my 
change, have control over which branches it hits, and be responsible for 
cleaning up the mess my change might have caused. The current model doesn’t 
give me any of that beyond the initial merge.

Volker

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Edward Welbourne]

23.01.2019, 21:38, "Alex Blasche" :
>>> At the end of the day each cherry-pick is a merge too

Merges and cherry-picks have a certain amount in common, but they are
not the same thing at all.

>>> and they can conflict too. The conflict resolution process is still
>>> the same. if everything is conflict free then a git merge would be
>>> no more difficult than a cherry-pick.

This is true.

On Mittwoch, 23. Januar 2019 19:43:06 CET Konstantin Tokarev wrote:
>> And when conflicts are present, cherry-picking N patches may result in N
>> times more work than merge in worst case (and same amount of work in the
>> best case)

Allan Sandfeld Jensen (23 January 2019 23:05)
> More than that. Once you have had cherry-pick only for a while git will be
> unable to find useful common ancestors for the changes, and will be unable to
> do smart three-way merging of cherry-picks,

OK, now you're just engaging in ill-informed FUD.
Cherry-picks do not involve any three-way anything.
You clearly do not understand the difference between merging and
cherry-picking.

A cherry-pick takes the diff involved in one commit and patches another
check-out with it.  A merge uses the digraph of commits in sophisticated
ways; a cherry-pick does not.

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Volker Hilsheimer]

On 23 Jan 2019, at 22:09, Allan Sandfeld Jensen 
mailto:k...@carewolf.com>> wrote:

On Mittwoch, 23. Januar 2019 21:42:35 CET Edward Welbourne wrote:
Jedrzej Nowacki wrote:
Advantages:
- no waiting for merges, a fix can be used right away after

  integration

- faster propagation of fixes targeting all branches, as there are

  no merges of merges

Alex Blasche (23 January 2019 18:09)

This is pure speculation because you potentially triple (or worse) the
amount of individual patches requiring a merge in gerrit when you
consider that you want to at least merge to 5.9, 512, dev and qt6. I
don't see this prediction come true.

Well, as soon as it hits dev, the patch is cherry-picked to every branch
that its footer says it belongs in.  Those branches where all goes well
see it one integration later.  Each branch that has a conflict needs
that resolved before we get to that second integrtion.  Contrast this
with a 5.12 -> 5.13 -> dev chain of merges, where dev doesn't get the
change that landed in 5.12 (even if that change could land in dev
without conflict) until
* there's been some delay between their change being accepted in 5.12
  and the next merge-bot run
* everyone who made change to 5.12 that conflicted on merging to 5.13
  has advised Liang on how to resolve their conflicts
* we've got the result through integration into 5.13
* everyone who's made changes to 5.13 or (as possibly just amended in
  merging) 5.12 that conflicts with anything in dev has again advised
  how to resolve their conflicts
* and we've got the result through a second integration, into dev.

When nothing but the change being considered has a conflict along
the way, that's twice as long; and any change to an upstream branch,
that does have a conflict, introduces delay for all the other changes
that landed in that branch, even if they don't have conflicts.  In the
middle of summer, when lots of folk are away on holiday, getting help
with resolving conflicts isn't easy - the folk who know those commits
won't be back for a month - and all changes, no matter how urgent, get
stuck behind any conflict we can't work out how to resolve.

So no, Jedrzej's claim is not *pure* speculation; it's at least quite a
lot adulterated with reasons to believe that many changes would, under
his scheme, propagate to all branches they're destined for sooner than
happens with our present scheme.

No, it is speculation, and it optimizing the least important case: bug-fixes
in dev. Dev is the branch that can wait the longest to get a bug-fix, the
stable branch is the branch that need it the most urgent. And fixing a bug in
5.12 now means you first fix it where you need it (5.12), then rewrite it for
dev, then resolve the inevitable conflict back so it can be merged, all
waiting for bots and release teams to stumple into the issues and delaying the
next 5.12.x release.

‘Allan


More people working and building against dev helps keep dev more stable (by 
virtue of discovering breakage sooner), and the proposal encourages more people 
to work on dev. That can be a good thing.

Urgency to get fixes into a stable branch is an argument that has come up a few 
times. The current 5.12.1 release seems to be delayed because some changes made 
it into 5.12 that shouldn’t have been there in the first place (given the 
definition of “stable”). It would be interesting to see some data about how 
many point releases we had to delay because “highly desirable fix for old bug 
was stuck in the process” vs “regression was introduced in a stable branch and 
only discovered during release testing”.

Without data, I would speculate that perhaps a model that leads to fewer 
changes made to stable branches because it requires an explicit decision (and 
perhaps extra work) to get changes there is not such a terrible idea. And 
making the work to make changes to a stable branch, ie the cost of maintenance, 
very visible is also a good thing.

Either way, nothing in the proposal prevents us from developing and push a fix 
for 5.12, and then develop a different fix for dev that we don’t cherry pick 
into 5.12 or other stable branches. This should be an exception, used in urgent 
situations where indeed we can’t wait. For those cases, there’s no merge 
conflict in the first place in the new model (because there’s no merge).


Volker



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Jaroslaw Kobus]

From: Volker Hilsheimer
Sent: Wednesday, January 23, 2019 5:25 PM
To: Jaroslaw Kobus
Cc: Jedrzej Nowacki; development@qt-project.org
Subject: Re: [Development] Proposal: New branch model

On 23 Jan 2019, at 17:08, Jaroslaw Kobus 
mailto:jaroslaw.ko...@qt.io>> wrote:

"All(**) changes would go to dev. From which they would be automatically
cherry-picked by a bot to other branches. The decision to which branch cherry-
pick, would be taken based on a tag in the commit message. We could add a
footer that marks the change risk level as in quip-5"

No sure I understand the above correctly. Let's say in dev branch some source 
file got refactored completely, so that no single line match the old version 
anymore, e.g. Qt 5.9. Now you need to fix the old code, which is in 5.9 branch 
- how in this case you may try to push your fix to dev?

Jarek


That’s where the “with exception of branch specific changes” clause (which the 
** points at) kicks in.

Is the fix needed in dev (or is the bug fixed by the refactoring)?

If yes, then fix it in dev, and then make a separate fix in the relevant LTS 
branches (perhaps starting with the cherry-pick’ed change). Or just accept that 
this bug won’t/can’t be fixed in the pre-refactoring codebase.

If no, then push the fix for the newest branch where it’s needed, from where it 
can be cherry picked further; don’t do anything in dev (including “don’t expect 
someone that knows nothing about your change to deal with the merge conflict”).


Volker


In this case there is additional step then. You would be forced now to check 
first both 5.9 and dev branches and detect if a fix would be "applyable" to the 
dev or not.

Besides, if you decide that your fix goes only to the 5.9 branch - in this case 
you follow the current model anyway, right? So, having two models isn't better 
than having just one I think.

Jarek
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Proposal: New branch model

[Liang Qi]

Lots of talks happened already on this thread.

I would like to think old model as "merge" model VS new model as
"cherry-pick" model.

In the old "merge" model, normally we will drop a branch out to
cherry-pick mode after having more than 3 branches and try to have 2
branches(change LTS branch to cherry-pick mode after a while). The
worst case is when we have 5.12/5.13/dev branches with regular
5.12->5.13->dev merges, and also 5.12.x and 5.13.y release branches,
more merges involved. You can imagine the "chaos" when a critical fix
is needed for multiple branches, especially the delay for development
in dev branch.

In this scenario, I will trigger a qt5 integration(with latest sha1
from all submodules) in 5.12, it will take about 1 day(at least 8
hours, a work day) to see whether it is broken or not. After fix
everything in 5.12, I will try 5.13, same process, and dev. The whole
round is quite painful, especially around feature frozen period, if
some API changes happened, more ping-pong rounds are involved, more
details see 
https://lists.qt-project.org/pipermail/development/2016-October/055084.html
(old url was invalid any more, after the mailing list scripts
regenerated the web pages.)

From last year(2018), I had a merge script/bot
(https://codereview.qt-project.org/#/q/owner:%22Qt+Forward+Merge+Bot%22,n,z)
to do the merge work, it checks qtbase and qtdeclarative daily, and do
merge if more than 5 new changes happened. And check other modules
twice a week, and do merge if there is new change old than 1 week.
BTW, qtwebengine is a bit special, and some module maintainers(like
J-P for qtquickcontrols2) triggered the merge regularly by themselves.

With merge model, and the bot, this situation also happened regularly.
Like my current attempt to have a qt5 5.12->dev merge plus a qt5 dev
submodule update,

* https://codereview.qt-project.org/#/c/249863/
* https://codereview.qt-project.org/#/c/250191/

* https://bugreports.qt.io/browse/QTBUG-72953
* https://bugreports.qt.io/browse/QTBUG-73224

Both related with changes in qtdeclarative 5.12(one is a fix for
QTBUG-72953, another is the source of QTBUG-73224), but a 5.12->dev
merge will bring them together to dev, I need to wait for a totally
fixed 5.12 for current qt5 dev integration, including fix in
qtwebengine and other modules. The current estimated time is 3 days or
a week.

In the "cherry-pick" model, we can think every other branches than dev
are in locked mode, controlled by release team or module maintainers.
The developers of patches will not be involved if "cherry-pick" bot
doesn't find conflicts and CI/COIN approved it.

The amount of total conflicts will be same in both models. The
difference is who is reponsible to fix it. I am the center point to
fix them currently, but it's very difficult to know the situation in
different branches in the whole qt5 umbrella. With new cherry-pick
mode, the owner of the change will be notified if conflicts happened.
I suggest it should also notify the module maintainer and release
team. Those people are more responsible than anyone else to have the
change happened in the target branches. And it will not block the
development in dev.

In new model, perhaps a merge monkey is not needed any more, but a
cherry-pick monkey will happen. And with the direct help from the
owner of the change, and not blocking dev branch, I consider it as
less pressure personally.

My concern is about "cherry-pick" a series of changes for a big
feature, especially during the period to have "dev" branches for both
5 and 6. I don't have solution for this issue yet.

Best regards,
Liang
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development