Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

Am 07.12.2017 17:18 schrieb "Emil Velikov" :

On 6 December 2017 at 12:37, Daniel Martin  wrote:
> Hi all,
>
> if anyone would like to have a look, I've pushed my current work on
> the merged proto repo here:
> https://github.com/bartsch/xorg-proto2k/
> It's generated as is with:
> https://github.com/bartsch/proto2k-generator/
>
> I used git-filter-branch to:
> - move files to specific directories and
> - prefix commit subjects with the proto name.
>
> Lots of files have been moved to tmp/, which was removed at the end.
> See commit "Remove temporary files (tmp/)".
>
> Atm, it just installs:
> - headers (+handling Xfuncproto.h and Xpoll.h) and
> - pkg-configs for each proto (haven't found any notable diff'erences
> to my distro pkg-configs)
>
> Now:
> - What should I do with the {AUTHORS,COPYING}.$proto files?
> - Should I remove some protos (i.e. trapproto)? The current list is:
> https://github.com/bartsch/xorg-proto2k/commit/
8b06f111f99c2f66a9327ef669ee17c530e7
> - Anything else I should change?
>
> Next:
> - I'm working on generating the specs.
>
Silly question: why are you moving things around?


I did that to reflect the installation hierarchy (at least for the headers)
and filter out files (i.e. autofoo and Makefiles) we don't need later.

Perhaps I misread something, but it seemed that devs prefer to have
the different repos in separate folders.
Aka something like
for repo in $repos; do
mv .../old/to/old/repo/$repo .../new/to/newrepo/$repo
done


I had the impression that this was the preferred way with git-subtree as it
doesn't allow us anything else without conflicts.

The repo you're prepared looks quite messy IMHO.


Can you be more specific, please? The mixed header files?

I can and will change stuff. I just need someone saying: We would like to
have it like ...
Though, not this week as I've a full time RL task atm..

Cheers,
Daniel
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Emil Velikov]

On 6 December 2017 at 12:37, Daniel Martin  wrote:
> Hi all,
>
> if anyone would like to have a look, I've pushed my current work on
> the merged proto repo here:
> https://github.com/bartsch/xorg-proto2k/
> It's generated as is with:
> https://github.com/bartsch/proto2k-generator/
>
> I used git-filter-branch to:
> - move files to specific directories and
> - prefix commit subjects with the proto name.
>
> Lots of files have been moved to tmp/, which was removed at the end.
> See commit "Remove temporary files (tmp/)".
>
> Atm, it just installs:
> - headers (+handling Xfuncproto.h and Xpoll.h) and
> - pkg-configs for each proto (haven't found any notable diff'erences
> to my distro pkg-configs)
>
> Now:
> - What should I do with the {AUTHORS,COPYING}.$proto files?
> - Should I remove some protos (i.e. trapproto)? The current list is:
> 
> https://github.com/bartsch/xorg-proto2k/commit/8b06f111f99c2f66a9327ef669ee17c530e7
> - Anything else I should change?
>
> Next:
> - I'm working on generating the specs.
>
Silly question: why are you moving things around?

Perhaps I misread something, but it seemed that devs prefer to have
the different repos in separate folders.
Aka something like
for repo in $repos; do
mv .../old/to/old/repo/$repo .../new/to/newrepo/$repo
done

The repo you're prepared looks quite messy IMHO.

-Emil
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 6 December 2017 at 16:23, Gioele Barabucci  wrote:
> Hi,
>
> 06.12.2017 13:37 Daniel Martin:
>>
>> PS: Just talked to Peter, he's okay with filter-branch as it gives us
>> git-log without a struggle and references to other commits can be
>> looked up in the old repos
>
> A suggestion: you could add the old IDs into the new commit logs. This makes
> it possible to search for old commits with a simple grep.
>
> You can do this with a `--msg-filter` pass like the following one. This
> filter must be run before `filter_branch_prefix_subject`.
>
> git filter-branch --msg-filter '
> cat && echo && echo "Original-Commit-ID: $GIT_COMMIT"
> ' $branch

Thanks for the tip. Added.
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Gioele Barabucci]

Hi,

06.12.2017 13:37 Daniel Martin:

PS: Just talked to Peter, he's okay with filter-branch as it gives us
git-log without a struggle and references to other commits can be
looked up in the old repos
A suggestion: you could add the old IDs into the new commit logs. This 
makes it possible to search for old commits with a simple grep.


You can do this with a `--msg-filter` pass like the following one. This 
filter must be run before `filter_branch_prefix_subject`.


git filter-branch --msg-filter '
cat && echo && echo "Original-Commit-ID: $GIT_COMMIT"
' $branch

Regards,

--
Gioele Barabucci 
http://gioele.io

___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

Hi all,

if anyone would like to have a look, I've pushed my current work on
the merged proto repo here:
https://github.com/bartsch/xorg-proto2k/
It's generated as is with:
https://github.com/bartsch/proto2k-generator/

I used git-filter-branch to:
- move files to specific directories and
- prefix commit subjects with the proto name.

Lots of files have been moved to tmp/, which was removed at the end.
See commit "Remove temporary files (tmp/)".

Atm, it just installs:
- headers (+handling Xfuncproto.h and Xpoll.h) and
- pkg-configs for each proto (haven't found any notable diff'erences
to my distro pkg-configs)

Now:
- What should I do with the {AUTHORS,COPYING}.$proto files?
- Should I remove some protos (i.e. trapproto)? The current list is:

https://github.com/bartsch/xorg-proto2k/commit/8b06f111f99c2f66a9327ef669ee17c530e7
- Anything else I should change?

Next:
- I'm working on generating the specs.


Cheers,
Daniel

PS: Just talked to Peter, he's okay with filter-branch as it gives us
git-log without a struggle and references to other commits can be
looked up in the old repos.
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 30 November 2017 at 14:28, Emil Velikov  wrote:
> On 29 November 2017 at 13:40, Daniel Martin  wrote:
>> On 28 November 2017 at 16:51, Emil Velikov  wrote:
>>> One reasonably easy way to fold the repositories [while preserving all
>>> the history] is via git filter-branch.
>>
>> Yes, but, filter-branch rewrites the history too. As Peter stated
>> earlier, this breaks commit messages referencing other commits.
>>
> Right. I did not know there are such references in the protos codebase.
> Seemingly are are a few :-\ Two ideas come to mind:
>  - amend + add reference to the original sha
>  - amend + adjust the sha to the new one

Tracking the old and new sha might be difficult.
Amending changes the history of following commits too. ;)

>>> One could even script the process ;-)
>>
>> PoC: https://github.com/bartsch/proto2k-generator/blob/master/generator.sh
>
> Not sure if one needs all the fancy stuff with ls-tree - as we want
> separate subdirs a simple mkdir + mv should suffice.

Only changing the subdirs is easy and as this changes the history
anyway I used it to test what's possible with filter-branch as I
haven't played with it yet, i.e.:
- add a $fooproto prefix to the subject
- remove files we may delete later anyway
- sort files into specific directories (specs -> specs/$fooproto/...,
.h -> include/X11/{fonts,extensions,...}

> Either way, I'm just sharing some ideas. It'll be up-to the core X
> devs to decide on the approach.

Me too. And if the decision is to f*ck up the history, I contributed
an example howto do it the hard way. ;)
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Emil Velikov]

On 29 November 2017 at 13:40, Daniel Martin  wrote:
> On 28 November 2017 at 16:51, Emil Velikov  wrote:
>> One reasonably easy way to fold the repositories [while preserving all
>> the history] is via git filter-branch.
>
> Yes, but, filter-branch rewrites the history too. As Peter stated
> earlier, this breaks commit messages referencing other commits.
>
Right. I did not know there are such references in the protos codebase.
Seemingly are are a few :-\ Two ideas come to mind:
 - amend + add reference to the original sha
 - amend + adjust the sha to the new one

>> See xserver commit 35dc7c75150733dbcef8a18b6796f49a7c48ebee which
>> imports modesetting.
>
> So, we have an example where we accepted to rewrite history.
>
AFAICT in that case there are no references, so it's a little safer ;-)


>> One could even script the process ;-)
>
> PoC: https://github.com/bartsch/proto2k-generator/blob/master/generator.sh

Not sure if one needs all the fancy stuff with ls-tree - as we want
separate subdirs a simple mkdir + mv should suffice.

Either way, I'm just sharing some ideas. It'll be up-to the core X
devs to decide on the approach.

HTH
Emil
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Peter Hutterer]

On Wed, Nov 29, 2017 at 03:44:53PM +0100, Daniel Martin wrote:
> On 29 November 2017 at 15:01, Daniel Stone  wrote:
> > Hi,
> >
> > On 28 November 2017 at 12:38, Daniel Martin  wrote:
> >> On 27 November 2017 at 23:18, Peter Hutterer  
> >> wrote:
> >>> have you looked at subtree merges? Keeps the repo history for each merged
> >>> repository and the one from the top-level repository. And it forces you to
> >>> merge into a directory anyway, so that's a win right there.
> >>
> >> Oh? No, didn't knew that before. But, tested it:
> >> $ git remote add -f randrproto
> >> git://anongit.freedesktop.org/xorg/proto/randrproto
> >> $ git subtree add --prefix=randrproto/ randrproto master
> >>
> >> The history in the toplevel looks good. But, log -p shows the original
> >> paths in the commits. Okay, it doesn't rewrite history. I can deal
> >> with that.
> >> Though,
> >> $ git log (--graph) randrproto/randr.h
> >> just shows one commit from the subtree add.
> >
> > git log --follow -- randrproto/randr.h
> 
> Nope, doesn't work, even if I'm in randrproto/ and omit the path.
> But, it works if I'm in the toplevel and omit the path. O.o?

it is a bit strange. We used the subtree merge to merge ratbagd into
libratbag and now I'm getting:

git log ratbagd/
  is the history up until the merge
cd ratbagd && git log . 
 is the history since the merge

Adding --follow seems to have no effect on either

> Imho git-subtree is not for merging repos. It exists to source some
> 3rd party code into the tree, be able to update it easily and
> preferably squash its history.

The history is there, it just seems that git log seems confused as to what
to do, apparently it can't correctly follow the files past the merge.

Cheers,
   Peter
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 29 November 2017 at 15:52, Pekka Paalanen  wrote:
> On Wed, 29 Nov 2017 15:44:53 +0100
> Daniel Martin  wrote:
>
>> On 29 November 2017 at 15:01, Daniel Stone  wrote:
>> > Hi,
>> >
>> > On 28 November 2017 at 12:38, Daniel Martin  
>> > wrote:
>> >> On 27 November 2017 at 23:18, Peter Hutterer  
>> >> wrote:
>> >>> have you looked at subtree merges? Keeps the repo history for each merged
>> >>> repository and the one from the top-level repository. And it forces you 
>> >>> to
>> >>> merge into a directory anyway, so that's a win right there.
>> >>
>> >> Oh? No, didn't knew that before. But, tested it:
>> >> $ git remote add -f randrproto
>> >> git://anongit.freedesktop.org/xorg/proto/randrproto
>> >> $ git subtree add --prefix=randrproto/ randrproto master
>> >>
>> >> The history in the toplevel looks good. But, log -p shows the original
>> >> paths in the commits. Okay, it doesn't rewrite history. I can deal
>> >> with that.
>> >> Though,
>> >> $ git log (--graph) randrproto/randr.h
>> >> just shows one commit from the subtree add.
>> >
>> > git log --follow -- randrproto/randr.h
>>
>> Nope, doesn't work, even if I'm in randrproto/ and omit the path.
>> But, it works if I'm in the toplevel and omit the path. O.o?
>
>--follow
>Continue listing the history of a file beyond renames (works
>only for a single file).
>
> You tried with a single file?

Yes. Just try:

mkdir test
cd test
git init && git commit --alow-empty -m "init"
git remote add -f randrproto git://anongit.freedesktop.org/xorg/proto/randrproto
git subtree add --prefix=randrproto/ randrproto master

# randr.h is where we want it:
ls randrproto/randr.h

# this works:
git log --follow -- randr.h

# doesn't work:
git log --follow -- randrproto/randr.h

# doesn't work:
cd randrproto
git log --follow -- randr.h

# git blame works in toplevel:
git blame randrproto/randr.h
# and in subdir:
cd randrproto
git blame randr.h
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Pekka Paalanen]

On Wed, 29 Nov 2017 15:44:53 +0100
Daniel Martin  wrote:

> On 29 November 2017 at 15:01, Daniel Stone  wrote:
> > Hi,
> >
> > On 28 November 2017 at 12:38, Daniel Martin  
> > wrote:  
> >> On 27 November 2017 at 23:18, Peter Hutterer  
> >> wrote:  
> >>> have you looked at subtree merges? Keeps the repo history for each merged
> >>> repository and the one from the top-level repository. And it forces you to
> >>> merge into a directory anyway, so that's a win right there.  
> >>
> >> Oh? No, didn't knew that before. But, tested it:
> >> $ git remote add -f randrproto
> >> git://anongit.freedesktop.org/xorg/proto/randrproto
> >> $ git subtree add --prefix=randrproto/ randrproto master
> >>
> >> The history in the toplevel looks good. But, log -p shows the original
> >> paths in the commits. Okay, it doesn't rewrite history. I can deal
> >> with that.
> >> Though,
> >> $ git log (--graph) randrproto/randr.h
> >> just shows one commit from the subtree add.  
> >
> > git log --follow -- randrproto/randr.h  
> 
> Nope, doesn't work, even if I'm in randrproto/ and omit the path.
> But, it works if I'm in the toplevel and omit the path. O.o?

   --follow
   Continue listing the history of a file beyond renames (works
   only for a single file).

You tried with a single file?


Thanks,
pq


pgpGys_oEpVGr.pgp
Description: OpenPGP digital signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 29 November 2017 at 15:01, Daniel Stone  wrote:
> Hi,
>
> On 28 November 2017 at 12:38, Daniel Martin  wrote:
>> On 27 November 2017 at 23:18, Peter Hutterer  
>> wrote:
>>> have you looked at subtree merges? Keeps the repo history for each merged
>>> repository and the one from the top-level repository. And it forces you to
>>> merge into a directory anyway, so that's a win right there.
>>
>> Oh? No, didn't knew that before. But, tested it:
>> $ git remote add -f randrproto
>> git://anongit.freedesktop.org/xorg/proto/randrproto
>> $ git subtree add --prefix=randrproto/ randrproto master
>>
>> The history in the toplevel looks good. But, log -p shows the original
>> paths in the commits. Okay, it doesn't rewrite history. I can deal
>> with that.
>> Though,
>> $ git log (--graph) randrproto/randr.h
>> just shows one commit from the subtree add.
>
> git log --follow -- randrproto/randr.h

Nope, doesn't work, even if I'm in randrproto/ and omit the path.
But, it works if I'm in the toplevel and omit the path. O.o?

Imho git-subtree is not for merging repos. It exists to source some
3rd party code into the tree, be able to update it easily and
preferably squash its history.
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Stone]

Hi,

On 28 November 2017 at 12:38, Daniel Martin  wrote:
> On 27 November 2017 at 23:18, Peter Hutterer  wrote:
>> have you looked at subtree merges? Keeps the repo history for each merged
>> repository and the one from the top-level repository. And it forces you to
>> merge into a directory anyway, so that's a win right there.
>
> Oh? No, didn't knew that before. But, tested it:
> $ git remote add -f randrproto
> git://anongit.freedesktop.org/xorg/proto/randrproto
> $ git subtree add --prefix=randrproto/ randrproto master
>
> The history in the toplevel looks good. But, log -p shows the original
> paths in the commits. Okay, it doesn't rewrite history. I can deal
> with that.
> Though,
> $ git log (--graph) randrproto/randr.h
> just shows one commit from the subtree add.

git log --follow -- randrproto/randr.h

Cheers,
Daniel
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 28 November 2017 at 16:51, Emil Velikov  wrote:
> One reasonably easy way to fold the repositories [while preserving all
> the history] is via git filter-branch.

Yes, but, filter-branch rewrites the history too. As Peter stated
earlier, this breaks commit messages referencing other commits.

> See xserver commit 35dc7c75150733dbcef8a18b6796f49a7c48ebee which
> imports modesetting.

So, we have an example where we accepted to rewrite history.

> One could even script the process ;-)

PoC: https://github.com/bartsch/proto2k-generator/blob/master/generator.sh
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Emil Velikov]

On 28 November 2017 at 12:38, Daniel Martin  wrote:
> On 27 November 2017 at 23:18, Peter Hutterer  wrote:
>> On Mon, Nov 27, 2017 at 09:07:12PM +0100, Daniel Martin wrote:
>>> On 22 November 2017 at 07:45, Adam Jackson  wrote:
>>> > On Wed, 2017-11-22 at 10:25 +1000, Peter Hutterer wrote:
>>> >> On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
>>> >> > Your script splits each proto into a subdirectory, does it really make 
>>> >> > sense to
>>> >> > do that, or should the final proto package have everything together in 
>>> >> > the root?
>>> >>
>>> >> please no! if you merge all repos the history will be messy. With 
>>> >> subdirectories
>>> >> at least you get a nice git log for each individual repo if you specify 
>>> >> the
>>> >> directory name.
>>> >
>>> > Agreed, that'd be a useful property to preserve given how random the
>>> > repo and file names are.
>>>
>>> Just started to test an idea today, to get an even better history. The
>>> idea is basically (omitting details):
>>> 1. format-patch each commit into a patch, the filename includes the
>>> commit time (UTC)
>>>- patches of all repos become sorted and when applying them we get
>>> a nice history
>>> 2. hammer with sed on the patches
>>>- add a proto prefix to the subject
>>>- change initial position of files (i.e. move headers into
>>> include/X11/..., txts into doc/), no need to move files later
>>
>> if there are any commit messages that reference others ("Reverts commit
>> abcd123") you lose the reference because the sha is now different. so I'd
>> rather not do a re-commit of everything.
>>
>>> Surprisingly, this works pretty good for the first commits (reduced
>>> testing surface for the beginning). Last stop was at renames of files.
>>> But, I got another idea, which I can test tomorrow.
>>>
>>> The merge into subdirs looks straight forward and we can do that if I
>>> find a show-stopper. Deal?
>>
>> have you looked at subtree merges? Keeps the repo history for each merged
>> repository and the one from the top-level repository. And it forces you to
>> merge into a directory anyway, so that's a win right there.
>
> Oh? No, didn't knew that before. But, tested it:
> $ git remote add -f randrproto
> git://anongit.freedesktop.org/xorg/proto/randrproto
> $ git subtree add --prefix=randrproto/ randrproto master
>
> The history in the toplevel looks good. But, log -p shows the original
> paths in the commits. Okay, it doesn't rewrite history. I can deal
> with that.
> Though,
> $ git log (--graph) randrproto/randr.h
> just shows one commit from the subtree add.
>
One reasonably easy way to fold the repositories [while preserving all
the history] is via git filter-branch.
See xserver commit 35dc7c75150733dbcef8a18b6796f49a7c48ebee which
imports modesetting.

One could even script the process ;-)
Emil
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 27 November 2017 at 23:18, Peter Hutterer  wrote:
> On Mon, Nov 27, 2017 at 09:07:12PM +0100, Daniel Martin wrote:
>> On 22 November 2017 at 07:45, Adam Jackson  wrote:
>> > On Wed, 2017-11-22 at 10:25 +1000, Peter Hutterer wrote:
>> >> On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
>> >> > Your script splits each proto into a subdirectory, does it really make 
>> >> > sense to
>> >> > do that, or should the final proto package have everything together in 
>> >> > the root?
>> >>
>> >> please no! if you merge all repos the history will be messy. With 
>> >> subdirectories
>> >> at least you get a nice git log for each individual repo if you specify 
>> >> the
>> >> directory name.
>> >
>> > Agreed, that'd be a useful property to preserve given how random the
>> > repo and file names are.
>>
>> Just started to test an idea today, to get an even better history. The
>> idea is basically (omitting details):
>> 1. format-patch each commit into a patch, the filename includes the
>> commit time (UTC)
>>- patches of all repos become sorted and when applying them we get
>> a nice history
>> 2. hammer with sed on the patches
>>- add a proto prefix to the subject
>>- change initial position of files (i.e. move headers into
>> include/X11/..., txts into doc/), no need to move files later
>
> if there are any commit messages that reference others ("Reverts commit
> abcd123") you lose the reference because the sha is now different. so I'd
> rather not do a re-commit of everything.
>
>> Surprisingly, this works pretty good for the first commits (reduced
>> testing surface for the beginning). Last stop was at renames of files.
>> But, I got another idea, which I can test tomorrow.
>>
>> The merge into subdirs looks straight forward and we can do that if I
>> find a show-stopper. Deal?
>
> have you looked at subtree merges? Keeps the repo history for each merged
> repository and the one from the top-level repository. And it forces you to
> merge into a directory anyway, so that's a win right there.

Oh? No, didn't knew that before. But, tested it:
$ git remote add -f randrproto
git://anongit.freedesktop.org/xorg/proto/randrproto
$ git subtree add --prefix=randrproto/ randrproto master

The history in the toplevel looks good. But, log -p shows the original
paths in the commits. Okay, it doesn't rewrite history. I can deal
with that.
Though,
$ git log (--graph) randrproto/randr.h
just shows one commit from the subtree add.

Am I doing it wrong?
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Peter Hutterer]

On Mon, Nov 27, 2017 at 09:07:12PM +0100, Daniel Martin wrote:
> On 22 November 2017 at 07:45, Adam Jackson  wrote:
> > On Wed, 2017-11-22 at 10:25 +1000, Peter Hutterer wrote:
> >> On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
> >> > Your script splits each proto into a subdirectory, does it really make 
> >> > sense to
> >> > do that, or should the final proto package have everything together in 
> >> > the root?
> >>
> >> please no! if you merge all repos the history will be messy. With 
> >> subdirectories
> >> at least you get a nice git log for each individual repo if you specify the
> >> directory name.
> >
> > Agreed, that'd be a useful property to preserve given how random the
> > repo and file names are.
> 
> Just started to test an idea today, to get an even better history. The
> idea is basically (omitting details):
> 1. format-patch each commit into a patch, the filename includes the
> commit time (UTC)
>- patches of all repos become sorted and when applying them we get
> a nice history
> 2. hammer with sed on the patches
>- add a proto prefix to the subject
>- change initial position of files (i.e. move headers into
> include/X11/..., txts into doc/), no need to move files later

if there are any commit messages that reference others ("Reverts commit
abcd123") you lose the reference because the sha is now different. so I'd
rather not do a re-commit of everything.
 
> Surprisingly, this works pretty good for the first commits (reduced
> testing surface for the beginning). Last stop was at renames of files.
> But, I got another idea, which I can test tomorrow.
> 
> The merge into subdirs looks straight forward and we can do that if I
> find a show-stopper. Deal?

have you looked at subtree merges? Keeps the repo history for each merged
repository and the one from the top-level repository. And it forces you to
merge into a directory anyway, so that's a win right there.

Cheers,
   Peter
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Daniel Martin]

On 22 November 2017 at 07:45, Adam Jackson  wrote:
> On Wed, 2017-11-22 at 10:25 +1000, Peter Hutterer wrote:
>> On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
>> > Your script splits each proto into a subdirectory, does it really make 
>> > sense to
>> > do that, or should the final proto package have everything together in the 
>> > root?
>>
>> please no! if you merge all repos the history will be messy. With 
>> subdirectories
>> at least you get a nice git log for each individual repo if you specify the
>> directory name.
>
> Agreed, that'd be a useful property to preserve given how random the
> repo and file names are.

Just started to test an idea today, to get an even better history. The
idea is basically (omitting details):
1. format-patch each commit into a patch, the filename includes the
commit time (UTC)
   - patches of all repos become sorted and when applying them we get
a nice history
2. hammer with sed on the patches
   - add a proto prefix to the subject
   - change initial position of files (i.e. move headers into
include/X11/..., txts into doc/), no need to move files later

Surprisingly, this works pretty good for the first commits (reduced
testing surface for the beginning). Last stop was at renames of files.
But, I got another idea, which I can test tomorrow.

The merge into subdirs looks straight forward and we can do that if I
find a show-stopper. Deal?

Cheers,
Daniel
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Adam Jackson]

On Wed, 2017-11-22 at 10:25 +1000, Peter Hutterer wrote:
> On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
> > Your script splits each proto into a subdirectory, does it really make 
> > sense to
> > do that, or should the final proto package have everything together in the 
> > root?
> 
> please no! if you merge all repos the history will be messy. With 
> subdirectories
> at least you get a nice git log for each individual repo if you specify the
> directory name.

Agreed, that'd be a useful property to preserve given how random the
repo and file names are.

- ajax
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Peter Hutterer]

On Wed, Nov 22, 2017 at 12:50:16PM -0800, Dylan Baker wrote:
> Quoting Peter Hutterer (2017-11-21 16:25:47)
> > On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
> > > Quoting Keith Packard (2017-11-21 12:51:24)
> > > > Adam Jackson  writes:
> > > > 
> > > > > Also, git://people.freedesktop.org/~keithp/newproto appears to contain
> > > > > the script used to generate the merged repo.
> > > > 
> > > > Right, that's probably more useful today. The trick was to get the
> > > > headers merged without losing any of the history.
> > > > 
> > > > > I would be entirely in favor of merging the protocol header repos,
> > > > > fwiw. For that matter I'd be in favor of generating them from the xcb
> > > > > xml, but let's burn one bridge at a time.
> > > > 
> > > > Who wants to take another run at this wall?
> > > > 
> > > > -- 
> > > > -keith
> > > 
> > > Your script splits each proto into a subdirectory, does it really make 
> > > sense to
> > > do that, or should the final proto package have everything together in 
> > > the root?
> > 
> > please no! if you merge all repos the history will be messy. With 
> > subdirectories
> > at least you get a nice git log for each individual repo if you specify the
> > directory name.
> > 
> > Cheers,
> >Peter
> 
> With that in mind, does it still makes sense to merge the meson conversions 
> I've
> sent out, since the toplevel meson will likely simply be a bunch of `subdir()`
> calls?

that still works then, doesn't it? so I think I still makes sense.

Cheers,
   Peter

___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Dylan Baker]

Quoting Peter Hutterer (2017-11-21 16:25:47)
> On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
> > Quoting Keith Packard (2017-11-21 12:51:24)
> > > Adam Jackson  writes:
> > > 
> > > > Also, git://people.freedesktop.org/~keithp/newproto appears to contain
> > > > the script used to generate the merged repo.
> > > 
> > > Right, that's probably more useful today. The trick was to get the
> > > headers merged without losing any of the history.
> > > 
> > > > I would be entirely in favor of merging the protocol header repos,
> > > > fwiw. For that matter I'd be in favor of generating them from the xcb
> > > > xml, but let's burn one bridge at a time.
> > > 
> > > Who wants to take another run at this wall?
> > > 
> > > -- 
> > > -keith
> > 
> > Your script splits each proto into a subdirectory, does it really make 
> > sense to
> > do that, or should the final proto package have everything together in the 
> > root?
> 
> please no! if you merge all repos the history will be messy. With 
> subdirectories
> at least you get a nice git log for each individual repo if you specify the
> directory name.
> 
> Cheers,
>Peter

With that in mind, does it still makes sense to merge the meson conversions I've
sent out, since the toplevel meson will likely simply be a bunch of `subdir()`
calls?

Dylan


signature.asc
Description: signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Peter Hutterer]

On Tue, Nov 21, 2017 at 03:28:20PM -0800, Dylan Baker wrote:
> Quoting Keith Packard (2017-11-21 12:51:24)
> > Adam Jackson  writes:
> > 
> > > Also, git://people.freedesktop.org/~keithp/newproto appears to contain
> > > the script used to generate the merged repo.
> > 
> > Right, that's probably more useful today. The trick was to get the
> > headers merged without losing any of the history.
> > 
> > > I would be entirely in favor of merging the protocol header repos,
> > > fwiw. For that matter I'd be in favor of generating them from the xcb
> > > xml, but let's burn one bridge at a time.
> > 
> > Who wants to take another run at this wall?
> > 
> > -- 
> > -keith
> 
> Your script splits each proto into a subdirectory, does it really make sense 
> to
> do that, or should the final proto package have everything together in the 
> root?

please no! if you merge all repos the history will be messy. With subdirectories
at least you get a nice git log for each individual repo if you specify the
directory name.

Cheers,
   Peter
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Dylan Baker]

Quoting Keith Packard (2017-11-21 12:51:24)
> Adam Jackson  writes:
> 
> > Also, git://people.freedesktop.org/~keithp/newproto appears to contain
> > the script used to generate the merged repo.
> 
> Right, that's probably more useful today. The trick was to get the
> headers merged without losing any of the history.
> 
> > I would be entirely in favor of merging the protocol header repos,
> > fwiw. For that matter I'd be in favor of generating them from the xcb
> > xml, but let's burn one bridge at a time.
> 
> Who wants to take another run at this wall?
> 
> -- 
> -keith

Your script splits each proto into a subdirectory, does it really make sense to
do that, or should the final proto package have everything together in the root?

Dylan


signature.asc
Description: signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Keith Packard]

Adam Jackson  writes:

> Also, git://people.freedesktop.org/~keithp/newproto appears to contain
> the script used to generate the merged repo.

Right, that's probably more useful today. The trick was to get the
headers merged without losing any of the history.

> I would be entirely in favor of merging the protocol header repos,
> fwiw. For that matter I'd be in favor of generating them from the xcb
> xml, but let's burn one bridge at a time.

Who wants to take another run at this wall?

-- 
-keith


signature.asc
Description: PGP signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Adam Jackson]

On Tue, 2017-11-21 at 11:54 -0800, Keith Packard wrote:
> Daniel Martin  writes:
> 
> > Hi,
> > 
> > I've ever wondered why are the proto headers split up into distinct
> > repos? (It takes "ages" to just copy (install) a few files with
> > autotools.)
>
> They were split as part of the great dis-aggregation. I had a prototype
> of them merged back together, but that was never finished. I'm not sure
> what the status is at this point, but you might want to look at this:
> 
> git://people.freedesktop.org/~keithp/proto

Also, git://people.freedesktop.org/~keithp/newproto appears to contain
the script used to generate the merged repo.

I would be entirely in favor of merging the protocol header repos,
fwiw. For that matter I'd be in favor of generating them from the xcb
xml, but let's burn one bridge at a time.

- ajax
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Keith Packard]

Daniel Martin  writes:

> Hi,
>
> I've ever wondered why are the proto headers split up into distinct
> repos? (It takes "ages" to just copy (install) a few files with
> autotools.)

They were split as part of the great dis-aggregation. I had a prototype
of them merged back together, but that was never finished. I'm not sure
what the status is at this point, but you might want to look at this:

git://people.freedesktop.org/~keithp/proto

> Dylan started to add meson support, but doing this for all protos
> sounds like a lot of copy'n'paste to do, lots of patches . Wouldn't it
> make more sense in general to merge all proto repos into one (meson
> only) and have options if you don't want a specific proto to be
> installed as we do with xcb?

I can't imagine why you'd want to skip any particular protocol headers
for install; they're not large. I guess some package build may assume
that the presence of a proto header implies the presence of a library?

-- 
-keith


signature.asc
Description: PGP signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Dylan Baker]

Quoting Daniel Martin (2017-11-21 01:59:42)
> Hi,
> 
> I've ever wondered why are the proto headers split up into distinct
> repos? (It takes "ages" to just copy (install) a few files with
> autotools.)
> 
> Dylan started to add meson support, but doing this for all protos
> sounds like a lot of copy'n'paste to do, lots of patches . Wouldn't it
> make more sense in general to merge all proto repos into one (meson
> only) and have options if you don't want a specific proto to be
> installed as we do with xcb?
> 
> Cheers,
> Daniel
> 
> PS: A quick search revealed that a merge is possible even with keeping
> the history.

For reference here's the proto packages that archlinux provides:
bigreqsproto
compositeproto
damageproto
dmxproto
dri2proto
dri3proto
fixesproto
fontsproto
glproto
inputproto
kbproto
libxdmcp
presentproto
printproto
randrproto
recordproto
renderproto
resourceproto
scrnsaverproto
videoproto
xcmiscproto
xextproto
xf86dgaproto
xf86driproto
xf86vidmodeproto
xineramaproto


signature.asc
Description: signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: Merged repo for protocol headers? Why are they split?

[Dylan Baker]

I've been doing what I need for mesa on platforms that lack proper package/port
management (aka, windows and macos). AFAIK it should be completely possible to
merge all the protos into a single xorg-proto (or whatever color you like your
bikeshed), but no one's stepped up to take on that project.

There are a couple of questions I have about a merged proto repo, namely some of
the protos still have options for handling very old OSes (some or all of which
are not in common use anymore) and some of the protos themselves are not in
common use anymore. Do we want to add the obsolete protos and do we want to drop
options for some of the more archaic OSes?

Dylan

Quoting Daniel Martin (2017-11-21 01:59:42)
> Hi,
> 
> I've ever wondered why are the proto headers split up into distinct
> repos? (It takes "ages" to just copy (install) a few files with
> autotools.)
> 
> Dylan started to add meson support, but doing this for all protos
> sounds like a lot of copy'n'paste to do, lots of patches . Wouldn't it
> make more sense in general to merge all proto repos into one (meson
> only) and have options if you don't want a specific proto to be
> installed as we do with xcb?
> 
> Cheers,
> Daniel
> 
> PS: A quick search revealed that a merge is possible even with keeping
> the history.


signature.asc
Description: signature
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Merged repo for protocol headers? Why are they split?

[Daniel Martin]

Hi,

I've ever wondered why are the proto headers split up into distinct
repos? (It takes "ages" to just copy (install) a few files with
autotools.)

Dylan started to add meson support, but doing this for all protos
sounds like a lot of copy'n'paste to do, lots of patches . Wouldn't it
make more sense in general to merge all proto repos into one (meson
only) and have options if you don't want a specific proto to be
installed as we do with xcb?

Cheers,
Daniel

PS: A quick search revealed that a merge is possible even with keeping
the history.
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel