Re: How does one package a multirepo project?
Dear DDs/DMs, If you want to follow this thread, look at https://bugs.debian.org/972608 Cheers, Xavier
Re: How does one package a multirepo project?
Quoting Julien Puydt (2020-10-20 18:40:10) > Le mardi 20 octobre 2020 à 18:00 +0200, Jonas Smedegaard a écrit : > > > > - bad: the tarball contains the whole sources (no repack) ; > > > > - bad: the bare checkout isn't cleared. > > > > > > Still current. > > > > Please file bugreports about concrete weaknesses of Debian tools. > > > > If you are still asking a question here, then I recommend to make > > that more explicit. > > The lack of repack was because I had empty lines ; this problem is > fixed, and was coming from me. > > The fact that the bare checkout stays after uscan has run is probably > a bug. Is it supposed to stay around? Please treat "probably a bug" as a bug and file a bugreport. That's more efficient than asking thousands¹ of developers for their opinion before filing it. - Jonas ¹ ...assuming a reasonable fraction of human subscribers + human readers/googlers/recipients of non-human proxy subscribers are developers: https://lists.debian.org/stats/debian-devel.png -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How does one package a multirepo project?
Le 20/10/2020 à 18:40, Julien Puydt a écrit : > Le mardi 20 octobre 2020 à 18:00 +0200, Jonas Smedegaard a écrit : - bad: the tarball contains the whole sources (no repack) ; - bad: the bare checkout isn't cleared. >>> >>> Still current. >> >> Please file bugreports about concrete weaknesses of Debian tools. >> >> If you are still asking a question here, then I recommend to make >> that more explicit. > > The lack of repack was because I had empty lines ; this problem is > fixed, and was coming from me. > > The fact that the bare checkout stays after uscan has run is probably a > bug. Is it supposed to stay around? Yes it's seems to be a bug
Re: How does one package a multirepo project?
Le mardi 20 octobre 2020 à 18:00 +0200, Jonas Smedegaard a écrit : > > > - bad: the tarball contains the whole sources (no repack) ; > > > - bad: the bare checkout isn't cleared. > > > > Still current. > > Please file bugreports about concrete weaknesses of Debian tools. > > If you are still asking a question here, then I recommend to make > that more explicit. The lack of repack was because I had empty lines ; this problem is fixed, and was coming from me. The fact that the bare checkout stays after uscan has run is probably a bug. Is it supposed to stay around? JP
Re: How does one package a multirepo project?
Le lundi 19 octobre 2020 à 10:06 +0200, Jonas Smedegaard a écrit : > You might find some inspiration in the source package > jsbundle-web-interfaces which uses version type "group" and > mode=git, > and sets individual version numbers for each binary package. > > Another example is matrix-mirage where one component is fixed at a > specific release. I tried quite a few things today, and didn't get anywhere useful. If I use as d/watch: version=4 opts=component=algorithm,mode=git,\ uversionmangle=s/(.*)/algorithm-$1/ \ https://github.com/jupyterlab/lumino \ refs/tags/@lumino/algorithm@@ANY_VERSION@ group then I get a lumino_algorithm-1.3.3.orig-algorithm.tar.xz and its content looks good. But if my d/watch lists the first two components I'm interested in : version=4 opts=component=algorithm,mode=git,\ uversionmangle=s/(.*)/algorithm-$1/ \ https://github.com/jupyterlab/lumino \ refs/tags/@lumino/algorithm@@ANY_VERSION@ group opts=component=application,mode=git,\ uversionmangle=s/(.*)/application-$1/ \ https://github.com/jupyterlab/lumino \ refs/tags/@lumino/application@@ANY_VERSION@ group then I get the first tarball right, but then uscan dies : Successfully repacked ../lumino-algorithm-1.3.3.tar.xz as ../lumino_algorithm-1.3.3.orig-algorithm.tar.xz, deleting 443 files from it. fatal: not a valid object name: refs/tags/@lumino/application@1.11.0 uscan die: git archive failed at /usr/share/perl5/Devscripts/Uscan/Output.pm line 58. and if in the two-components d/watch I s/application/coreutils/g, I *do* get two tarballs! Notice that in any case the git bare repository has a name of the form lumino-temporary..git, where it looks like the number is increasing with time. That is probably (I'm not sure) bad : not all of the components need the same thing, and in the triple component- version-commit, for example, two components could have the same version and still need different commits! I'm still trying to know if the problem is sitting on the chair (I'm a bit tired these days) or if the tool is not up to the job. In any case : help! I need somebody. Help! Not just anybody... JP
Re: How does one package a multirepo project?
Quoting Julien Puydt (2020-10-20 16:14:17) > Le mardi 20 octobre 2020 à 10:02 +0200, Julien Puydt a écrit : > > Hi, > > > > Le lundi 19 octobre 2020 à 20:03 +, PICCA Frederic-Emmanuel a > > écrit : > > > what about the git mode of uscan > > > > > > then you would have all the tags ? > > > > Ah, yes, I had forgotten that! > > > > I tried with a simpler version (one component), and my d/watch is : > > > > version=4 > > > > opts=\ > > component=algorithm,\ > > dversionmangle=auto,repacksuffix=+ds,mode=git,compression=gzip,\ > > filenamemangle=s/lumino-(@ANY_VERSION@@ARCHIVE_EXT@)/lumino- > > algorithm- > > $1/ \ > > https://github.com/jupyterlab/lumino \ > > refs/tags/@lumino/algorithm@@ANY_VERSION@ group > > > > with d/copyright having (on a single line) : > > > > Files-Excluded-algorithm: api-extractor-base.json CONTRIBUTING.md > > examples lerna.json notebooks package.json RELEASE.md review > > yarn.lock > > packages/application packages/collections packages/commands > > packages/coreutils packages/datagrid packages/datastore > > packages/default-theme packages/disposable packages/domutils > > packages/dragdrop packages/keyboard packages/messaging > > packages/polling > > packages/properties packages/signaling packages/virtualdom > > packages/widgets > > > > The result is : > > > - bad: the name lumino-1.3.3.tar.gz for the tarball is not correct ; > > uversionmangle=s/(.*)/algorithm-$1/,\ > > got me out of this! > > > - bad: the tarball contains the whole sources (no repack) ; > > - bad: the bare checkout isn't cleared. > > Still current. Please file bugreports about concrete weaknesses of Debian tools. If you are still asking a question here, then I recommend to make that more explicit. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How does one package a multirepo project?
Le mardi 20 octobre 2020 à 10:02 +0200, Julien Puydt a écrit : > Hi, > > Le lundi 19 octobre 2020 à 20:03 +, PICCA Frederic-Emmanuel a > écrit : > > what about the git mode of uscan > > > > then you would have all the tags ? > > Ah, yes, I had forgotten that! > > I tried with a simpler version (one component), and my d/watch is : > > version=4 > > opts=\ > component=algorithm,\ > dversionmangle=auto,repacksuffix=+ds,mode=git,compression=gzip,\ > filenamemangle=s/lumino-(@ANY_VERSION@@ARCHIVE_EXT@)/lumino- > algorithm- > $1/ \ > https://github.com/jupyterlab/lumino \ > refs/tags/@lumino/algorithm@@ANY_VERSION@ group > > with d/copyright having (on a single line) : > > Files-Excluded-algorithm: api-extractor-base.json CONTRIBUTING.md > examples lerna.json notebooks package.json RELEASE.md review > yarn.lock > packages/application packages/collections packages/commands > packages/coreutils packages/datagrid packages/datastore > packages/default-theme packages/disposable packages/domutils > packages/dragdrop packages/keyboard packages/messaging > packages/polling > packages/properties packages/signaling packages/virtualdom > packages/widgets > > The result is : > - bad: the name lumino-1.3.3.tar.gz for the tarball is not correct ; uversionmangle=s/(.*)/algorithm-$1/,\ got me out of this! > - bad: the tarball contains the whole sources (no repack) ; > - bad: the bare checkout isn't cleared. Still current. JP
Re: How does one package a multirepo project?
Hi, Le lundi 19 octobre 2020 à 20:03 +, PICCA Frederic-Emmanuel a écrit : > what about the git mode of uscan > > then you would have all the tags ? Ah, yes, I had forgotten that! I tried with a simpler version (one component), and my d/watch is : version=4 opts=\ component=algorithm,\ dversionmangle=auto,repacksuffix=+ds,mode=git,compression=gzip,\ filenamemangle=s/lumino-(@ANY_VERSION@@ARCHIVE_EXT@)/lumino-algorithm- $1/ \ https://github.com/jupyterlab/lumino \ refs/tags/@lumino/algorithm@@ANY_VERSION@ group with d/copyright having (on a single line) : Files-Excluded-algorithm: api-extractor-base.json CONTRIBUTING.md examples lerna.json notebooks package.json RELEASE.md review yarn.lock packages/application packages/collections packages/commands packages/coreutils packages/datagrid packages/datastore packages/default-theme packages/disposable packages/domutils packages/dragdrop packages/keyboard packages/messaging packages/polling packages/properties packages/signaling packages/virtualdom packages/widgets The result is : - good: a bare checkout is made ; - good: a tarball gets generated ; - bad: the name lumino-1.3.3.tar.gz for the tarball is not correct ; - good: the symlink to lumino_1.3.3.orig-algorithm.tar.gz looks better - bad: the tarball contains the whole sources (no repack) ; - bad: the bare checkout isn't cleared. Here is the end of running "LANG=C uscan -v" : uscan info: Upstream URL(+tag) to download is identified as https://github.com/jupyterlab/lumino refs/tags/@lumino/algorithm@1.3.3 uscan info: Filename (filenamemangled) for downloaded file: lumino- 1.3.3.tar.gz uscan: Newest version of algorithm on remote site is 1.3.3, local version is 0~O uscan: => Newer package available from => https://github.com/jupyterlab/lumino refs/tags/@lumino/algorithm@1.3.3 uscan info: Downloading upstream package: lumino-1.3.3.tar.gz Cloning into bare repository '../lumino-temporary.277930.git'... remote: Enumerating objects: 442, done. remote: Counting objects: 100% (442/442), done. remote: Compressing objects: 100% (303/303), done. remote: Total 442 (delta 166), reused 258 (delta 112), pack-reused 0 Receiving objects: 100% (442/442), 614.72 KiB | 1.91 MiB/s, done. Resolving deltas: 100% (166/166), done. uscan info: Successfully downloaded package: lumino-1.3.3.tar.gz uscan info: Start checking for common possible upstream OpenPGP signature files uscan info: End checking for common possible upstream OpenPGP signature files uscan info: Missing OpenPGP signature. uscan info: New orig.tar.* tarball version (oversionmangled): 1.3.3 uscan info: Launch mk-origtargz with options: --package lumino --version 1.3.3 --repack-suffix +ds --component algorithm --compression gzip --directory .. --copyright-file debian/copyright ../lumino-1.3.3.tar.gz Successfully symlinked ../lumino-1.3.3.tar.gz to ../lumino_1.3.3.orig- algorithm.tar.gz. uscan info: New orig.tar.* tarball version (after mk-origtargz): 1.3.3 uscan warn: rename ../lumino_1.3.3.orig-algorithm.tar.gz to ../lumino_1.3.3.orig-algorithm.tar.gz Use of uninitialized value in string eq at /usr/share/perl5/Devscripts/Uscan/WatchFile.pm line 451. uscan info: Scan finished (I think that "Use of uninitialized value ..." is just a warning since the log doesn't end there) What do I still miss? JP
Re: How does one package a multirepo project?
Hi, Le lundi 19 octobre 2020 à 11:51 +0200, Julien Puydt a écrit : > Even in that case, will uscan see the tag subpackage42/3.14 on > github? > In my experience, it only sees a handful of last tags, so it will see > the releases of something like subpackage1/* to subpackage23/*, but > not the other ones. For example, it doesn't see the last ipywidgets > release 7.5.1's tag, so I had to download it by hand before calling > gbp import-orig. My experiments on uscan have given me the impression: (1) that if there are too many tags, some aren't seen (github only gives a list of ten by default) ; (2) that if one components appears tagless, nothing is done. Notice that it's possible to get more tags by (and that's what clicking "Next" does) : https://github.com/jupyterlab/lumino/tags?after=%40lumino%2Fapplication%401.11.0 so perhaps there's a trick to get more tags at once... Cheers, JP
Re: How does one package a multirepo project?
Hi, Le lundi 19 octobre 2020 à 20:33 +0200, Jonas Smedegaard a écrit : > Quoting Julien Puydt (2020-10-19 19:27:11) > > Am I getting somewhere with today's tools? Can someone propose a > > nicer way in the future? > > My apologies for mistaking the scope of your question. And my apologies for not being very clear in the first place. :-) Thanks, JP
Re: How does one package a multirepo project?
Quoting Julien Puydt (2020-10-19 19:27:11) > Am I getting somewhere with today's tools? Can someone propose a nicer > way in the future? My apologies for mistaking the scope of your question. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How does one package a multirepo project?
Hi,
Le lundi 19 octobre 2020 à 12:15 +0200, Jonas Smedegaard a écrit :
> Quoting Pirate Praveen (2020-10-19 12:01:47)
> > On 2020, ഒക്ടോബർ 19 12:45:28 PM IST, Julien Puydt
> > wrote:
> > > I was trying to update the ipywidgets package. It once had a
> > > quite
> > > normal upstream, but then things went wild, if not stellar :
> > > they
> > > went monorepo.
> > ...
> > > So basically my question is the one in the mail subject : how
> > > does
> > > one package a multirepo project?
> > >
> >
> > Look at node-rollup-plugin-* packages. The source packages will
> > have a
> > lot of duplication. I think uscan should provide an option to
> > include
> > only specific directories when repacking to make handling
> > monorepos
> > easier.
>
> To strip upstream content completely from Debian redistribution, use
> Files-Excluded-foo: in topmost section of debian/copyright - see
> jsbundle-web-interfaces for an example of that.
>
> To strip upstream content from entering the Debian git while still
> getting redistributed - notably to avoid upstream git hints from
> messing
> with a different use of git in Debian - use git-buildpackage and its
> --filter option. Again, jsbundle-web-interfaces is an example of
> that.
If I try to use the following script to update my d/copyright and
create a d/watch (I haven't any yet).
project = 'lumino'
github = f'https://github.com/jupyterlab/{project}'
packages = ['algorithm', 'application', 'collections', 'commands',
'coreutils', 'datagrid', 'datastore', 'default-theme',
'disposable', 'domutils', 'dragdrop', 'keyboard',
'messaging', 'polling', 'properties', 'signaling',
'virtualdom', 'widgets']
# In the real world, the following list of names is the result of:
#
# from os import listdir
# names = listdir('.')
#
# FIXME: since I don't have a main, where do I get CHANGELOG.md,
LICENSE.md and README.md from?
# WORKAROUND: edit by hand one of the components to let it get those
files!
#
names = ['api-extractor-base.json', 'CHANGELOG.md', 'CONTRIBUTING.md',
'examples',
'lerna.json', 'LICENSE', 'notebooks', 'package.json',
'packages',
'README.md', 'RELEASE.md', 'review' 'yarn.lock']
print('\n d/copyright\n==
==\n')
for package in packages:
chunks = []
for name in names:
if name != 'packages' and name != 'debian':
chunks.append(name)
for other in packages:
if other != package:
chunks.append(f'packages/{other}')
print(f'Files-Excluded-{package}: ' + ' '.join(chunks)+'\n')
print('\n d/watch\n
\n')
chunks = []
for package in packages:
chunk = 'opts=\\\n'
chunk = chunk + f'component={package},\\\n'
chunk = chunk +
'dversionmangle=auto,repacksuffix=+ds,compression=gzip,\\\n'
# FIXME: what if there are so many tags we don't see the relevant
one?
chunk = chunk +
f'filenamemangle=s/.*@(@ANY_VERSION@@ARCHIVE_EXT)/{project}-{package}-
$1/ \\\n'
chunk = chunk + f'{github}/tags \\\n'
chunk = chunk + f'.*/{package}@@ANY_VERSION@@ARCHIVE_EXT@ group\n'
chunks.append(chunk)
print('\n'.join(chunks))
It looks like uscan doesn't see the tag for the "algorithm" component,
as it isn't on the first page. And it looks like it doesn't try to look
for the other components. I tried to put "application" first, and now
it sees there's a newer package available, but I haven't found how to
get it to download. And of course after seeing there's a new
"application", it doesn't see "algorithm" and stops here.
Am I getting somewhere with today's tools? Can someone propose a nicer
way in the future?
Cheers,
JP
Re: How does one package a multirepo project?
Le lundi 19 octobre 2020 à 16:42 +0530, Pirate Praveen a écrit :
>
> On Mon, Oct 19, 2020 at 11:51, Julien Puydt
> wrote:
> > Here is what I have in d/control for my src:lumino experiments :
> >
> > Provides: node-lumino-algorithm (= 1.3.3),
> > node-lumino-application (= 1.11.0),
> > node-lumino-collections (= 1.3.3),
> > node-lumino-commands (= 1.11.3),
> > node-lumino-coreutils (= 1.5.3),
> > node-lumino-datagrid (= 1.14.0),
> > node-lumino-datastore (= 0.11.0),
> > node-lumino-default-theme (= 0.5.0),
> > node-lumino-disposable (= 1.4.3),
> > node-lumino-domutils (= 1.2.3),
> > node-lumino-dragdrop (= 1.6.4),
> > node-lumino-keyboard (= 1.2.3),
> > node-lumino-messaging (= 1.4.3),
> > node-lumino-polling (= 1.3.3),
> > node-lumino-properties (= 1.2.3),
> > node-lumino-signaling (= 1.4.3),
> > node-lumino-virtualdom (= 1.7.3),
> > node-lumino-widgets (= 1.14.0)
> >
> > can you imagine the size of the version string in d/changelog with
> > that
> > many parts?
> >
> > I'm pondering writing a small Python script to print a suitable
> > Provides: so I'll just have to paste its output and wrap-and-sort
> > to
> > update this...
>
> pkg-js-tools (dh-sequence-nodejs) can do this automatically. Use
> {nodejs:Provides}.
It assumes I use the uscan features for grouping ; and I'm not sure yet
it does the trick (needs a main source, if I followed correctly).
JP
Re: How does one package a multirepo project?
Quoting Pirate Praveen (2020-10-19 13:15:20) > On Mon, Oct 19, 2020 at 12:15, Jonas Smedegaard wrote: > > To strip upstream content completely from Debian redistribution, use > > Files-Excluded-foo: in topmost section of debian/copyright - see > > jsbundle-web-interfaces for an example of that. > > > > To strip upstream content from entering the Debian git while still > > getting redistributed - notably to avoid upstream git hints from > > messing with a different use of git in Debian - use git-buildpackage > > and its --filter option. Again, jsbundle-web-interfaces is an > > example of that. > > I'd consider that as a wrok around. When you have many directories and > you want to only include a single directory, then having to manually > exclude all the directories instead of specifying all the directories > we want to exclude is a lot of manual work and not future proof if > upstream add more directories. It'd be nice if uscan do the inverse > for looking at a list of directories to exclude and provide a list of > directories to include instead. I agree that that your idea would be a nice improvement to uscan, and I encourage you to file a wishlist bugreport against devscript to help make that happen. I understood the question as "How does one package a multirepo project *today*?" rather than "How does one package a multirepo project *in a possible future*?" (emphasis mine). - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How does one package a multirepo project?
Le 19/10/2020 à 12:01, Pirate Praveen a écrit : > > > On 2020, ഒക്ടോബർ 19 12:45:28 PM IST, Julien Puydt > wrote: >> Hi, >> >> I was trying to update the ipywidgets package. It once had a quite >> normal upstream, but then things went wild, if not stellar : they went >> monorepo. > ... >> So basically my question is the one in the mail subject : how does one >> package a multirepo project? >> > > Look at node-rollup-plugin-* packages. The source packages will have a lot of > duplication. I think uscan should provide an option to include only specific > directories when repacking to make handling monorepos easier. uscan has: * Files-Excluded * Files-Included (which overwrites Files-Excluded) Example: Files-Excluded-component1: lib/* Files-Included-component1: lib/foo/*
Re: How does one package a multirepo project?
On Mon, Oct 19, 2020 at 12:15, Jonas Smedegaard wrote: To strip upstream content completely from Debian redistribution, use Files-Excluded-foo: in topmost section of debian/copyright - see jsbundle-web-interfaces for an example of that. To strip upstream content from entering the Debian git while still getting redistributed - notably to avoid upstream git hints from messing with a different use of git in Debian - use git-buildpackage and its --filter option. Again, jsbundle-web-interfaces is an example of that. I'd consider that as a wrok around. When you have many directories and you want to only include a single directory, then having to manually exclude all the directories instead of specifying all the directories we want to exclude is a lot of manual work and not future proof if upstream add more directories. It'd be nice if uscan do the inverse for looking at a list of directories to exclude and provide a list of directories to include instead.
Re: How does one package a multirepo project?
On Mon, Oct 19, 2020 at 11:51, Julien Puydt
wrote:
Here is what I have in d/control for my src:lumino experiments :
Provides: node-lumino-algorithm (= 1.3.3),
node-lumino-application (= 1.11.0),
node-lumino-collections (= 1.3.3),
node-lumino-commands (= 1.11.3),
node-lumino-coreutils (= 1.5.3),
node-lumino-datagrid (= 1.14.0),
node-lumino-datastore (= 0.11.0),
node-lumino-default-theme (= 0.5.0),
node-lumino-disposable (= 1.4.3),
node-lumino-domutils (= 1.2.3),
node-lumino-dragdrop (= 1.6.4),
node-lumino-keyboard (= 1.2.3),
node-lumino-messaging (= 1.4.3),
node-lumino-polling (= 1.3.3),
node-lumino-properties (= 1.2.3),
node-lumino-signaling (= 1.4.3),
node-lumino-virtualdom (= 1.7.3),
node-lumino-widgets (= 1.14.0)
can you imagine the size of the version string in d/changelog with
that
many parts?
I'm pondering writing a small Python script to print a suitable
Provides: so I'll just have to paste its output and wrap-and-sort to
update this...
pkg-js-tools (dh-sequence-nodejs) can do this automatically. Use
{nodejs:Provides}.
Re: How does one package a multirepo project?
Quoting Pirate Praveen (2020-10-19 12:01:47) > On 2020, ഒക്ടോബർ 19 12:45:28 PM IST, Julien Puydt > wrote: > >I was trying to update the ipywidgets package. It once had a quite > >normal upstream, but then things went wild, if not stellar : they > >went monorepo. > ... > >So basically my question is the one in the mail subject : how does > >one package a multirepo project? > > > > Look at node-rollup-plugin-* packages. The source packages will have a > lot of duplication. I think uscan should provide an option to include > only specific directories when repacking to make handling monorepos > easier. To strip upstream content completely from Debian redistribution, use Files-Excluded-foo: in topmost section of debian/copyright - see jsbundle-web-interfaces for an example of that. To strip upstream content from entering the Debian git while still getting redistributed - notably to avoid upstream git hints from messing with a different use of git in Debian - use git-buildpackage and its --filter option. Again, jsbundle-web-interfaces is an example of that. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How does one package a multirepo project?
Quoting Julien Puydt (2020-10-19 11:51:05) > Le lundi 19 octobre 2020 à 10:06 +0200, Jonas Smedegaard a écrit : > > Quoting Julien Puydt (2020-10-19 09:15:28) > > > - uscan doesn't work correctly anymore, as the multiplication of > > > tags makes them disappear in the list quite fast ; > > > > Please see uscan v4 and its version types "group" and "checksum". > > I find it gives unwieldly versions when there is a lot of packages Please see uscan v4 version type "checksum" > > > - and what does one want to watch exactly anyway? > > > > Ideally we want to watch upstream releases of all upstream parts. > > > > If "upstream releases" are git commits, then watch that. > > Tagged commits, yes. Sounds good, but isn't ; see below. > > > If "upstream releases" are something more obscure like timestamps of > > files (yes, some do that!) then somehow watch that - or try convince > > upstream to also/instead use an easier watchable mechanism. > > The situation is worse than that : a same commit can be a release for > a directory, and give something bad for another. > > Imagine a project named "monorepo", with only two packages/directories > foo/ and bar/ and two tagged commits : > - 0xdeadbeef is tagged foo/3.14, and bar is broken for it ; > - 0x1337beef is tagged bar/2.72 and foo is broken for it. > > How can I get uscan version 4 to do anything sane about it? What you describe here is what I above call "something more obscure". In such situations I see these options: a) try convince upstream to also/instead release more straightforward. b) maintain _two_ source packages: project-foo and project-bar, both watching same upstream source but for each releasing at a different pace. There are more options - but it seems you are looking for easier-to-maintain ones. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How does one package a multirepo project?
On 2020, ഒക്ടോബർ 19 12:45:28 PM IST, Julien Puydt wrote: >Hi, > >I was trying to update the ipywidgets package. It once had a quite >normal upstream, but then things went wild, if not stellar : they went >monorepo. ... >So basically my question is the one in the mail subject : how does one >package a multirepo project? > Look at node-rollup-plugin-* packages. The source packages will have a lot of duplication. I think uscan should provide an option to include only specific directories when repacking to make handling monorepos easier. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: How does one package a multirepo project?
Hi, Le lundi 19 octobre 2020 à 10:06 +0200, Jonas Smedegaard a écrit : > Hi Julien, and others, > > Quoting Julien Puydt (2020-10-19 09:15:28) > > I was trying to update the ipywidgets package. It once had a quite > > normal upstream, but then things went wild, if not stellar : they > > went > > monorepo. > > > > For those lucky ones who never crossed the principle, the idea is > > to > > have a single repository, and make dozens of different packages > > live > > within. Basically, different directories now are different > > packages, > > with different release schedules. At the moment, > > https://github.com/jupyter-widgets/ipywidgets has 936 tags. > > > > There are several issues at hand : > > > > - uscan doesn't work correctly anymore, as the multiplication of > > tags > > makes them disappear in the list quite fast ; > > Please see uscan v4 and its version types "group" and "checksum". I find it gives unwieldly versions when there is a lot of packages (we're talking about a dozen packages here, with worse offenders probably around the corner) ; and we're left with : - an awfully long debian version string in debian/changelog ; - need to adapt by hand the versions of the Provides: in d/control for each sub-package. Here is what I have in d/control for my src:lumino experiments : Provides: node-lumino-algorithm (= 1.3.3), node-lumino-application (= 1.11.0), node-lumino-collections (= 1.3.3), node-lumino-commands (= 1.11.3), node-lumino-coreutils (= 1.5.3), node-lumino-datagrid (= 1.14.0), node-lumino-datastore (= 0.11.0), node-lumino-default-theme (= 0.5.0), node-lumino-disposable (= 1.4.3), node-lumino-domutils (= 1.2.3), node-lumino-dragdrop (= 1.6.4), node-lumino-keyboard (= 1.2.3), node-lumino-messaging (= 1.4.3), node-lumino-polling (= 1.3.3), node-lumino-properties (= 1.2.3), node-lumino-signaling (= 1.4.3), node-lumino-virtualdom (= 1.7.3), node-lumino-widgets (= 1.14.0) can you imagine the size of the version string in d/changelog with that many parts? I'm pondering writing a small Python script to print a suitable Provides: so I'll just have to paste its output and wrap-and-sort to update this... > If you are already aware of that, then perhaps by "doesn't work > correctly" you really mean "doesn't work same way as I am used to"? > > > - and what does one want to watch exactly anyway? > > Ideally we want to watch upstream releases of all upstream parts. > > If "upstream releases" are git commits, then watch that. Tagged commits, yes. Sounds good, but isn't ; see below. > If "upstream releases" are something more obscure like timestamps of > files (yes, some do that!) then somehow watch that - or try convince > upstream to also/instead use an easier watchable mechanism. The situation is worse than that : a same commit can be a release for a directory, and give something bad for another. Imagine a project named "monorepo", with only two packages/directories foo/ and bar/ and two tagged commits : - 0xdeadbeef is tagged foo/3.14, and bar is broken for it ; - 0x1337beef is tagged bar/2.72 and foo is broken for it. How can I get uscan version 4 to do anything sane about it? > > - even if uscan could keep up, how does one get the source? It > > should > > basically be a checkout of a different commit for each different > > directory! > > Please see uscan v4 and its mode=git. See above : the git mode is nice when there's a good global commit. I don't think that assumption is going to fly. > > - how does one even put a version number of this for a Debian > > package? > > (I guess something like "Provides: foo (= 3.14), bar (= 2.72)" can > > help a little, but what about the main package?) > > If main package is not versioned upstream, then I would say that's a > strong indication that you really want to track each underlying part > as a separate Debian package instead - i.e. that upstream "main > package" translates to a Debian metapackage. I don't understand what you mean here. A src:monorepo-something for each? With adapted d/copyright Files-Excluded: to get only the right directory? Even in that case, will uscan see the tag subpackage42/3.14 on github? In my experience, it only sees a handful of last tags, so it will see the releases of something like subpackage1/* to subpackage23/*, but not the other ones. For example, it doesn't see the last ipywidgets release 7.5.1's tag, so I had to download it by hand before calling gbp import- orig. > > PS: and to package the next ipywidgets, I started to work on lumino > > ( > > https://github.com/jupyterlab/lumino) with the same problem. In my > > experiments I numbered the main package 0~20200824+git93880412- > > 1... > > which is not ideal. > > Why is that not ideal? It is not ideal because : (1) I've just been lucky : they have
Re: How does one package a multirepo project?
Hi Julien, and others, Quoting Julien Puydt (2020-10-19 09:15:28) > I was trying to update the ipywidgets package. It once had a quite > normal upstream, but then things went wild, if not stellar : they went > monorepo. > > For those lucky ones who never crossed the principle, the idea is to > have a single repository, and make dozens of different packages live > within. Basically, different directories now are different packages, > with different release schedules. At the moment, > https://github.com/jupyter-widgets/ipywidgets has 936 tags. > > There are several issues at hand : > > - uscan doesn't work correctly anymore, as the multiplication of tags > makes them disappear in the list quite fast ; Please see uscan v4 and its version types "group" and "checksum". If you are already aware of that, then perhaps by "doesn't work correctly" you really mean "doesn't work same way as I am used to"? > - and what does one want to watch exactly anyway? Ideally we want to watch upstream releases of all upstream parts. If "upstream releases" are git commits, then watch that. If "upstream releases" are something more obscure like timestamps of files (yes, some do that!) then somehow watch that - or try convince upstream to also/instead use an easier watchable mechanism. > - even if uscan could keep up, how does one get the source? It should > basically be a checkout of a different commit for each different > directory! Please see uscan v4 and its mode=git. > - how does one even put a version number of this for a Debian package? > (I guess something like "Provides: foo (= 3.14), bar (= 2.72)" can > help a little, but what about the main package?) If main package is not versioned upstream, then I would say that's a strong indication that you really want to track each underlying part as a separate Debian package instead - i.e. that upstream "main package" translates to a Debian metapackage. > PS: and to package the next ipywidgets, I started to work on lumino ( > https://github.com/jupyterlab/lumino) with the same problem. In my > experiments I numbered the main package 0~20200824+git93880412-1... > which is not ideal. Why is that not ideal? I would drop the git part as it cannot be sorted (there is no guarantee that next git commit on same day leads to a higher version number). I.e. use 0~20200824-1 and if you come across needing to issue a second release on same day then use 0~20200824.1-1 for that. You might find some inspiration in the source package jsbundle-web-interfaces which uses version type "group" and mode=git, and sets individual version numbers for each binary package. Another example is matrix-mirage where one component is fixed at a specific release. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
