Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Nicholas D Steeves]

On Sat, Dec 29, 2018 at 04:51:49PM +1100, Dimitri John Ledkov wrote:
> On Sat, 29 Dec 2018 at 15:54, Nicholas D Steeves  wrote:
> >
> > Hi Alex, Cyril, Dimitri, and anyone else reading this,
> >
> > On Wed, Nov 28, 2018 at 08:41:18PM +0100, Alex Mestiashvili wrote:
> > >
> > > Hi Nicholas, it is in the new queue:
> > >
> > >  https://ftp-master.debian.org/new/libzstd_1.3.5+dfsg-2.html
> > >
> > > We just need to wait or ?
> >
> > I fear that waiting will put us too close to the freeze, and then it
> > will be an inappropriate time to make these changes.  I've asked
> > #ftp-masters on irc.oftc about the status of libzstd in NEW (2 months
> > and counting)
> >
> > Cyril and Dimitri, if either of you have a chance to ask an ftp-master
> > for feedback, and it wouldn't be too much of a bother, would you
> > please?
> 
> 
> I'm happy. From btrfs-progs point of view, it simply needs a binNMU to
> pick up and start using libzstd udeb which release team can schedule.

Brilliant! :-)  It looks like we're finally good to go.

I'll also ask the backports team if that (is_udeb_available)
conditional will now violate backports policy, eg: if it's a problem
that the the stretch-backports btrfs-progs will not have zstd support
when the buster one does.  I worry it might, and will follow up with a
btrfs-progs bug if it does.

Happy New Year!
Nicholas


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Nicholas D Steeves]

On Sat, Dec 29, 2018 at 04:53:02PM +0100, Cyril Brulebois wrote:
> Hi FTP team,
> 
> I've just been reminded (see below) of the zstd udeb addition currently
> sitting in NEW; the udeb addition was reviewed (even amended) and should
> be ready for use in other d-i components. Could you please let this
> package through? Thanks already!
> 
> Cheers,
> Cyril.

Thank you Cyril!  Yay, libzstd_1.3.5+dfsg-2 (with udeb) is in sid, and
has also migrated to testing!  I took a look at partman-btrfs, which
brings in the btrfs udeb.  Does partman-btrfs need to be modified to
also depend on the new zstd udeb, or will it be added as a recursive
dep when btrfs-progs is rebuilt?

Happy New Year!
Nicholas


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Cyril Brulebois]

Hi FTP team,

I've just been reminded (see below) of the zstd udeb addition currently
sitting in NEW; the udeb addition was reviewed (even amended) and should
be ready for use in other d-i components. Could you please let this
package through? Thanks already!

Cheers,
Cyril.


Nicholas D Steeves  (2018-12-28):
> Hi Alex, Cyril, Dimitri, and anyone else reading this,
> 
> On Wed, Nov 28, 2018 at 08:41:18PM +0100, Alex Mestiashvili wrote:
> > 
> > Hi Nicholas, it is in the new queue:
> > 
> >  https://ftp-master.debian.org/new/libzstd_1.3.5+dfsg-2.html
> > 
> > We just need to wait or ?
> 
> I fear that waiting will put us too close to the freeze, and then it
> will be an inappropriate time to make these changes.  I've asked
> #ftp-masters on irc.oftc about the status of libzstd in NEW (2 months
> and counting)
> 
> Cyril and Dimitri, if either of you have a chance to ask an ftp-master
> for feedback, and it wouldn't be too much of a bother, would you
> please?
> 
> Thanks!
> 
> Merry Christmas,
> Nicholas

-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Dimitri John Ledkov]

On Sat, 29 Dec 2018 at 15:54, Nicholas D Steeves  wrote:
>
> Hi Alex, Cyril, Dimitri, and anyone else reading this,
>
> On Wed, Nov 28, 2018 at 08:41:18PM +0100, Alex Mestiashvili wrote:
> >
> > Hi Nicholas, it is in the new queue:
> >
> >  https://ftp-master.debian.org/new/libzstd_1.3.5+dfsg-2.html
> >
> > We just need to wait or ?
>
> I fear that waiting will put us too close to the freeze, and then it
> will be an inappropriate time to make these changes.  I've asked
> #ftp-masters on irc.oftc about the status of libzstd in NEW (2 months
> and counting)
>
> Cyril and Dimitri, if either of you have a chance to ask an ftp-master
> for feedback, and it wouldn't be too much of a bother, would you
> please?


I'm happy. From btrfs-progs point of view, it simply needs a binNMU to
pick up and start using libzstd udeb which release team can schedule.

-- 
Regards,

Dimitri.

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Nicholas D Steeves]

Hi Alex, Cyril, Dimitri, and anyone else reading this,

On Wed, Nov 28, 2018 at 08:41:18PM +0100, Alex Mestiashvili wrote:
> 
> Hi Nicholas, it is in the new queue:
> 
>  https://ftp-master.debian.org/new/libzstd_1.3.5+dfsg-2.html
> 
> We just need to wait or ?

I fear that waiting will put us too close to the freeze, and then it
will be an inappropriate time to make these changes.  I've asked
#ftp-masters on irc.oftc about the status of libzstd in NEW (2 months
and counting)

Cyril and Dimitri, if either of you have a chance to ask an ftp-master
for feedback, and it wouldn't be too much of a bother, would you
please?

Thanks!

Merry Christmas,
Nicholas


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Alex Mestiashvili]

> It seems libzstd 1.3.5+dfsg-2 hasn't yet reached the archive.  Maybe
> it was not uploaded, or maybe it was rejected for some reason?
> 
>   
> https://salsa.debian.org/med-team/libzstd/commit/9b865b77d2bfc41c5865f255cf3e4aae18bbe934
> 
> Thanks you for working on this!
> Nicholas


Hi Nicholas, it is in the new queue:

 https://ftp-master.debian.org/new/libzstd_1.3.5+dfsg-2.html

We just need to wait or ?

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Nicholas D Steeves]

Hi Alex, Cyril, and anyone else reading this,

On Fri, Oct 12, 2018 at 10:09:48AM +0200, Cyril Brulebois wrote:
> Hi,
> 
> Alex Mestiashvili  (2018-10-12):
> > Fixed all the mentioned above issues in the repository.
> 
> That's looking good indeed.
> 
> Please note that by building a udeb you'll be subject to this once in a
> while:
> 
>   https://lists.debian.org/debian-devel-announce/2014/08/msg3.html
> 
> (That hasn't happened in a long while because I've been otherwise busy,
> but I still hope to resume regular d-i releases at some point.)
> 
> > Thank you for the detailed answer!
> 
> No problem, always easier/happier to catch such issues before packages
> reach the archive. ;)

It seems libzstd 1.3.5+dfsg-2 hasn't yet reached the archive.  Maybe
it was not uploaded, or maybe it was rejected for some reason?

  
https://salsa.debian.org/med-team/libzstd/commit/9b865b77d2bfc41c5865f255cf3e4aae18bbe934

Thanks you for working on this!
Nicholas


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Cyril Brulebois]

Hi,

Alex Mestiashvili  (2018-10-12):
> Fixed all the mentioned above issues in the repository.

That's looking good indeed.

Please note that by building a udeb you'll be subject to this once in a
while:

  https://lists.debian.org/debian-devel-announce/2014/08/msg3.html

(That hasn't happened in a long while because I've been otherwise busy,
but I still hope to resume regular d-i releases at some point.)

> Thank you for the detailed answer!

No problem, always easier/happier to catch such issues before packages
reach the archive. ;)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Alex Mestiashvili]

Hi,

On 10/11/2018 10:42 PM, Cyril Brulebois wrote:
> Hi,
> 
> And thanks for checking with us (explicit Cc's welcome, by the way).

Ok, will do in the future.

> 
> From a quick look at debian/control:
>> Package: libzstd1-udeb
>> Architecture: any
>> Multi-Arch: same
> 
> This field isn't needed for d-i.
> 
>> Section: debian-installer
>> Depends: ${misc:Depends},
>>  ${shlibs:Depends}
>> Package-Type: udeb
>> Description: fast lossless compression algorithm
>>  Zstd, short for Zstandard, is a fast lossless compression algorithm, 
>> targeting
>>  real-time compression scenarios at zlib-level compression ratio.
>>  .
>>  This package contains the shared library.
>>  .
>>  This is a minimal package for debian-installer.
> 
> FWIW: A single-line description is sufficient in a d-i context.
> 
> 
> From a quick look at debian/rules, a real issue:
>> override_dh_makeshlibs:
>> dh_makeshlibs -plibzstd1 -V'libzstd1 (>=1.3.5)' 
>> --add-udeb=libzstd-udeb
> 
> You're passing libzstd-udeb to --add-udeb, while the actual package is
> called libzstd1-udeb, so the generated shlibs file (see DEBIAN/shlibs in
> your build tree) will point to a package that doesn't exist, generating
> dependencies that can be satisfied for udebs using symbols from libzstd.
> 
> [I'm used to seeing a space after '>=' but apparently having no space
> seems supported as well:
> (sid-amd64-devel)kibi@armor:~/hack/libzstd$ grep '>=' 
> /var/lib/dpkg/info/*shlibs | grep -v '>= '
> /var/lib/dpkg/info/libgcrypt20:amd64.shlibs:libgcrypt 20 libgcrypt20 
> (>=1.8.0-0)
> /var/lib/dpkg/info/libgcrypt20:amd64.shlibs:udeb: libgcrypt 20 
> libgcrypt20-udeb (>=1.8.0-0)
> /var/lib/dpkg/info/libprocps6:amd64.shlibs:libprocps 6 libprocps6 
> (>=2:3.3.13-1)
> /var/lib/dpkg/info/libprocps7:amd64.shlibs:libprocps 7 libprocps7 
> (>=2:3.3.15-1)
> /var/lib/dpkg/info/libtasn1-6:amd64.shlibs:libtasn1 6 libtasn1-6 (>=4.7-0)
> ]
> 
> Other than that, the resulting udeb after a binary build looks OK to me.
> 
> 
> Cheers,
> 

Fixed all the mentioned above issues in the repository.
Thank you for the detailed answer!

Alex




signature.asc
Description: OpenPGP digital signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Cyril Brulebois]

Hi,

And thanks for checking with us (explicit Cc's welcome, by the way).

Alex Mestiashvili  (2018-10-11):
> On 10/09/2018 07:39 PM, Nicholas D Steeves wrote:
> > Control: retitle -1 please build libzstd udeb so that btrfs-progs can use 
> > zstd in Debian Installer
> > Control: reassign -1 src:libzstd/ 1.3.5+dfsg-1
> 
> Done, dropped zstd-udeb.
> 
> > Please read what Cyril (Debian Installer Team) wrote at these bugs in
> > case these questions have already been answered:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=898410
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886968
> 
> Thank you, that was useful.
> 
> I've updated the libzstd repository[0].
> 
> @Kibi, could you please review that?
> 
> basically I just call dh_makeshlibs with -V'libzstd1 (>=1.3.5)' as there
> are new symbols introduced in this version.

From a quick look at debian/control:
> Package: libzstd1-udeb
> Architecture: any
> Multi-Arch: same

This field isn't needed for d-i.

> Section: debian-installer
> Depends: ${misc:Depends},
>  ${shlibs:Depends}
> Package-Type: udeb
> Description: fast lossless compression algorithm
>  Zstd, short for Zstandard, is a fast lossless compression algorithm, 
> targeting
>  real-time compression scenarios at zlib-level compression ratio.
>  .
>  This package contains the shared library.
>  .
>  This is a minimal package for debian-installer.

FWIW: A single-line description is sufficient in a d-i context.


From a quick look at debian/rules, a real issue:
> override_dh_makeshlibs:
> dh_makeshlibs -plibzstd1 -V'libzstd1 (>=1.3.5)' 
> --add-udeb=libzstd-udeb

You're passing libzstd-udeb to --add-udeb, while the actual package is
called libzstd1-udeb, so the generated shlibs file (see DEBIAN/shlibs in
your build tree) will point to a package that doesn't exist, generating
dependencies that can be satisfied for udebs using symbols from libzstd.

[I'm used to seeing a space after '>=' but apparently having no space
seems supported as well:
(sid-amd64-devel)kibi@armor:~/hack/libzstd$ grep '>=' 
/var/lib/dpkg/info/*shlibs | grep -v '>= '
/var/lib/dpkg/info/libgcrypt20:amd64.shlibs:libgcrypt 20 libgcrypt20 (>=1.8.0-0)
/var/lib/dpkg/info/libgcrypt20:amd64.shlibs:udeb: libgcrypt 20 libgcrypt20-udeb 
(>=1.8.0-0)
/var/lib/dpkg/info/libprocps6:amd64.shlibs:libprocps 6 libprocps6 (>=2:3.3.13-1)
/var/lib/dpkg/info/libprocps7:amd64.shlibs:libprocps 7 libprocps7 (>=2:3.3.15-1)
/var/lib/dpkg/info/libtasn1-6:amd64.shlibs:libtasn1 6 libtasn1-6 (>=4.7-0)
]

Other than that, the resulting udeb after a binary build looks OK to me.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Alex Mestiashvili]

On 10/09/2018 07:39 PM, Nicholas D Steeves wrote:
> Control: retitle -1 please build libzstd udeb so that btrfs-progs can use 
> zstd in Debian Installer
> Control: reassign -1 src:libzstd/ 1.3.5+dfsg-1

Done, dropped zstd-udeb.

> 
> Please read what Cyril (Debian Installer Team) wrote at these bugs in
> case these questions have already been answered:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=898410
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886968

Thank you, that was useful.

I've updated the libzstd repository[0].

@Kibi, could you please review that?

basically I just call dh_makeshlibs with -V'libzstd1 (>=1.3.5)' as there
are new symbols introduced in this version.

Thank you,
Alex

[0] https://salsa.debian.org/med-team/libzstd/



signature.asc
Description: OpenPGP digital signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Nicholas D Steeves]

On Tue, Oct 09, 2018 at 08:41:35PM +, Holger Wansing wrote:
> 
>  
> > > Are there any udeb related docs available?
> > > 
> > 
> > Sorry, I don't know of any.
> 
> Maybe the d-i internals?
> https://d-i.debian.org/doc/internals/

Thank you Holger!  Yes, that's the one:

  https://d-i.debian.org/doc/internals/ch03.html

Bookmarked!

Cheers,
Nicholas


signature.asc
Description: PGP signature

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Holger Wansing]

 
> > Are there any udeb related docs available?
> > 
> 
> Sorry, I don't know of any.

Maybe the d-i internals?
https://d-i.debian.org/doc/internals/



-- 
Sent from my Jolla phone
http://www.jolla.com/

Bug#908834: please build libzstd udeb so that btrfs-progs can use zstd in Debian Installer

[Nicholas D Steeves]

Control: retitle -1 please build libzstd udeb so that btrfs-progs can use zstd 
in Debian Installer
Control: reassign -1 src:libzstd/ 1.3.5+dfsg-1

On Tue, Oct 09, 2018 at 06:24:47PM +0200, Alex Mestiashvili wrote:
> On 10/09/2018 04:32 PM, Dimitri John Ledkov wrote:
> >  is o
> > On Tue, 9 Oct 2018 at 12:23, Alex Mestiashvili  
> > wrote:
> >>
> >> On 09/14/2018 08:04 PM, Nicholas D Steeves wrote:
...
> >>>
> >>> Would you please build a zstd udeb so that btrfs-progs can use zstd in
> >>> Debian Installer and Rescue System?  It uses zstd for transparent
> >>> filesystem compression.
> >>>
> >>> eg: `chattr +c`, or `btrfs filesystem defrag -c`, or via a mount
> >>> option `compress=zstd`.  I believe the first and last of these use the
> >>> kernel's libzstd, and that the udeb is primarily required for
> >>> `btrfs-repair` to handle zstd extents in the Rescue System.  Also, please 
> >>> continue to CC Dmitri Ledkov, Debian's btrfs-progs maintainer.
> >>>
...
> >>
> >> As far as I see it's enough to add udeb stanzas in d/control in order to
> >> build the udebs[0].
> >> Is there anything else to consider before uploading lisbzstd with udebs?
> >>
> > 
> > No, that's not at all enough. It ends up creating two empty packages,
> > without any files in them.
> 
> Oh, I see. I thought there is some debhelper magic involved and didn't
> check the generated packages..
>

I'm also surprised it didn't run a second dh_install run with $DESTDIR
set to debian/udeb-package.  Or alternatively dh_install debian/udeb-package.

> > 
> > One needs to actually install a library into the library udeb and
> > tools into tools-udeb.
> > Note for fbtrfs only library-udeb is needed.
> 
> Does that also apply for btrfs-repair? Initial bug report is about zstd
> udeb as I see.

Sorry for the inaccuracy; I've retitled and reassigned this bug.
Initially I thought zstd was the name of the source package.

> > 
> > Also do get it reviewed, as last time unwritten rules w.r.t. udebs got
> > enforced and above patch was rejected on ground of not strict enough
> > alternative shlibs deps generated.
> 
> Thank you for clarifying, but I didn't understand the reason of reject :).

Ditto, me neither.  If I had to guess maybe ftpmasters want a manually
generated symbols file for libzstd and libzstd-udeb?

https://wiki.debian.org/UsingSymbolsFiles

> @debian-boot folks, please review and please either fix it or explain
> what is required.
>

Please read what Cyril (Debian Installer Team) wrote at these bugs in
case these questions have already been answered:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=898410
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886968

> Are there any udeb related docs available?
> 

Sorry, I don't know of any.

Kind regards,
Nicholas


signature.asc
Description: PGP signature