Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Ian Jackson]

Sean Whitton writes ("Bug#188731: debian-policy: "strip --strip-unneeded" is 
insufficient"):
> Indeed, but the shading will get a lot less useful if it merely
> indicates that there is /some/ common tool that implements the
> requirement.  We probably want to restrict the shading to dh-style
> debhelper.

I agree.  This would be really good.

But, a problem: what about the compat version ?  Some things may be
done only at certain compat levels.

I wouldn't like to see something done to policy which pushes people to
only use the latest released compat level.   So either the policy
shading (or whatever) needs to lag very substantially, or we need at
least two kinds of shading.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Sean Whitton]

Hello,

On Sun 28 Oct 2018 at 04:04PM GMT, Niels Thykier wrote:

>> Well, more specifically debhelper.
>>
>
> FTR, I would be happy if the solution was not specific to "debhelper" in
> the long term.  In particular, there is a world of difference between
> "debhelper", "cdbs with debhelper", "dh-style debhelper", and
> "dhmk-style debhelper".

Indeed, but the shading will get a lot less useful if it merely
indicates that there is /some/ common tool that implements the
requirement.  We probably want to restrict the shading to dh-style
debhelper.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Sean Whitton]

control: tag -1 -patch +pending

Hello,

On Sun 28 Oct 2018 at 03:51PM GMT, Niels Thykier wrote:

>>  .. [#]
>> -   You might also want to use the options ``--remove-section=.comment``
>> -   and ``--remove-section=.note`` on both shared libraries and
>> -   executables, and ``--strip-debug`` on static libraries.
>> +   You might also want to use the option ``--strip-debug`` on static
>> +   libraries.
>>
>
> For static libraries, you want to use --strip-debug *instead* of
> --strip-unneeded (as opposed to "also" as the text implies) AFAICT.  At
> least, debhelper always used --strip-debug without --strip-unneeded on
> static libraries.
>
> Note that for static libraries, it might be prudent to remind people to
> use "-D" / "--enable-deterministic-archives" to ensure a deterministic
> result (and thereby comply with the policy's recommendation to support
> reproducible builds).

I've updated the footnote.  Thanks.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Niels Thykier]

Sean Whitton:
> Hello Niels,
> 
> On Sun 28 Oct 2018 at 02:22PM GMT, Niels Thykier wrote:
> 
>> I think I agree with your suggestion of shading policy requirements that
>> are already covered by common tools.
> 
> Well, more specifically debhelper.
> 

FTR, I would be happy if the solution was not specific to "debhelper" in
the long term.  In particular, there is a world of difference between
"debhelper", "cdbs with debhelper", "dh-style debhelper", and
"dhmk-style debhelper".

>> At the moment, I am hesitant as to whether I should second Sean's text
>> as it is or point out that text fails to handle cross-building[1] (note
>> the cross building issue is not a regression).
>>   On one front, it would be technically correct - on the other front, I
>> fear most maintainers will be overloaded by irrelevant details that is
>> already handled for them.
> 
> At present Policy does not require crossbuilding support.  So while it
> would be nice to include details of how to properly support
> crossbuilding, it's not as though we're contradicting another
> requirement in Policy.  So it seems like including the crossbuilding
> details would be a separate bug.
> 

Indeed - and accordingly, it is not a regression. :)

Thanks,
~Niels

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Niels Thykier]

Sean Whitton:
> control: tag -1 +patch
> 
> Hello,
> 
> On Fri 28 Sep 2018 at 05:38AM GMT, Niels Thykier wrote:
> 
>>  * We now have auto-generated dbgsym packages by dh_strip (which were
>>just an idea when Bill wrote that answer).
>>
>>  * Policy mentions "--remove-section=.comment --remove-section=.note" in
>>a footnote as a "You might also want to use ...".
>>
>>  * Policy section 10.1 implies that "INSTALL = install -s" is a useful
>>way of stripping binaries (both in text and examples).  Besides not
>>covering the .comment + .note sections it also neuters dh_strip's
>>ability to create dbgsym packages.
>>
>>
>> I think we should update the policy to say that you should use
>> "--strip-unneeded --remove-section=.comment --remove-section=.note" and
>> maybe recommend delegating that task (where possible) to dh_strip as it
>> provide dbgsym packages.
> 
> Thank you for following up.
> 
> Here is a minimal patch, for which I am seeking seconds.  [...]
> 

Hi,

Seconded with one remark.

> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index 21c4c37..7106afe 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -48,12 +48,20 @@ used:
>  CC = gcc
>  CFLAGS = -O2 -g -Wall # sane warning options vary between programs
>  LDFLAGS = # none
> -INSTALL = install -s # (or use strip on the files in debian/tmp)
> 
> -Note that by default all installed binaries should be stripped, either
> -by using the ``-s`` flag to ``install``, or by calling ``strip`` on the
> -binaries after they have been copied into ``debian/tmp`` but before the
> -tree is made into a package.
> +By default all installed binaries should be stripped by calling
> +
> +::
> +
> +strip --strip-unneeded --remove-section=.comment --remove-section=.note 
> binaries
> +
> +on the binaries after they have been copied into ``debian/tmp`` but
> +before the tree is made into a package.
> +
> +It is not recommended to strip binaries by passing the ``-s`` flag to
> +``install``, because this fails to remove .comment and .note sections,
> +and also prevents the automatic creation of dbgsym binary packages by
> +tools like ``dh_strip``.
> 
>  Although binaries in the build tree should be compiled with debugging
>  information by default, it can often be difficult to debug programs if
> @@ -114,7 +122,7 @@ All installed shared libraries should be stripped with
> 
>  ::
> 
> -strip --strip-unneeded your-lib
> +strip --strip-unneeded --remove-section=.comment --remove-section=.note 
> your-lib
> 
>  (The option ``--strip-unneeded`` makes ``strip`` remove only the symbols
>  which aren't needed for relocation processing.) Shared libraries can
> @@ -123,7 +131,8 @@ linking are in a separate part of the ELF object file.  
> [#]_
> 
>  Note that under some circumstances it may be useful to install a shared
>  library unstripped, for example when building a separate package to
> -support debugging.
> +support debugging.  The debhelper `dh_strip`` tool can create such
> +packages automatically.
> 
>  Shared object files (often ``.so`` files) that are not public
>  libraries, that is, they are not meant to be linked to by third party
> @@ -741,9 +750,8 @@ restricted to ASCII when it is possible to do so.
> shared library, like ``mklibs`` does in the Debian installer project.
> 
>  .. [#]
> -   You might also want to use the options ``--remove-section=.comment``
> -   and ``--remove-section=.note`` on both shared libraries and
> -   executables, and ``--strip-debug`` on static libraries.
> +   You might also want to use the option ``--strip-debug`` on static
> +   libraries.
> 

For static libraries, you want to use --strip-debug *instead* of
--strip-unneeded (as opposed to "also" as the text implies) AFAICT.  At
least, debhelper always used --strip-debug without --strip-unneeded on
static libraries.

Note that for static libraries, it might be prudent to remind people to
use "-D" / "--enable-deterministic-archives" to ensure a deterministic
result (and thereby comply with the policy's recommendation to support
reproducible builds).

>  .. [#]
> A common example are the so-called "plug-ins", internal shared
> 
> =
> 
> To obtain a side-by-side diff:
> 
> % git clone salsa.debian.org:dbnpolicy/policy.git debian-policy
> % cd debian-policy
> % git difftool -y -x icdiff master..origin/bug188731-spwhitton
> 
> Alternatively, visit
> 
> https://salsa.debian.org/dbnpolicy/policy/commit/3cc86484767ac0aead9b7466c074ade5021ef225?view=parallel
> 

Thanks,
~Niels

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Sean Whitton]

Hello Niels,

On Sun 28 Oct 2018 at 02:22PM GMT, Niels Thykier wrote:

> I think I agree with your suggestion of shading policy requirements that
> are already covered by common tools.

Well, more specifically debhelper.

> At the moment, I am hesitant as to whether I should second Sean's text
> as it is or point out that text fails to handle cross-building[1] (note
> the cross building issue is not a regression).
>   On one front, it would be technically correct - on the other front, I
> fear most maintainers will be overloaded by irrelevant details that is
> already handled for them.

At present Policy does not require crossbuilding support.  So while it
would be nice to include details of how to properly support
crossbuilding, it's not as though we're contradicting another
requirement in Policy.  So it seems like including the crossbuilding
details would be a separate bug.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Niels Thykier]

Russ Allbery:
> Sean Whitton  writes:
> 
>> Thank you for following up.
> 
>> Here is a minimal patch, for which I am seeking seconds.  I didn't
>> include a recommendation to use dh_strip because we generally keep such
>> recommendations in footnotes rather than the text of Policy, and we are
>> trying to reduce the number of footnotes -- but I don't mind adding it
>> if others think it would be a good idea (and it wouldn't need
>> seconding).
> 
> Looks good to me.  Seconded.
> 
> On an entirely unrelated note (and probably a larger project than anyone
> will be able to tackle any time soon), I think it would be very helpful if
> we had some way of annotating what parts of Policy are just handled for
> you if you use debhelper in the normal way (perhaps with references to the
> tools involved).  Maybe with background shading or an icon or something?
> I think Policy needs to state all of the rules, including the ones
> implemented by other tools, but at the same time nearly everyone doing
> Debian packaging is using debhelper (either directly or indirectly), so
> it's a bit of a disservice to people to ask them to wade through a bunch
> of specifications for stuff that normally they don't and shouldn't think
> about.
> 

I think I agree with your suggestion of shading policy requirements that
are already covered by common tools.

At the moment, I am hesitant as to whether I should second Sean's text
as it is or point out that text fails to handle cross-building[1] (note
the cross building issue is not a regression).
  On one front, it would be technically correct - on the other front, I
fear most maintainers will be overloaded by irrelevant details that is
already handled for them.

Well, given it is not a regression, the choice is strictly speaking
"obvious" in this case but the general issue stands.

Thanks,
~Niels

[1] For cross build support, you would have to use
"$(DEB_HOST_GNU_TYPE)-strip" rather than "strip"... Except if you are
doing a "Canadian Cross" build in which case you *might* need
"$(DEB_TARGET_GNU_TYPE)-strip" instead in very rare cases.

Plus you have to ensure DEB_HOST_GNU_TYPE is defined - either by
including a dpkg makefile (recommended) or defining it yourself via
dpkg-buildflags which will imply $(shell ...).  We would like to avoid
the latter due to the high overhead involved in doing that.

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Russ Allbery]

Sean Whitton  writes:

> Thank you for following up.

> Here is a minimal patch, for which I am seeking seconds.  I didn't
> include a recommendation to use dh_strip because we generally keep such
> recommendations in footnotes rather than the text of Policy, and we are
> trying to reduce the number of footnotes -- but I don't mind adding it
> if others think it would be a good idea (and it wouldn't need
> seconding).

Looks good to me.  Seconded.

On an entirely unrelated note (and probably a larger project than anyone
will be able to tackle any time soon), I think it would be very helpful if
we had some way of annotating what parts of Policy are just handled for
you if you use debhelper in the normal way (perhaps with references to the
tools involved).  Maybe with background shading or an icon or something?
I think Policy needs to state all of the rules, including the ones
implemented by other tools, but at the same time nearly everyone doing
Debian packaging is using debhelper (either directly or indirectly), so
it's a bit of a disservice to people to ask them to wade through a bunch
of specifications for stuff that normally they don't and shouldn't think
about.

-- 
Russ Allbery (r...@debian.org)   

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Sean Whitton]

control: tag -1 +patch

Hello,

On Fri 28 Sep 2018 at 05:38AM GMT, Niels Thykier wrote:

>  * We now have auto-generated dbgsym packages by dh_strip (which were
>just an idea when Bill wrote that answer).
>
>  * Policy mentions "--remove-section=.comment --remove-section=.note" in
>a footnote as a "You might also want to use ...".
>
>  * Policy section 10.1 implies that "INSTALL = install -s" is a useful
>way of stripping binaries (both in text and examples).  Besides not
>covering the .comment + .note sections it also neuters dh_strip's
>ability to create dbgsym packages.
>
>
> I think we should update the policy to say that you should use
> "--strip-unneeded --remove-section=.comment --remove-section=.note" and
> maybe recommend delegating that task (where possible) to dh_strip as it
> provide dbgsym packages.

Thank you for following up.

Here is a minimal patch, for which I am seeking seconds.  I didn't
include a recommendation to use dh_strip because we generally keep such
recommendations in footnotes rather than the text of Policy, and we are
trying to reduce the number of footnotes -- but I don't mind adding it
if others think it would be a good idea (and it wouldn't need
seconding).

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index 21c4c37..7106afe 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -48,12 +48,20 @@ used:
 CC = gcc
 CFLAGS = -O2 -g -Wall # sane warning options vary between programs
 LDFLAGS = # none
-INSTALL = install -s # (or use strip on the files in debian/tmp)

-Note that by default all installed binaries should be stripped, either
-by using the ``-s`` flag to ``install``, or by calling ``strip`` on the
-binaries after they have been copied into ``debian/tmp`` but before the
-tree is made into a package.
+By default all installed binaries should be stripped by calling
+
+::
+
+strip --strip-unneeded --remove-section=.comment --remove-section=.note 
binaries
+
+on the binaries after they have been copied into ``debian/tmp`` but
+before the tree is made into a package.
+
+It is not recommended to strip binaries by passing the ``-s`` flag to
+``install``, because this fails to remove .comment and .note sections,
+and also prevents the automatic creation of dbgsym binary packages by
+tools like ``dh_strip``.

 Although binaries in the build tree should be compiled with debugging
 information by default, it can often be difficult to debug programs if
@@ -114,7 +122,7 @@ All installed shared libraries should be stripped with

 ::

-strip --strip-unneeded your-lib
+strip --strip-unneeded --remove-section=.comment --remove-section=.note 
your-lib

 (The option ``--strip-unneeded`` makes ``strip`` remove only the symbols
 which aren't needed for relocation processing.) Shared libraries can
@@ -123,7 +131,8 @@ linking are in a separate part of the ELF object file.  [#]_

 Note that under some circumstances it may be useful to install a shared
 library unstripped, for example when building a separate package to
-support debugging.
+support debugging.  The debhelper `dh_strip`` tool can create such
+packages automatically.

 Shared object files (often ``.so`` files) that are not public
 libraries, that is, they are not meant to be linked to by third party
@@ -741,9 +750,8 @@ restricted to ASCII when it is possible to do so.
shared library, like ``mklibs`` does in the Debian installer project.

 .. [#]
-   You might also want to use the options ``--remove-section=.comment``
-   and ``--remove-section=.note`` on both shared libraries and
-   executables, and ``--strip-debug`` on static libraries.
+   You might also want to use the option ``--strip-debug`` on static
+   libraries.

 .. [#]
A common example are the so-called "plug-ins", internal shared

=

To obtain a side-by-side diff:

% git clone salsa.debian.org:dbnpolicy/policy.git debian-policy
% cd debian-policy
% git difftool -y -x icdiff master..origin/bug188731-spwhitton

Alternatively, visit

https://salsa.debian.org/dbnpolicy/policy/commit/3cc86484767ac0aead9b7466c074ade5021ef225?view=parallel

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#188731: debian-policy: "strip --strip-unneeded" is insufficient

[Niels Thykier]

On Mon, 11 May 2015 11:18:29 +0200 Bill Allombert 
wrote:
> On Mon, May 11, 2015 at 01:39:43PM +0500, Andrey Rahmatullin wrote:
> > On Sat, Apr 12, 2003 at 07:24:12PM +0200, Matthias Urlichs wrote:
> > > Section 11.2 says 
> > > 
> > >   strip --strip-unneeded your-lib
> > This is still true (the section is 10.2 though).
> > 
> > > Lintian, however, complains if the sections .comment or .note are
> > > present, which strip doesn't think are unneeded.
> > This is still true, the tag is binary-has-unneeded-section.
> > Emitted (non-overridden): 537, overridden: 71, total: 608
> > 
> > > I don't know whether this is a bug in policy, strip, or lintian, but
> > > since Debian's "install -s" and dh_strip both use the additional options
> > > 
> > >   --remove-section=.comment --remove-section=.note
> > > 
> > > I think that it's a policy bug.
> > This is still true for dh_strip but not for install -s (since 2005,
> > coreutils 5.93-1).
> 
> So to summary:
> Policy 11.2 recommends:
>   strip --strip-unneeded
> dh_strip does:
>   strip --strip-unneeded --remove-section=.comment --remove-section=.note
> install -s does currently:
>   strip --strip-unneeded
> lintian checks for:
>   strip --strip-unneeded --remove-section=.comment --remove-section=.note
> 
> is that correct ?
> 
> Did 'install -s' used to do 
> 'strip --strip-unneeded --remove-section=.comment --remove-section=.note' ?
> Do we know why it was changed ?
> 
> Cheers,
> -- 
> Bill. 
> 
> Imagine a large red swirl here. 
> 
> 

Hi,

AFAICT, Bill's follow up is nearly a accurate with the following remarks:

 * The Policy section is (now?) 10.2 (plus some bits in 10.1)

 * We now have auto-generated dbgsym packages by dh_strip (which were
   just an idea when Bill wrote that answer).

 * Policy mentions "--remove-section=.comment --remove-section=.note" in
   a footnote as a "You might also want to use ...".

 * Policy section 10.1 implies that "INSTALL = install -s" is a useful
   way of stripping binaries (both in text and examples).  Besides not
   covering the .comment + .note sections it also neuters dh_strip's
   ability to create dbgsym packages.


I think we should update the policy to say that you should use
"--strip-unneeded --remove-section=.comment --remove-section=.note" and
maybe recommend delegating that task (where possible) to dh_strip as it
provide dbgsym packages.

Thanks,
~Niels

Bug#188731: debian-policy: strip --strip-unneeded is insufficient

[Andrey Rahmatullin]

On Mon, May 11, 2015 at 11:18:29AM +0200, Bill Allombert wrote:
 So to summary:
 Policy 11.2 recommends:
   strip --strip-unneeded
 dh_strip does:
   strip --strip-unneeded --remove-section=.comment --remove-section=.note
 install -s does currently:
   strip --strip-unneeded
 lintian checks for:
   strip --strip-unneeded --remove-section=.comment --remove-section=.note
 
 is that correct ?
Yes.

 Did 'install -s' used to do 
 'strip --strip-unneeded --remove-section=.comment --remove-section=.note' ?

Yes, lintian 0.4.4 (1998) mentions it:

  * (checks/binaries) At Yann Dirson's suggestion, added a tag for
stripped binaries that still have the .comment and .note sections.
This tag is info-level, thus not normally displayed.
dh_strip and install -s have been patched to strip these sections,
but most of the distribution has not been recompiled with the patched
versions.
  * (checks/binaries.desc) Described the new tag.
(closes #22495).

In lintian 1.20.1 (2001) the tag was promoted from I to W, after the patch
for install was removed in coreutils 5.93-1 the tag was demoted back to I
in lintian 1.23.14 (2005).

 Do we know why it was changed ?
coreutils 5.93-1 changelog says:

  * Remove some ancient debian-specific patches
- install no longer calls strip with special options


-- 
WBR, wRAR


signature.asc
Description: Digital signature

Bug#188731: debian-policy: strip --strip-unneeded is insufficient

[Bill Allombert]

On Mon, May 11, 2015 at 01:39:43PM +0500, Andrey Rahmatullin wrote:
 On Sat, Apr 12, 2003 at 07:24:12PM +0200, Matthias Urlichs wrote:
  Section 11.2 says 
  
  strip --strip-unneeded your-lib
 This is still true (the section is 10.2 though).
 
  Lintian, however, complains if the sections .comment or .note are
  present, which strip doesn't think are unneeded.
 This is still true, the tag is binary-has-unneeded-section.
 Emitted (non-overridden): 537, overridden: 71, total: 608
 
  I don't know whether this is a bug in policy, strip, or lintian, but
  since Debian's install -s and dh_strip both use the additional options
  
  --remove-section=.comment --remove-section=.note
  
  I think that it's a policy bug.
 This is still true for dh_strip but not for install -s (since 2005,
 coreutils 5.93-1).

So to summary:
Policy 11.2 recommends:
  strip --strip-unneeded
dh_strip does:
  strip --strip-unneeded --remove-section=.comment --remove-section=.note
install -s does currently:
  strip --strip-unneeded
lintian checks for:
  strip --strip-unneeded --remove-section=.comment --remove-section=.note

is that correct ?

Did 'install -s' used to do 
'strip --strip-unneeded --remove-section=.comment --remove-section=.note' ?
Do we know why it was changed ?

Cheers,
-- 
Bill. ballo...@debian.org

Imagine a large red swirl here. 


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#188731: debian-policy: strip --strip-unneeded is insufficient

[Andrey Rahmatullin]

On Sat, Apr 12, 2003 at 07:24:12PM +0200, Matthias Urlichs wrote:
 Section 11.2 says 
 
   strip --strip-unneeded your-lib
This is still true (the section is 10.2 though).

 Lintian, however, complains if the sections .comment or .note are
 present, which strip doesn't think are unneeded.
This is still true, the tag is binary-has-unneeded-section.
Emitted (non-overridden): 537, overridden: 71, total: 608

 I don't know whether this is a bug in policy, strip, or lintian, but
 since Debian's install -s and dh_strip both use the additional options
 
   --remove-section=.comment --remove-section=.note
 
 I think that it's a policy bug.
This is still true for dh_strip but not for install -s (since 2005,
coreutils 5.93-1).

-- 
WBR, wRAR


signature.asc
Description: Digital signature