Re: OCaml binary stripping (was: Re: OCaml 5 again)

2023-07-13 Thread Mark Wielaard 
Hi,

On Wed, Jul 12, 2023 at 03:38:35PM -0600, Jerry James wrote:
> On Wed, Jul 12, 2023 at 3:34 PM Richard W.M. Jones  wrote:
> > I looked through the section names with objdump -h and none of them
> > looked unusual.  However I think in the past when we analyzed this we
> > found it was doing something really crazy, like actually appending the
> > bytecode to the file (ie. not using the ELF format at all).
> 
> That was the conclusion I had just come to.  In the unstripped binary,
> the ELF structure accounts for about 700K, but the file size is over
> 3M.  It looks like they really do just append the bytecode to the
> file.

Aha, yeah, then any tool that tries to process the ELF file will
likely just overwrite that or throw it away because it has no way of
knowing that data is even part of the file.

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: OCaml binary stripping (was: Re: OCaml 5 again)

2023-07-12 Thread Jerry James 
On Wed, Jul 12, 2023 at 3:34 PM Richard W.M. Jones  wrote:
> I looked through the section names with objdump -h and none of them
> looked unusual.  However I think in the past when we analyzed this we
> found it was doing something really crazy, like actually appending the
> bytecode to the file (ie. not using the ELF format at all).

That was the conclusion I had just come to.  In the unstripped binary,
the ELF structure accounts for about 700K, but the file size is over
3M.  It looks like they really do just append the bytecode to the
file.
-- 
Jerry James
http://www.jamezone.org/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: OCaml binary stripping (was: Re: OCaml 5 again)

2023-07-12 Thread Richard W.M. Jones 
On Wed, Jul 12, 2023 at 09:21:40PM +0200, Mark Wielaard wrote:
> Hi,
> 
> On Wed, Jul 12, 2023 at 09:54:49AM -0600, Jerry James wrote:
> > On Wed, Jul 12, 2023 at 4:38 AM Richard W.M. Jones  
> > wrote:
> > > During the OCaml 5 rebuild I found there's a generic problem on s390x
> > > & ppc64le (ie. the bytecode-only architectures) involving stripping of
> > > OCaml binaries. [...]
> > >
> > > The binary is correct when built, but gets corrupted after one of
> > > these steps is applied (not sure which):
> > >
> > >   + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
> > >   + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip
> 
> OK, that is later than I suspect.  I believe these steps run after the
> debuginfo packages are created.  And I don't know much about these
> steps, so if that is the case my advise might not be relevant.
> 
> > > Anyway this could potentially affect any s390x / ppc64le OCaml package
> > > that contains a binary so it's something to look out for ...  For some
> > > reason dune doesn't seem to be affected.
> > [...]
> > I encountered that with a couple of packages, ocaml-findlib and
> > coccinelle.  Sorry I missed the others.  Yes, stripping those binaries
> > removes the bytecode payload, leaving only the bytecode interpreter.
> > I also noticed that dune isn't affected, which is interesting.  I
> > wonder if it is the difference between -output-complete-exe and
> > -custom.  Dune now uses the former instead of the latter.
> 
> It would be interesting to know what the difference between these
> methods is. Specifically if they create different names (or
> alloced/unalloced) ELF sections.

A way to reproduce the issue is:

  $ echo 'print_endline "hello world"' > test.ml
  $ echo 'void f(void) {}' > test-c.c
  $ ocamlc -custom -o test test.ml test-c.c 
  $ ll test
  -rwxr-xr-x. 1 rjones rjones 1406334 Jul 12 22:25 test
  $ ./test 
  hello world
  $ strip ./test
  $ ./test --version
  unknown option --version
  $ ./test 
  no bytecode file specified

If you replace -custom with -output-complete-exe then it does actually
work fine, so probably that's the answer to our problem.  Anyhow,
continuing with the analysis of -custom ...

I looked through the section names with objdump -h and none of them
looked unusual.  However I think in the past when we analyzed this we
found it was doing something really crazy, like actually appending the
bytecode to the file (ie. not using the ELF format at all).

I will look at whether we can fix this using -output-complete-exe in
supermin.  The documentation says:

  -output-complete-exe

Build a self-contained executable by linking a C object file
containing the bytecode program, the OCaml runtime system and any
other static C code given to ocamlc. The resulting effect is
similar to -custom, except that the bytecode is embedded in the C
code so it is no longer accessible to tools such as ocamldebug. On
the other hand, the resulting binary is resistant to strip.

which looks like what we need.

Rich.

> If the issue is with creating the debuginfo packages, then it likely
> is because there is an (unallocated) ELF section which is moved from
> the main executable to the .debug file (because debuginfo-find
> believes it isn't needed at runtime.
>
> In that case, and you can find out how such sections are named, then
> you might be able to workaround it by adding someting like:
> 
> %define _find_debuginfo_opts --keep-section .special_ocaml_bytecode
> 
> Cheers,
> 
> Mark

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: OCaml binary stripping (was: Re: OCaml 5 again)

2023-07-12 Thread Mark Wielaard 
Hi,

On Wed, Jul 12, 2023 at 09:54:49AM -0600, Jerry James wrote:
> On Wed, Jul 12, 2023 at 4:38 AM Richard W.M. Jones  wrote:
> > During the OCaml 5 rebuild I found there's a generic problem on s390x
> > & ppc64le (ie. the bytecode-only architectures) involving stripping of
> > OCaml binaries. [...]
> >
> > The binary is correct when built, but gets corrupted after one of
> > these steps is applied (not sure which):
> >
> >   + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
> >   + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip

OK, that is later than I suspect.  I believe these steps run after the
debuginfo packages are created.  And I don't know much about these
steps, so if that is the case my advise might not be relevant.

> > Anyway this could potentially affect any s390x / ppc64le OCaml package
> > that contains a binary so it's something to look out for ...  For some
> > reason dune doesn't seem to be affected.
> [...]
> I encountered that with a couple of packages, ocaml-findlib and
> coccinelle.  Sorry I missed the others.  Yes, stripping those binaries
> removes the bytecode payload, leaving only the bytecode interpreter.
> I also noticed that dune isn't affected, which is interesting.  I
> wonder if it is the difference between -output-complete-exe and
> -custom.  Dune now uses the former instead of the latter.

It would be interesting to know what the difference between these
methods is. Specifically if they create different names (or
alloced/unalloced) ELF sections.

If the issue is with creating the debuginfo packages, then it likely
is because there is an (unallocated) ELF section which is moved from
the main executable to the .debug file (because debuginfo-find
believes it isn't needed at runtime.

In that case, and you can find out how such sections are named, then
you might be able to workaround it by adding someting like:

%define _find_debuginfo_opts --keep-section .special_ocaml_bytecode

Cheers,

Mark
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: OCaml binary stripping (was: Re: OCaml 5 again)

2023-07-12 Thread Jerry James 
On Wed, Jul 12, 2023 at 4:38 AM Richard W.M. Jones  wrote:
> During the OCaml 5 rebuild I found there's a generic problem on s390x
> & ppc64le (ie. the bytecode-only architectures) involving stripping of
> OCaml binaries.  A good example is supermin on s390x:
>
>   # uname -a
>   Linux e132ed8a0a9b4411b210e43e253581f0 6.4.0-59.fc39.s390x #1 SMP Mon Jun 
> 26 11:55:15 UTC 2023 s390x GNU/Linux
>   # supermin --version
>   unknown option --version
>   # echo $?
>   127
>
> The binary is correct when built, but gets corrupted after one of
> these steps is applied (not sure which):
>
>   + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
>   + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip
>
> In the past we had this commit in supermin:
>
>   commit 18ebd2cc631f119a10374a914d5c725288060349
>   Author: Dan Horák 
>   Date:   Thu Sep 15 12:09:34 2016 +0200
>
> - Do not break the binary on interpreted builds (#1375213)
>
>   diff --git a/supermin.spec b/supermin.spec
>   index 30af5fb..8d4f03b 100644
>   --- a/supermin.spec
>   +++ b/supermin.spec
>   @@ -1,3 +1,8 @@
>   +%ifnarch %{ocaml_native_compiler}
>   +%global __strip /bin/true
>   +%global debug_package %{nil}
>   +%endif
>   +
>   [etc]
>
> which basically kills off stripping.  However that may or may not
> still work.  It was removed because it was no longer necessary (as we
> had an s390x native code backend), and because of this bug:
>
>   https://bugzilla.redhat.com/show_bug.cgi?id=1915570
>
> So I'm going with this instead to see if it cures the problem:
>
>   %ifnarch %{ocaml_native_compiler}
>   %global __strip /bin/true
>   %global _lto_cflags %nil
>   %global debug_package %{nil}
>   %endif
>
> (https://src.fedoraproject.org/rpms/supermin/c/61395e3f5a0fdbbea28ec03c6befcf482cb96792?branch=rawhide)
>
> Anyway this could potentially affect any s390x / ppc64le OCaml package
> that contains a binary so it's something to look out for ...  For some
> reason dune doesn't seem to be affected.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> nbdkit - Flexible, fast NBD server with plugins
> https://gitlab.com/nbdkit/nbdkit

I encountered that with a couple of packages, ocaml-findlib and
coccinelle.  Sorry I missed the others.  Yes, stripping those binaries
removes the bytecode payload, leaving only the bytecode interpreter.
I also noticed that dune isn't affected, which is interesting.  I
wonder if it is the difference between -output-complete-exe and
-custom.  Dune now uses the former instead of the latter.
-- 
Jerry James
http://www.jamezone.org/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

OCaml binary stripping (was: Re: OCaml 5 again)

2023-07-12 Thread Richard W.M. Jones 
During the OCaml 5 rebuild I found there's a generic problem on s390x
& ppc64le (ie. the bytecode-only architectures) involving stripping of
OCaml binaries.  A good example is supermin on s390x:

  # uname -a
  Linux e132ed8a0a9b4411b210e43e253581f0 6.4.0-59.fc39.s390x #1 SMP Mon Jun 26 
11:55:15 UTC 2023 s390x GNU/Linux
  # supermin --version
  unknown option --version
  # echo $?
  127

The binary is correct when built, but gets corrupted after one of
these steps is applied (not sure which):

  + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
  + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip

In the past we had this commit in supermin:

  commit 18ebd2cc631f119a10374a914d5c725288060349
  Author: Dan Horák 
  Date:   Thu Sep 15 12:09:34 2016 +0200

- Do not break the binary on interpreted builds (#1375213)

  diff --git a/supermin.spec b/supermin.spec
  index 30af5fb..8d4f03b 100644
  --- a/supermin.spec
  +++ b/supermin.spec
  @@ -1,3 +1,8 @@
  +%ifnarch %{ocaml_native_compiler}
  +%global __strip /bin/true
  +%global debug_package %{nil}
  +%endif
  +
  [etc]

which basically kills off stripping.  However that may or may not
still work.  It was removed because it was no longer necessary (as we
had an s390x native code backend), and because of this bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1915570

So I'm going with this instead to see if it cures the problem:

  %ifnarch %{ocaml_native_compiler}
  %global __strip /bin/true
  %global _lto_cflags %nil
  %global debug_package %{nil}
  %endif

(https://src.fedoraproject.org/rpms/supermin/c/61395e3f5a0fdbbea28ec03c6befcf482cb96792?branch=rawhide)

Anyway this could potentially affect any s390x / ppc64le OCaml package
that contains a binary so it's something to look out for ...  For some
reason dune doesn't seem to be affected.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue