Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi josch, On Tue, 06 Aug 2019 20:11:07 +0200, Johannes Schauer wrote: > Quoting Stephen Kitt (2019-08-06 18:43:24) > > Could we avoid (some of) this by treating “source” as a new architecture? > > There would be a mostly do-nothing buildd which would repack the source > > package as a binary package in /usr/src/... > > The large number of binary packages needed with this method could of course > be reduced if only selected source packages get built that way. But such a > method already exists in the form of foo-source binary packages. > > Changing the build dependency syntax instead would have the advantage, that > we do not need more binary or source packages at all because all the > necessary logic would come from the resolvers themselves. Thanks for the detailed explanations, changing the build dependency syntax does indeed seem like a better approach. Regards, Stephen pgp4enki5g859.pgp Description: OpenPGP digital signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi, Quoting Stephen Kitt (2019-08-06 18:43:24) > > recently there was some discussion about this in #debian-apt. I don't have > > the logs of that discussion so others might want to expand on what I > > remember from back then. In no particular order plus my own thoughts. > > > > apt developers are in favour of such a feature but it would need > > implementation in dpkg first. This means that dpkg would have to also track > > "installed" (unpacked) source packages in /usr/src in a similar way of how > > it currently tracks installed binary packages. > > > > Tons of software that parses the Build-Depends field has to be patched. At > > least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3, > > wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas, > > haskell-debian, dh-exec, autopkgtest... > > Could we avoid (some of) this by treating “source” as a new architecture? > There would be a mostly do-nothing buildd which would repack the source > package as a binary package in /usr/src/... > > That would still mean we’d have to add meaningful arch-qualified > build-dependencies in all the affected tools, and it doesn’t help with the > transitive dependencies (although the binary package containing the source > could carry the appropriate dependencies), so perhaps it’s not such a great > idea. sure, it could work in theory. It would just be very ugly. Lets have a look at how this would look like: To get the raw sources in /usr/src you would generate around 31000 new binary packages of the "source" architecture. These would have to be Multi-Arch:foreign so that they can satisfy dependencies coming from source packages being built for any architecture. I don't see a way to express the Architecture field of source packages via this method, which is another downside. As you already point out, properly encoding build dependencies that way is another can of worms. For this to work, you would have to generate meta packages for every architecture we support for every source package. This is so that the host architecture gets properly forwarded down the dependency graph. This would mean another few hundred thousand more binary packages. The large number of binary packages needed with this method could of course be reduced if only selected source packages get built that way. But such a method already exists in the form of foo-source binary packages. Changing the build dependency syntax instead would have the advantage, that we do not need more binary or source packages at all because all the necessary logic would come from the resolvers themselves. Thanks! cheers, josch signature.asc Description: signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, 28 Jul 2019 16:33:21 +0200, Johannes Schauer wrote: > Quoting Mo Zhou (2019-07-28 16:03:43) > > On 2019-07-28 13:13, Ian Jackson wrote: > > > This is maybe not directly helpful to you right now, but: > > > > > > If we could build-depend on source packages, you could combine these > > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source package". > > Is such demand common enough among developers? > > recently there was some discussion about this in #debian-apt. I don't have > the logs of that discussion so others might want to expand on what I > remember from back then. In no particular order plus my own thoughts. > > apt developers are in favour of such a feature but it would need > implementation in dpkg first. This means that dpkg would have to also track > "installed" (unpacked) source packages in /usr/src in a similar way of how > it currently tracks installed binary packages. > > Tons of software that parses the Build-Depends field has to be patched. At > least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3, > wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas, > haskell-debian, dh-exec, autopkgtest... Could we avoid (some of) this by treating “source” as a new architecture? There would be a mostly do-nothing buildd which would repack the source package as a binary package in /usr/src/... That would still mean we’d have to add meaningful arch-qualified build-dependencies in all the affected tools, and it doesn’t help with the transitive dependencies (although the binary package containing the source could carry the appropriate dependencies), so perhaps it’s not such a great idea. Regards, Stephen pgpv5VyQb9Dn8.pgp Description: OpenPGP digital signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On 28/07/2019 20:32, Paul Wise wrote: > On Sun, Jul 28, 2019 at 10:04 PM Mo Zhou wrote: > >> Is such demand common enough among developers? > > There are several ways this would be useful: > To replace all out-of-tree Linux kernel module -source packages, so > that dkms/etc doesn't need a binary package duplicating the source. So how / where exactly would you build modules? Modify dkms to use source packages? module-assistant again? -- Tzafrir
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Johannes Schauer writes ("Re: B-D on src package? (was: Re: Challenge from
Julia's non-standard vendored openblas"64_""):
> We have to think about a good syntax for the Build-Depends field
> which is able to express a build dependency on source packages
> unpacked to /usr/src
Can I make a radical suggestion that the package should expect the
source to be in ../src (or similar) rather than in /usr/src ?
I appreciate we usually do this stuff in buildds but there are other
environments or situations where it would be much nicer if the source
code were somewhere that could be writeable by an ordinary user.
In particular if A src-build-depends B, usually that means that A
compiles B in a funny way. To do development work properly on A, it
is necessary to be constantly editing B. Now probably A has an ad-hoc
way to point to a different B but that is fragile and annoying.
I can see few downsides from treating
Build-Depends: src:foo:src
(or whatever) to mean "src:foo" unpacked in a directory ../src/foo
(or some suitable other prefix).
(It should not mean that "foo.dsc" et al are available anywhere - just
the unpacked version, as found from dpkg-source -x.)
Ian.
Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi Benda, On 2019-07-30 15:21, Benda Xu wrote: > Does Julia's built-in package manager support to build packages from > source, like R? If so, the Debian version of Julia's package manager > could be set to build from source by default. My memory about Pkg.jl's behavior is fuzzy. At least currently I don't know how to build e.g. Arpack.jl from source instead of using their pre-built binaries. Pkg.jl document mentioned nothing about prebuilt binaries: https://docs.julialang.org/en/v1/stdlib/Pkg/
Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi Mo, Mo Zhou writes: > 64-bit version without symbol mangling will run into problem as long > as the user tries to install some fundamental .jl packages with the > Julia's built-in package manager. Does Julia's built-in package manager support to build packages from source, like R? If so, the Debian version of Julia's package manager could be set to build from source by default. Yours, Benda
Re: Challenge from Julia's non-standard vendored openblas"64_"
On Mon, Jul 29, 2019 at 9:35 AM Mo Zhou wrote: > > Hi Marco, > > After the merger of (64bit-indexing) * (multi-thread flavor) > feature enhancement, an libopenblas-julia package will > look abrupt, and break the symmetry of package layout > (libopenblas-julia doesn't need development files) : > > ~/D/o/o/debian ❯❯❯ rg \^Package control > 19:Package: libopenblas0 > 41:Package: libopenblas0-pthread > 64:Package: libopenblas0-openmp > 87:Package: libopenblas0-serial > 134:Package: libopenblas-dev > 156:Package: libopenblas-pthread-dev > 179:Package: libopenblas-openmp-dev > 202:Package: libopenblas-serial-dev > 227:Package: libopenblas64-0 > 245:Package: libopenblas64-0-pthread > 264:Package: libopenblas64-0-openmp > 283:Package: libopenblas64-0-serial > 302:Package: libopenblas64-dev > 321:Package: libopenblas64-pthread-dev > 341:Package: libopenblas64-openmp-dev > 361:Package: libopenblas64-serial-dev > > Instead, if we embed the openblas source > to src:julia, no extra binary package is needed. > So I prefer (3). > > It's not the maintenance burden that matters most > at this point. It's simply that a maintainer doesn't > want his/her package to look weird. > Then we would have two copies of openblas in the archive, which smells it would look quite weird from FTP Team's perspective. Regards, Aron
Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi Marco, After the merger of (64bit-indexing) * (multi-thread flavor) feature enhancement, an libopenblas-julia package will look abrupt, and break the symmetry of package layout (libopenblas-julia doesn't need development files) : ~/D/o/o/debian ❯❯❯ rg \^Package control 19:Package: libopenblas0 41:Package: libopenblas0-pthread 64:Package: libopenblas0-openmp 87:Package: libopenblas0-serial 134:Package: libopenblas-dev 156:Package: libopenblas-pthread-dev 179:Package: libopenblas-openmp-dev 202:Package: libopenblas-serial-dev 227:Package: libopenblas64-0 245:Package: libopenblas64-0-pthread 264:Package: libopenblas64-0-openmp 283:Package: libopenblas64-0-serial 302:Package: libopenblas64-dev 321:Package: libopenblas64-pthread-dev 341:Package: libopenblas64-openmp-dev 361:Package: libopenblas64-serial-dev Instead, if we embed the openblas source to src:julia, no extra binary package is needed. So I prefer (3). It's not the maintenance burden that matters most at this point. It's simply that a maintainer doesn't want his/her package to look weird. On 2019-07-29 00:12, Marco d'Itri wrote: > On Jul 28, Mo Zhou wrote: > >> 2. Specifically compile a libopenblas64_.so >>from src:openblas for julia's use. >>This looks very bad for src:openblas's >>package layout. > Why would this look bad? We have plenty of source packages which > generate multiple variations of binary packages. > udebs, for a start. Many libraries which generate both a static and > dynamic package. The older inn2 releases if you want to see something > really ugly. > While this solution may require some packaging work it is obviously both > the technically correct one and the one which will not harm users.
Re: Challenge from Julia's non-standard vendored openblas"64_"
On Jul 28, Mo Zhou wrote: > 2. Specifically compile a libopenblas64_.so >from src:openblas for julia's use. >This looks very bad for src:openblas's >package layout. Why would this look bad? We have plenty of source packages which generate multiple variations of binary packages. udebs, for a start. Many libraries which generate both a static and dynamic package. The older inn2 releases if you want to see something really ugly. While this solution may require some packaging work it is obviously both the technically correct one and the one which will not harm users. -- ciao, Marco signature.asc Description: PGP signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 10:33 PM Johannes Schauer wrote: > If we choose the "src:" prefix to pick source instead of binary packages, then > it's important that we don't have any binary packages called "src" and no > source packages with a name equal to a valid debian architecture. Could we have a double suffix instead to avoid these issues? Build-Depends: foo:src:src -> src:foo unpacked in /usr/src/foo Build-Depends: foo:src -> b-d of src:foo for the current host architecture Build-Depends: foo:src:amd64-> b-d of src:foo for amd64 Build-Depends: foo:src:native -> b-d of src:foo for the current build architecture -- bye, pabs https://wiki.debian.org/PaulWise
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 10:04 PM Mo Zhou wrote: > Is such demand common enough among developers? There are several ways this would be useful: To replace all librust-*-dev and golang-*-dev packages (they contain source code). To replace all toolchain -source packages, so that cross-compiling toolchains can be built from separate source packages. To replace all out-of-tree Linux kernel module -source packages, so that dkms/etc doesn't need a binary package duplicating the source. Anywhere you want to build multiple independent build configurations of the same source code in different ways. -- bye, pabs https://wiki.debian.org/PaulWise
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Mo Zhou 于 2019年7月28日周日 16:03写道： > On 2019-07-28 13:13, Ian Jackson wrote: > > This is maybe not directly helpful to you right now, but: > > > > If we could build-depend on source packages, you could combine these > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source > package". > Is such demand common enough among developers? > Yes, for those static linked languages, at least for Go. // send from my mobile device
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi,
Quoting Mo Zhou (2019-07-28 16:03:43)
> On 2019-07-28 13:13, Ian Jackson wrote:
> > This is maybe not directly helpful to you right now, but:
> >
> > If we could build-depend on source packages, you could combine these
> > two ideas into something that might be less awful.
> More than once have I thought of "what if I can B-D on a source package". Is
> such demand common enough among developers?
recently there was some discussion about this in #debian-apt. I don't have the
logs of that discussion so others might want to expand on what I remember from
back then. In no particular order plus my own thoughts.
apt developers are in favour of such a feature but it would need implementation
in dpkg first. This means that dpkg would have to also track "installed"
(unpacked) source packages in /usr/src in a similar way of how it currently
tracks installed binary packages.
Tons of software that parses the Build-Depends field has to be patched. At
least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3,
wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas,
haskell-debian, dh-exec, autopkgtest...
We have to think about a good syntax for the Build-Depends field which is able
to express a build dependency on source packages unpacked to /usr/src as well
as a build dependency on the build dependencies of a certain source package for
a given host architecture. Some initial thoughts:
Build-Depends: src:foo:src -> src:foo unpacked in /usr/src/foo
Build-Depends: src:foo -> b-d of src:foo for the current host
architecture
Build-Depends: src:foo:amd64-> b-d of src:foo for amd64
Build-Depends: src:foo:native -> b-d of src:foo for the current build
architecture
Here, ":src" is a new suffix next to :native, :any or :${arch} indicating the
"source architecture", meaning an unpacked source package. It is also open what
a B-D on foo:src (without the src: prefix) would mean. Maybe there is a more
elegant solution.
If we choose the "src:" prefix to pick source instead of binary packages, then
it's important that we don't have any binary packages called "src" and no
source packages with a name equal to a valid debian architecture.
I think it's important to separate a dependency on the source code of src:foo
and a dependency on the build dependencies of src:foo. Sometimes one needs only
parts of src:foo and if unpacked source and its build dependencies are always
installed together, then there is no way to get one without the other.
And there has to be a syntax for how to "install" these from the command line
using apt-get.
And there is the question of whether source packages of different versions
should be installable at the same time, maybe into /usr/src/foo-${version}.
Currently we have foo-source binary packages which install their content into
/usr/src but this requires such a package to be created and there is currently
no way to get the precise build dependencies on that package.
There is also the question about build profiles. Should it be possible to only
request the build dependencies of a source package with a certain set of build
profiles active?
Are other distributions doing something similar already?
Thanks!
cheers, josch
signature.asc
Description: signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 07:03:43AM -0700, Mo Zhou wrote: > > This is maybe not directly helpful to you right now, but: > > > > If we could build-depend on source packages, you could combine these > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source > package". > Is such demand common enough among developers? `apt-file search -l /usr/src` may be of interest (after you skip all kernel-related packages). -- WBR, wRAR signature.asc Description: PGP signature
B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On 2019-07-28 13:13, Ian Jackson wrote: > This is maybe not directly helpful to you right now, but: > > If we could build-depend on source packages, you could combine these > two ideas into something that might be less awful. More than once have I thought of "what if I can B-D on a source package". Is such demand common enough among developers? > Anyway, good luck.
Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi Phil, On 2019-07-28 10:14, Phil Morrell wrote: > Without knowing anything more than what you've written here (which I > didn't find too long), it sounds like maintenance burden is the concern. The situation forces we Julia package maintainers to decide whether to use such a non-standard openblas variant. If we did't link against it, the built-in package manager (similar to python's pip) will be hard for Debian users to use. If we linked against it, we get a weird and duplicated binary in the archive but the usefulness wouldn't be harmed. > Am I right in thinking that there still exists non-Julia software for > which your solution is cleaner than symbol mangling? Is that LAPACK? The 64-bit indexing variant is common in the scientific computing area. Providing the BLAS64/LAPACK64 alternative group without mangling symbol names makes sense, because the other BLAS64/LAPACK64 alternatives such as BLIS and intel's MKL don't mangle symbol. > What you would do today if you were packaging it from scratch? If you > would pick the Julia approach for ease of maintenance then a SONAME > transition seems "simple" enough. If you would implement the cleaner > no-mangling approach, then a libopenblas-julia compatibility dependency > (option 2) would isolate the problem to the julia ecosystem. I tend to choose option 3. Expecially if I'm packaging it from scratch.
Re: Challenge from Julia's non-standard vendored openblas"64_"
Mo Zhou writes ("Challenge from Julia's non-standard vendored openblas"64_""):
> [stuff]
How awkward.
> 2. Specifically compile a libopenblas64_.so
>from src:openblas for julia's use.
...
> 3. Embed openblas source and let Julia's
This is maybe not directly helpful to you right now, but:
If we could build-depend on source packages, you could combine these
two ideas into something that might be less awful.
Anyway, good luck.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 02:30:15AM -0700, Mo Zhou wrote: > problem is that the 64-bit indexing variant doesn't > have a standard SONAME. Without knowing anything more than what you've written here (which I didn't find too long), it sounds like maintenance burden is the concern. Am I right in thinking that there still exists non-Julia software for which your solution is cleaner than symbol mangling? Is that LAPACK? > long time ago, we (mainly BLAS/LAPACK maintainers) > decided to use the "SONAME=libXXX64.so" convention > without mangling symbol names. Mangling is not > considered because only openblas supports it. What you would do today if you were packaging it from scratch? If you would pick the Julia approach for ease of maintenance then a SONAME transition seems "simple" enough. If you would implement the cleaner no-mangling approach, then a libopenblas-julia compatibility dependency (option 2) would isolate the problem to the julia ecosystem. > their vendored OpenBLAS to "libopenblas64_.so.*", Ugh, vendoring "compiles for me, so it's your problem". > I have no confidence at all to convince > upstream to change their widely-spread practice. Even when that's the case, it's usually still worth reporting the issue upstream, so they know the pain they're introducing to potential users. All the best from an outsider, and thank you for tackling difficult interoperability decisions in Debian. -- Phil Morrell (emorrp1) signature.asc Description: PGP signature
Challenge from Julia's non-standard vendored openblas"64_"
Hi developers, A long existing historical problem that stems from Fortran finally started to bite people. --- Background BLAS/LAPACK is a set of very stable API/ABIs for dense numerical linear algebra, of "libc-level" importance to scientific computing users. The standard/reference/low-performance implementation was written in fortran. Fortran compilers support a weird feature to change the default INTEGER length. Which means if you write a fortran subroutine (in C-pseudo) sasum(INT N, float* X, INT INCX) You'll get the sasum(int64_t, float*, int64_t) function with a "-fdefault-integer-8" option. The "INT" arguments in the above example are used for 1-d array indexing. The scientific computing community has got used to this fortran feature, and many compatible BLAS/LAPACK implementations support build flags to select the "default integer size" even if the implementation is written in C. The historical problem is that the 64-bit indexing variant doesn't have a standard SONAME. --- Debian's WIP BLAS64/LAPACK64 libraries I'm working on the 64bit-indexing support for Debian's BLAS/LAPACK libraries. As discussed very long time ago, we (mainly BLAS/LAPACK maintainers) decided to use the "SONAME=libXXX64.so" convention without mangling symbol names. Mangling is not considered because only openblas supports it. Actually, isolating the 32-bit and 64-bit indexing variants with different SONAMES is expected to be enough for avoiding confliction. --- Julia's practice To avoid symbol confliction with the 32bit-indexing BLAS/LAPACK, Julia upstream changed the SONAME of their vendored OpenBLAS to "libopenblas64_.so.*", and **MANGLED** all the BLAS/LAPACK symbols by appending the "64_" suffix. As a result, a portion of the pre-built binaries they release are linked against this libopenblas64_.so, and requiring the mangled symbols (readelf -sW xxx). At the same time the linux distribution Julia packages built against 32bit version of 64-bit version without symbol mangling will run into problem as long as the user tries to install some fundamental .jl packages with the Julia's built-in package manager. I have no confidence at all to convince upstream to change their widely-spread practice. If I 1. Patch Julia code and link it against the WIP BLAS64/LAPACK64 libraries (where there is no symbol mangling). Users will always have problem installing any external .jl packages. 2. Specifically compile a libopenblas64_.so from src:openblas for julia's use. This looks very bad for src:openblas's package layout. 3. Embed openblas source and let Julia's build system compile whatever it wants. In this case everything will be fine, except for the technical gracefulness. I hesitate to choose a solution from them. Advise? --- P.S. This mail is again very long. But I'm afraid that most people won't be able to understand the problem if I only describe the core problem with several simple words.
