Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Drew Parsons]

On Sat, 2018-05-19 at 10:55 +, Lumin wrote:
> On Fri, May 18, 2018 at 11:49:05PM +0800, Drew Parsons wrote:
> > 
> > I wonder if the simplest solution is to just have 
> > intel-mkl Depends: libblas.  i.e. use policy to simply prevent a
> > sole
> > mkl installation.  
> > 
> > That way, the mkl alternative will always have a free BLAS to press
> > it's preference against.
> 
> That's exactly a part of the present implementation.
> 
> It depends on libblas3 | libblas.so.3, and enhances libblas.so.3, but
> never provides libblas.so.3 . Similar for lapack.

It's come together well then.  Thanks for your efforts on it, Lumin.

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Lumin]

On Fri, May 18, 2018 at 11:49:05PM +0800, Drew Parsons wrote:
> 
> I wonder if the simplest solution is to just have 
> intel-mkl Depends: libblas.  i.e. use policy to simply prevent a sole
> mkl installation.  
>
> That way, the mkl alternative will always have a free BLAS to press
> it's preference against.

That's exactly a part of the present implementation.

It depends on libblas3 | libblas.so.3, and enhances libblas.so.3, but
never provides libblas.so.3 . Similar for lapack.

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Drew Parsons]

On Sat, 2018-05-12 at 03:22 +, Lumin wrote:
> Hi Sébastien,
> 
> > But IMO it's acceptable to not perfectly deal with the
> > corner case
> > where only MKL is installed, as long as some warning is displayed.
> 
> I insist on removing the Provides, even if it looks weird. For sake
> of
> debconf correctness, I can't find a better way other than removing
> it.
...
> As a compromise, let's regard MKL as a "non-free" enhancement to free
> BLAS/LAPACK implementations. An Enhances: field should be nice for
> us,
> which alleviates the discomfort of leaving Provides: blank.

I wonder if the simplest solution is to just have 
intel-mkl Depends: libblas.  i.e. use policy to simply prevent a sole
mkl installation.  

That way, the mkl alternative will always have a free BLAS to press
it's preference against.

Drew

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Lumin]

Hi Sébastien,

> Using a Pre-Depends here is IMO wrong. Quoting Policy §7.2:

Thanks. I didn't notice that when considering ways to avoid corner
cases.

> I also think that removing the Provides is not a good idea. The alternative is
> provided by the package, and that should be made clear in the dependency
> relationships.

> I'm sorry but I don't have an ideal solution to the problems we previously
> discussed. But IMO it's acceptable to not perfectly deal with the corner case
> where only MKL is installed, as long as some warning is displayed.

I insist on removing the Provides, even if it looks weird. For sake of
debconf correctness, I can't find a better way other than removing it.
This is a special situation where we are dealing with non-free blobs.
The choice of providing alternative while leaving the Provides field blank
is due to debconf correctness -- which is more important than the
Provides field since it directly reflects the user's choice.

* When there is only libmkl-rt as libblas.so.3 provider, libblas.so.3
  is still used as the default implementation EVEN IF THE USER REJECTS
  TO USE IT AS DEFAULT in debconf dialog.

That means when there is MKL, there MUST be another free alternative.
However, assigning Provides to MKL may lead to unexpected corner cases
where the consideration of debconf dialog turns in vain, because it is
not simple to make sure the existence of another free alternative.

In a word, MKL itself is non-free alternative to BLAS/LAPACK, and there
should be no problem to disallow MKL to satisfy libblas.so.3 / liblapack.so.3
requirement in a dependency tree where there are LOTS OF GPL-licensed
software. I'll override it if leaving Provides blank violates policy
because legal safety is greater than technological corectness.

As a compromise, let's regard MKL as a "non-free" enhancement to free
BLAS/LAPACK implementations. An Enhances: field should be nice for us,
which alleviates the discomfort of leaving Provides: blank.


signature.asc
Description: PGP signature

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Drew Parsons]

On Sun, 2018-05-06 at 10:13 +0200, Sébastien Villemot wrote:
> 
> Yes, but it’s the GPL that forbids distribution of a binary linking
> together
> GPL code and non-free code.

But we're not distributing GPL binaries for use with nonfree MKL, we're
distributing them to use with OpenBLAS or ATLAS.

If the user then installs MKL and GPL software on it, then that's their
responsibility. And they're allowed to, so long as they don't
distribute it that way.

Drew

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Lumin]

Just forgot to CC the RFS bug.

- Forwarded message from Lumin  -

Date: Sun, 6 May 2018 08:29:29 +
From: Lumin 
To: sebast...@debian.org
Cc: debian-scie...@lists.debian.org
Subject: Re: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS
User-Agent: Mutt/1.9.5 (2018-04-13)

Hello guys,

I've just updated the MKL packaging, and added a BLAS performance tester
there. I'll test the package soon, but let me first describe the
changes.

> He put forward a simpler solution: Just don't provide libblas.so.3, such
> that MKL will never be used to satisfy the dependency of libblas.so.3 .
> Based on his idea, my new solution is the follows:
> 
>   libmkl-rt --
>   Depends: libblas3 | libblas.so.3
>   Provides: NOTHING  ... (4)
> 
> So it's totally safe now. If there is MKL, there must be a free
> libblas.so.3 implementation with a priority definitely larger than 1,
> overriding MKL in terms of auto-mode alternatives. On the other hand,
> if that alternative is manually set, then there is nothing to worry
> about. This solution is also able to resolve problems found in (1) and (3).

Now libmkl-rt doesn't provide libblas.so.3 and liblapack.so.3, and it
pre-depends on libblas3 | libblas.so.3 and liblapack3 | liblapack.so.3 .
Similar change was applied to libmkl-dev.

> Apart from all things discussed above, there is upstream confirmation
> to the ambiguous license declaration in several headers. See [1]

Now the copyright file is updated. Copyright is clean.

> Thanks. There is a small bug: in both libmkl-rt.postinst.in and
> libmkl-dev.postinst.in, when the users reply "no", you put both BLAS and 
> LAPACK
> alternatives in auto mode if MKL was selected for BLAS. You should rather 
> split
> that in two tests: one for BLAS, one for LAPACK, because in theory it's
> possible to have BLAS pointing to MKL and not LAPACK.

One more debconf menu is added to configure script. A multiselect box
will be presented to the user so that we can treat BLAS and LAPACK
separately. That multiselect menu only appear once. For example, if the
user chose to point BLAS to MKL but not LAPACK, i.e.

  libblas.so.3-x86_64-linux-gnu -> libmkl_rt.so
  liblapack.so.3-x86_64-linux-gnu -> won't be touched

then following the configuration, these symlink will be set accordingly:

  libblas.so.3-i386... -> libmkl_rt.so
  liblapack.so.3-i386... -> won't be touched
  libblas.so-{amd64,i386} -> libmkl_rt.so
  liblapack.so-{amd64,i386} -> won't be touched.

For detail see [1][2][3]. Is this acceptable? If it's good, I'll test it
soon.


Previously Sébastien expressed his interest on benchmarking. I'm
interested in that too. So apart from the debconf part, I wrote a simple
benchmarker[4] in Julia[5] for comparing several BLAS implementations.
Thanks to Julia's convenient FFI functionality, I can test all
implementations within a single run, without any modification to environment
variable or alternatives.

Test result on my laptop (CPU=i5-7440HQ) matches expectation:

dnrm2 Julia
  0.33 seconds
dnrm2 /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
  0.74 seconds (3 allocations: 48 bytes)
  dnrm2 Error :1.1368683772161603e-13
dnrm2 /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
  0.38 seconds (3 allocations: 48 bytes)
  dnrm2 Error :0.0
dnrm2 /usr/lib/x86_64-linux-gnu/libopenblas_haswellp-r0.2.20.so
  0.31 seconds (3 allocations: 48 bytes)
  dnrm2 Error :0.0
dnrm2 /usr/lib/x86_64-linux-gnu/libmkl_rt.so
  0.029561 seconds (3 allocations: 48 bytes)
  dnrm2 Error :0.0
dgemm Julia
  4.362279 seconds (2 allocations: 128.000 MiB, 0.20% gc time)
dgemm /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
 47.893710 seconds (10 allocations: 160 bytes)
  dgemm Error :2.0735139719127268e-10
dgemm /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
 10.412422 seconds (10 allocations: 160 bytes)
  dgemm Error :2.4175670445887973e-11
dgemm /usr/lib/x86_64-linux-gnu/libopenblas_haswellp-r0.2.20.so
  1.211220 seconds (10 allocations: 160 bytes)
  dgemm Error :2.770610675980814e-11
dgemm /usr/lib/x86_64-linux-gnu/libmkl_rt.so
  1.103356 seconds (10 allocations: 160 bytes)
  dgemm Error :2.7982744719588258e-11

Netlib is always the slowest one. For small matrices OpenBLAS is
very competitive. For large matrices MKL is the fastest.


[1] 
https://salsa.debian.org/science-team/intel-mkl/blob/master/debian/libmkl-rt.templates
[2] 
https://salsa.debian.org/science-team/intel-mkl/blob/master/debian/libmkl-rt.config.in
[3] 
https://salsa.debian.org/science-team/intel-mkl/blob/master/debian/libmkl-rt.postinst.in
[4] 
https://salsa.debian.org/science-team/intel-mkl/blob/master/debian/tests/blascomp.jl
[5] http://julialang.org/
My Julia version is 0.6.2, which doesn't exist in Debian archive.

- End forwarded message -

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Sébastien Villemot]

On Sun, May 06, 2018 at 08:00:53AM +, Lumin wrote:
> On Wed, May 02, 2018 at 10:03:38AM -0500, Dirk Eddelbuettel wrote:
> > 
> > On 2 May 2018 at 14:41, Lumin wrote:
> > | Seems that things are getting more complicated. Recall that here we'are
> > | going to prevent users from GPL violation in situations such as this
> > | one:
> > | 
> > |   debootstrap; apt install libmkl-rt; apt install octave; octave ... (1)
> > 
> > Are you sure? I do not think that is correct. Downloading and installating
> > MKL, and running it with R or Octave (or any other package linking the BLAS
> > interface) does not constitute a GPL violation AFAIK.
> 
> Not sure. I admit that I'm not good at long licences such as GPL.
> Whether it violates GPL or not, what we do in config/postinst won't
> change: make sure it's the user's explicit choice to use MKL as the
> default BLAS/LAPACK implementation, and in contrast, MKL will not be
> used without the explicit choice.
> 
> > (There may well be limitations on further redistribution of the aggregate
> > even though even the MKL now limits redistribution as it is of course still 
> > a
> > no-source-code piece of software.)
> 
> Intel's ISSL license allows redistribution, as long as no file is
> changed.

Yes, but it’s the GPL that forbids distribution of a binary linking together
GPL code and non-free code.

I’m not entirely sure whether it is legal to locally have such a binary on your
hard drive, that you would not redistribute (but anyways nobody will know).

The problem is that, by providing GPL'd software (e.g. GNU Octave), MKL and a
way to dynamically link the two together, one may consider that Debian is
distributing such an binary and therefore violating the GPL. So at the very
least we must clearly warn the user about that risk and not have MKL the
default BLAS implementation.

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  http://sebastien.villemot.name
⠈⠳⣄  http://www.debian.org


signature.asc
Description: PGP signature

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Lumin]

On Wed, May 02, 2018 at 10:03:38AM -0500, Dirk Eddelbuettel wrote:
> 
> On 2 May 2018 at 14:41, Lumin wrote:
> | Seems that things are getting more complicated. Recall that here we'are
> | going to prevent users from GPL violation in situations such as this
> | one:
> | 
> |   debootstrap; apt install libmkl-rt; apt install octave; octave ... (1)
> 
> Are you sure? I do not think that is correct. Downloading and installating
> MKL, and running it with R or Octave (or any other package linking the BLAS
> interface) does not constitute a GPL violation AFAIK.

Not sure. I admit that I'm not good at long licences such as GPL.
Whether it violates GPL or not, what we do in config/postinst won't
change: make sure it's the user's explicit choice to use MKL as the
default BLAS/LAPACK implementation, and in contrast, MKL will not be
used without the explicit choice.

> (There may well be limitations on further redistribution of the aggregate
> even though even the MKL now limits redistribution as it is of course still a
> no-source-code piece of software.)

Intel's ISSL license allows redistribution, as long as no file is
changed.

> Thanks for all your work on this though. Much appreciated.
> 
> Dirk
> 
> -- 
> http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Dirk Eddelbuettel]

On 2 May 2018 at 14:41, Lumin wrote:
| Seems that things are getting more complicated. Recall that here we'are
| going to prevent users from GPL violation in situations such as this
| one:
| 
|   debootstrap; apt install libmkl-rt; apt install octave; octave ... (1)

Are you sure? I do not think that is correct. Downloading and installating
MKL, and running it with R or Octave (or any other package linking the BLAS
interface) does not constitute a GPL violation AFAIK.

(There may well be limitations on further redistribution of the aggregate
even though even the MKL now limits redistribution as it is of course still a
no-source-code piece of software.)

Thanks for all your work on this though. Much appreciated.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

Bug#897238: Re: Bits about Intel MKL packaging -- Higher Priority than OpenBLAS

[Lumin]

Hi Sébastien,

> > Since libmkl-rt Provides libblas.so.3, it is not totally safe even
> > if we set its priority to 1. That's because MKL will still be selected
> > when there is only one libblas.so.3 provider, namely MKL. Due to
> > this reason, I appended libopenblas-base as a dependency of libmkl-rt,
> > so that MKL will never be selected by update-alternatives in auto mode.
> > (I'm not sure what will happen if a package which provides libblas.so.3
> > and also depends libblas.so.3 . Let me simply depend on openblas
> > instead.)
> 
> I had overlooked this corner case, thanks.
> 
> It should be however very rare, because all packages that depend on a
> BLAS/LAPACK implementation use something like "Depends: libblas3 | 
> libblas.so.3"
> (i.e. a concrete package before the virtual one). But it can indeed happen if
> the user first selects MKL then the depending package.
> 
> I think depending on openblas is a bit weird. If the users reply "no" and MKL
> is the only alternative available, then I'd rather display a debconf message
> warning the user about the situation.
> 
> The test about MKL being the only option could be something like:
> 
> if [ $(update-alternatives --query libblas.so.3-@DEB_HOST_MULTIARCH@ | grep 
> ^Alternative: | wc -l) = 1 ]

Seems that things are getting more complicated. Recall that here we'are
going to prevent users from GPL violation in situations such as this
one:

  debootstrap; apt install libmkl-rt; apt install octave; octave ... (1)

Another solution which avoids directly depending on OpenBLAS came up in
my mind, which looks like:

  libmkl-rt --
  Pre-Depends: libblas3 | libblas.so.3; Provides: libblas.so.3   ... (2)

The solution (2) will keep the user safe in situation (1). However while
having a discussion with a Debian Maintainer, a bug of solution (2) was
pointed out by him:

  debootstrap; apt install libmkl-rt; apt purge libblas3; octave ... (3)

In situation (3), MKL is still the only provider of libblas.so.3, and
GPL will be violated even if the user answered "NO" when installing MKL.
If we are going to find a solution for (3), things will be even more
complex ...

He put forward a simpler solution: Just don't provide libblas.so.3, such
that MKL will never be used to satisfy the dependency of libblas.so.3 .
Based on his idea, my new solution is the follows:

  libmkl-rt --
  Depends: libblas3 | libblas.so.3
  Provides: NOTHING  ... (4)

So it's totally safe now. If there is MKL, there must be a free
libblas.so.3 implementation with a priority definitely larger than 1,
overriding MKL in terms of auto-mode alternatives. On the other hand,
if that alternative is manually set, then there is nothing to worry
about. This solution is also able to resolve problems found in (1) and (3).

> 
> The test about MKL being the only option could be something like:
> 
> if [ $(update-alternatives --query libblas.so.3-@DEB_HOST_MULTIARCH@ | grep 
> ^Alternative: | wc -l) = 1 ]

I think this is not needed if we use solution (4). MKL will never be
the only libblas.so.3 "provider" existing on system.

Apart from all things discussed above, there is upstream confirmation
to the ambiguous license declaration in several headers. See [1]

The blockers are cleared. I think I'll update the package as proposed,
and the copyright information as said in [1] before this weekend.

[1] https://github.com/intel/mkl-dnn/issues/206#issuecomment-385772103

Regards,
Lumin


signature.asc
Description: PGP signature