Re: VTK and ParaView parallelization

[Anton Gladky]

Hi François,

thanks for a suggestion! I have added you to Debian Science Team on salsa.
Feel free to contribute!

Best regards


Anton


Am Do., 22. Apr. 2021 um 21:46 Uhr schrieb François Mazen :

> Hello,
>
> I'm wondering why VTK and ParaView are build with the "Sequential"
> default backend for parallelization (vtkSMPTools backend)?
> This makes ParaView run slower compared to official binary releases
> from Kitware, which are built with Threaded Building Blocks (TBB).
>
> Hence, any serious user of ParaView and VTK will have to use a custom
> build, losing all the advantages of Debian packaging.
>
> It seems that TBB is available as Debian packages. Do you think that
> the Debian package may be built with TBB?
> The modification is simple: add -DVTK_SMP_IMPLEMENTATION_TYPE="TBB" to
> the debian/rules file, and add tbb as dependency in debian/control
> file.
>
> (By the way, I'm DM and I'll be happy to be part of the Debian Science
> Team to contribute to the tools that I use everyday, mostly linked to
> scientific computation. My salsa login is mzf.)
>
> Best Regards,
> François
>
>
>

VTK and ParaView parallelization

[François Mazen]

Hello,

I'm wondering why VTK and ParaView are build with the "Sequential"
default backend for parallelization (vtkSMPTools backend)?
This makes ParaView run slower compared to official binary releases
from Kitware, which are built with Threaded Building Blocks (TBB).

Hence, any serious user of ParaView and VTK will have to use a custom
build, losing all the advantages of Debian packaging.

It seems that TBB is available as Debian packages. Do you think that
the Debian package may be built with TBB?
The modification is simple: add -DVTK_SMP_IMPLEMENTATION_TYPE="TBB" to
the debian/rules file, and add tbb as dependency in debian/control
file.

(By the way, I'm DM and I'll be happy to be part of the Debian Science
Team to contribute to the tools that I use everyday, mostly linked to
scientific computation. My salsa login is mzf.)

Best Regards,
François




signature.asc
Description: This is a digitally signed message part

Re: Packaging xsimd

[Drew Parsons]

As far as --march=native goes, in my opinion it's ok since it's only 
used in tests, i.e. testing that the files build and run.
This is because it's actually a header-only package, no binaries 
(executables or libraries) are provided. So the package that we ship 
hasn't been "built" with --march=native, it's only been tested with it.


I suggest not explicitly banning the failing arches at this point.  They 
shouldn't hold up migration to testing (after the stable release, that 
is) since they haven't previously been provided.  The migration ban on 
failed builds applies when a package has previously built successfully, 
so it doesn't apply to new packages.  Having the build logs for failed 
builds can be useful for tracking what their problem is, and monitor 
progress when the builds are fixed.


The exception might be if the package explicitly pulls in machine 
language for specific arches, which can be done from C, in which case 
the package is designed to not build on other arches.  In that case it 
makes more sense to ban the failing arches, if it's known they can never 
work (until explicitly ported).


On that note, not all arches have the --march=native flag anyway.  
CMakeFiles.txt in xtensor tests if the flag is available when 
configuring tests. So other arches might build if that test and that 
conditional configuration is patched in.


Drew


On 2021-04-22 19:30, Nilesh Patra wrote:

On Thu, 22 Apr 2021 at 22:06, Julien Lamy  wrote:


Le 22/04/2021 à 18:29, Nilesh Patra a écrit :



On Thu, 22 Apr 2021 at 21:38, Julien Lamy mailto:l...@unistra.fr>> wrote:

No, it can work with only SSE and only SSE2 enabled, which

would match

the baseline (tested with a non-AVX machine, I could not get

my

hands on
a non-SSE2 box).



* Is this portable to arches other than x86 and arm?


Theoretically yes: there is a fallback mode which codes the

SIMD

instructions as loops. I have not tested it in a non-x86 and

non-ARM

environment.


I will try testing it. It looks unlikely since it seems to need a

native

architecture. Likely build time tests will
not work, but I'll check nevertheless


Thanks, I'll be curious of the results.


I tried in a ppc64el porter box, and I get several of:

/home/nilesh/xsimd/xsimd/test/test_batch_bool.cpp:315:1: error:
'gtest_type_params_batch_bool_test_' was not declared in this scope;
did you mean 'gtest_type_params_batch_bool_test_NameGenerator'?
  315 | TYPED_TEST(batch_bool_test, load_store)
  | ^~
/home/nilesh/xsimd/xsimd/test/test_batch_bool.cpp:315:1: error:
template argument 3 is invalid
  315 | TYPED_TEST(batch_bool_test, load_store)
  | ^~
/home/nilesh/xsimd/xsimd/test/test_batch_bool.cpp:315:1: error:
'gtest_type_params_batch_bool_test_' was not declared in this scope;
did you mean 'gtest_type_params_batch_bool_test_NameGenerator'?
  315 | TYPED_TEST(batch_bool_test, load_store)
  | ^~

And a failing build. Both for build time as well as autopkgtests.
Do you think we should for now limit arches to amd64 i386 and arm64 in
d/control for now?




* Readme has instructions to build documentation, and you

have added

Build Depends as well, to build it. However they are

neither

built nor

installed.
If you think building and installing docs make sense,

could

you fix

it? Please install docs in a separate binary package if so.


Done.

I've also bumped the version to 7.5.0, released yesterday

while I was

packaging :)


I do not see your changes on salsa[1] - did you forget to push in

any case?

Yes :( Fixed now.


I have following comments to make:

* Why is the package named xsimd-dev instead of libxsimd-dev? It might
match xtensor, but AFAICS that's
against the library style packaging. For ref:
https://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html

* Please commit v7.5.0 to pristine-tar

* Some files in ./include have excerpts from code that belongs to
Boost Software license. For example:
./include/xsimd/math/xsimd_error.hpp
   This should be mentioned in d/cpopyright
   Also, this file: ./test/test_constant_batch.cpp has a different
copyright holder "Serge Guelton and QuantStack" so this should also be
mentioned explicitly with
   Files: ./test/test_constant_batch.cpp
   Copyright: Serge Guelton and QuantStack
   ..
   This is not exhaustive, please consider doing a scrutiny for the
entire codebase and mention copyrights for any files that differ/have
different license

* Autopkgtest on salsa CI fails, consider fixing it:
https://salsa.debian.org/science-team/xsimd/-/jobs/1599080

Nilesh

Re: Packaging xsimd

[Nilesh Patra]

On Thu, 22 Apr 2021 at 22:06, Julien Lamy  wrote:

> Le 22/04/2021 à 18:29, Nilesh Patra a écrit :
> >
> >
> > On Thu, 22 Apr 2021 at 21:38, Julien Lamy  > > wrote:
> >
> > No, it can work with only SSE and only SSE2 enabled, which would
> match
> > the baseline (tested with a non-AVX machine, I could not get my
> > hands on
> > a non-SSE2 box).
> >
> >
> >  > * Is this portable to arches other than x86 and arm?
> >
> > Theoretically yes: there is a fallback mode which codes the SIMD
> > instructions as loops. I have not tested it in a non-x86 and non-ARM
> > environment.
> >
> >
> > I will try testing it. It looks unlikely since it seems to need a native
> > architecture. Likely build time tests will
> > not work, but I'll check nevertheless
>
> Thanks, I'll be curious of the results.
>

I tried in a ppc64el porter box, and I get several of:

/home/nilesh/xsimd/xsimd/test/test_batch_bool.cpp:315:1: error:
'gtest_type_params_batch_bool_test_' was not declared in this scope; did
you mean 'gtest_type_params_batch_bool_test_NameGenerator'?
  315 | TYPED_TEST(batch_bool_test, load_store)
  | ^~
/home/nilesh/xsimd/xsimd/test/test_batch_bool.cpp:315:1: error: template
argument 3 is invalid
  315 | TYPED_TEST(batch_bool_test, load_store)
  | ^~
/home/nilesh/xsimd/xsimd/test/test_batch_bool.cpp:315:1: error:
'gtest_type_params_batch_bool_test_' was not declared in this scope; did
you mean 'gtest_type_params_batch_bool_test_NameGenerator'?
  315 | TYPED_TEST(batch_bool_test, load_store)
  | ^~

And a failing build. Both for build time as well as autopkgtests.
Do you think we should for now limit arches to amd64 i386 and arm64 in
d/control for now?


> >
> >  > * Readme has instructions to build documentation, and you have
> added
> >  > Build Depends as well, to build it. However they are neither
> > built nor
> >  > installed.
> >  > If you think building and installing docs make sense, could
> > you fix
> >  > it? Please install docs in a separate binary package if so.
> >
> > Done.
> >
> > I've also bumped the version to 7.5.0, released yesterday while I was
> > packaging :)
> >
> >
> > I do not see your changes on salsa[1] - did you forget to push in any
> case?
>
> Yes :( Fixed now.
>

I have following comments to make:

* Why is the package named xsimd-dev instead of libxsimd-dev? It might
match xtensor, but AFAICS that's
against the library style packaging. For ref:
https://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html

* Please commit v7.5.0 to pristine-tar

* Some files in ./include have excerpts from code that belongs to Boost
Software license. For example: ./include/xsimd/math/xsimd_error.hpp
   This should be mentioned in d/cpopyright
   Also, this file: ./test/test_constant_batch.cpp has a different
copyright holder "Serge Guelton and QuantStack" so this should also be
mentioned explicitly with
   Files: ./test/test_constant_batch.cpp
   Copyright: Serge Guelton and QuantStack
   ..
   This is not exhaustive, please consider doing a scrutiny for the entire
codebase and mention copyrights for any files that differ/have different
license

* Autopkgtest on salsa CI fails, consider fixing it:
https://salsa.debian.org/science-team/xsimd/-/jobs/1599080

Nilesh

Re: Packaging xsimd

[Nilesh Patra]

On Thu, 22 Apr 2021 at 21:38, Julien Lamy  wrote:

> No, it can work with only SSE and only SSE2 enabled, which would match
> the baseline (tested with a non-AVX machine, I could not get my hands on
> a non-SSE2 box).
>

> > * Is this portable to arches other than x86 and arm?
>
> Theoretically yes: there is a fallback mode which codes the SIMD
> instructions as loops. I have not tested it in a non-x86 and non-ARM
> environment.
>

I will try testing it. It looks unlikely since it seems to need a native
architecture. Likely build time tests will
not work, but I'll check nevertheless


> > * Readme has instructions to build documentation, and you have added
> > Build Depends as well, to build it. However they are neither built nor
> > installed.
> > If you think building and installing docs make sense, could you fix
> > it? Please install docs in a separate binary package if so.
>
> Done.
>
> I've also bumped the version to 7.5.0, released yesterday while I was
> packaging :)
>

I do not see your changes on salsa[1] - did you forget to push in any case?

[1]: https://salsa.debian.org/science-team/xsimd

Nilesh

Re: Packaging xsimd

[Julien Lamy]

Le 22/04/2021 à 13:29, Nilesh Patra a écrit :



On Thu, 22 Apr 2021 at 15:39, Julien Lamy > wrote:



Good to know. I've run salsa-ci on Gitlab.com and everything comes out
OK. If the package looks good to you, I'd welcome the sponsored upload;
the package being a header-only C++ library, the build should take less
than a minute.


I enabled Build-Time tests with nocheck, it seems to pass. However a 
couple of questions:


* The Readme here[1] says that it needs -march=native to be able to use 
this. This is a baseline violation.
    It is also a problem if build-time tests run on a baseline amd64 
machine. Is it suitable only for CPUs which have AVX?




No, it can work with only SSE and only SSE2 enabled, which would match 
the baseline (tested with a non-AVX machine, I could not get my hands on 
a non-SSE2 box).



* Is this portable to arches other than x86 and arm?


Theoretically yes: there is a fallback mode which codes the SIMD 
instructions as loops. I have not tested it in a non-x86 and non-ARM 
environment.


* Readme has instructions to build documentation, and you have added 
Build Depends as well, to build it. However they are neither built nor 
installed.
    If you think building and installing docs make sense, could you fix 
it? Please install docs in a separate binary package if so.


Done.

I've also bumped the version to 7.5.0, released yesterday while I was 
packaging :)


--
Julien

Re: Packaging xsimd

[Julien Lamy]

Le 22/04/2021 à 12:45, Drew Parsons a écrit :

On 2021-04-22 12:09, Julien Lamy wrote:

>>


Not really. Upload for packages which are not in testing can be done, 
and accpeted.
New packages targeting unstable is absolutely OK. It's just that they 
won't migrate to testing until bullseye is out.
That said, I'll happily sponsor you an upload (I hope the build is 
extremely resource consuming)


Good to know. I've run salsa-ci on Gitlab.com and everything comes out
OK. If the package looks good to you, I'd welcome the sponsored
upload; the package being a header-only C++ library, the build should
take less than a minute.



Thanks Julien.

It's good to run the tests (which might make the build a little longer, 
depending on the tests).


Tests can run in two modes:  build-time, and debci (autopkgtest)  (after 
installation and separate from building, testing the headers as installed).


debci is triggered by setting up debian/tests, which might launch the 
tests.


xtensor gives an example of how to set up tests for a header-only 
package. Both xtensor and xsimd have the BUILD_TESTS option in 
CMakeFiles.txt.


Build-time tests, if they're not already run, could be triggered (via 
override_dh_auto_test in debian/rules, for instance), but adding the 
BUILD_TESTS option to the cmake configuration might be enough for 
build-time tests.


I've followed what you did in xtensor: I've added autopkgtest and 
changed the name of the package from libxsimd-dev to xsimd-dev to match 
xtensor and xtl.


Thanks for the pointer to xtensor.
--
Julien

Re: Packaging xsimd

[Nilesh Patra]

On Thu, 22 Apr 2021 at 15:39, Julien Lamy  wrote:

>
> Good to know. I've run salsa-ci on Gitlab.com and everything comes out
> OK. If the package looks good to you, I'd welcome the sponsored upload;
> the package being a header-only C++ library, the build should take less
> than a minute.
>

I enabled Build-Time tests with nocheck, it seems to pass. However a couple
of questions:

* The Readme here[1] says that it needs -march=native to be able to use
this. This is a baseline violation.
   It is also a problem if build-time tests run on a baseline amd64
machine. Is it suitable only for CPUs which have AVX?

* Is this portable to arches other than x86 and arm?

* Readme has instructions to build documentation, and you have added Build
Depends as well, to build it. However they are neither built nor installed.
   If you think building and installing docs make sense, could you fix it?
Please install docs in a separate binary package if so.

[1]: https://github.com/xtensor-stack/xsimd#readme

Nilesh

Re: Packaging xsimd

[Drew Parsons]

On 2021-04-22 12:09, Julien Lamy wrote:

>>


Not really. Upload for packages which are not in testing can be done, 
and accpeted.
New packages targeting unstable is absolutely OK. It's just that they 
won't migrate to testing until bullseye is out.
That said, I'll happily sponsor you an upload (I hope the build is 
extremely resource consuming)


Good to know. I've run salsa-ci on Gitlab.com and everything comes out
OK. If the package looks good to you, I'd welcome the sponsored
upload; the package being a header-only C++ library, the build should
take less than a minute.



Thanks Julien.

It's good to run the tests (which might make the build a little longer, 
depending on the tests).


Tests can run in two modes:  build-time, and debci (autopkgtest)  (after 
installation and separate from building, testing the headers as 
installed).


debci is triggered by setting up debian/tests, which might launch the 
tests.


xtensor gives an example of how to set up tests for a header-only 
package. Both xtensor and xsimd have the BUILD_TESTS option in 
CMakeFiles.txt.


Build-time tests, if they're not already run, could be triggered (via 
override_dh_auto_test in debian/rules, for instance), but adding the 
BUILD_TESTS option to the cmake configuration might be enough for 
build-time tests.



Drew

Re: Packaging xsimd

[Julien Lamy]

Hi Nilesh

Le 22/04/2021 à 11:23, Nilesh Patra a écrit :

Hi Julien,

On Thu, 22 Apr 2021 at 11:51, Julien Lamy > wrote:


Dear all,
I'm working on a project which depends on xsimd, a C++ wrapper for SIMD
instructions (https://github.com/xtensor-stack/xsimd/
) and noticed that
there is an old RFP for it (#867915), mentioning a maintenance by
debian-science. I've created a first draft for this package and pushed
it on Salsa (https://salsa.debian.org/lamy-guest/xsimd
): if you're
interested, I'd be glad to join the team and maintain xsimd in
coordination with debian-science.


I have added you to the science team, please push your changes to the 
science-team namespace.

Welcome onboard \o/


Thanks! The full repo (including upstream and pristine-tar branches) is 
now on science-team/xsimd.




I'm not a DM, although I have done
previous packaging work with the Debian-Med team, and I realize that
the
hard freeze makes this very low priority and will prevent any upload of
this package before the effective release of Bullseye.


Not really. Upload for packages which are not in testing can be done, 
and accpeted.
New packages targeting unstable is absolutely OK. It's just that they 
won't migrate to testing until bullseye is out.
That said, I'll happily sponsor you an upload (I hope the build is 
extremely resource consuming)


Good to know. I've run salsa-ci on Gitlab.com and everything comes out 
OK. If the package looks good to you, I'd welcome the sponsored upload; 
the package being a header-only C++ library, the build should take less 
than a minute.


--
Julien



OpenPGP_signature
Description: OpenPGP digital signature

Re: Packaging xsimd

[Drew Parsons]

On 2021-04-22 11:23, Nilesh Patra wrote:

Hi Julien,

On Thu, 22 Apr 2021 at 11:51, Julien Lamy  wrote:


Dear all,
I'm working on a project which depends on xsimd, a C++ wrapper for
SIMD

...

I have added you to the science team, please push your changes to the
science-team namespace.
Welcome onboard \o/

...

Upload for packages which are not in testing can be done,
and accpeted.
New packages targeting unstable is absolutely OK. It's just that they
won't migrate to testing until bullseye is out.
That said, I'll happily sponsor you an upload (I hope the build is
extremely resource consuming)



Thanks Nilesh, and thanks Julien.

It will be nice to have xsimd available.  xtensor should be able to use 
it.


Drew

Packaging xsimd

[Julien Lamy]

Dear all,
I'm working on a project which depends on xsimd, a C++ wrapper for SIMD 
instructions (https://github.com/xtensor-stack/xsimd/) and noticed that 
there is an old RFP for it (#867915), mentioning a maintenance by 
debian-science. I've created a first draft for this package and pushed 
it on Salsa (https://salsa.debian.org/lamy-guest/xsimd): if you're 
interested, I'd be glad to join the team and maintain xsimd in 
coordination with debian-science. I'm not a DM, although I have done 
previous packaging work with the Debian-Med team, and I realize that the 
hard freeze makes this very low priority and will prevent any upload of 
this package before the effective release of Bullseye.


Cheers,
--
Julien Lamy, Ph.D.
Research engineer
ICube Lab, University of Strasbourg-CNRS



OpenPGP_signature
Description: OpenPGP digital signature