Re: hyphy - amd64 Test failures

[Juhani Numminen]

Hi Nilesh and Michael,

On Sat, 30 Jan 2021, 23.24 Nilesh Patra  wrote:

> Hi Michael,
>
> Thanks for your work on hyphy.
> It seems that with your last upload, CI tests are failing on amd64, but
> working everywhere else, as can be seen on tracker here[1]
>
> The log for amd64 on debci[2] seems to have compliance with salsa CI[3]
> I see the binaries generated as desired on corresponding buildd log,
>

The build logs lead me to believe something is missing though.

The target override_dh_install-arch attempts to install simd-dispatch into
binpkg hyphy-common. Now consider that the buildd network builds the
arch-independent binary packages separately. During arch-indep build, the
"-arch" suffixed target is ignored, so simd-dispatch perhaps surprisingly
does not end up in hyphy-common.


Regards,
Juhani

hyphy - amd64 Test failures

[Nilesh Patra]

Hi Michael,

Thanks for your work on hyphy.
It seems that with your last upload, CI tests are failing on amd64, but
working everywhere else, as can be seen on tracker here[1]

The log for amd64 on debci[2] seems to have compliance with salsa CI[3]
I see the binaries generated as desired on corresponding buildd log, and
I'm not sure about what exactly triggers this.

Is it in some way related to hyphy-common being marked as "M-A: foreign"?
Please let me know.


[1]: https://tracker.debian.org/pkg/hyphy
[2]:
https://ci.debian.net/data/autopkgtest/testing/amd64/h/hyphy/10128903/log.gz
[3]: https://salsa.debian.org/med-team/hyphy/-/jobs/1393102

Thanks and regards
Nilesh

Re: [help] porting brian on MIPS and POWER, or dropping support

[Michael Crusoe]

On Sat, 30 Jan 2021 at 17:18, Étienne Mollier 
wrote:

> Hi Michael,
>
> Michael Crusoe, on 2021-01-30 16:51:27 +0100:
> > On Sat, 30 Jan 2021 at 14:33, Étienne Mollier <
> etienne.moll...@mailoo.org>
> > wrote:
>
> > > [1] https://salsa.debian.org/med-team/brian/-/tree/master
> >
> >
> https://salsa.debian.org/med-team/brian/-/blob/master/debian/patches/gsl-compiler-arg.patch#L29
> >
> > -march=native is not allowed as it violates the architecture baseline,
> > unless there is an alternative binary or library built without it.
>
> I would generally agree, but in this case I understood the code
> was compiled (in the background) on the end user's machine[3].
> I don't exclude that I could have misunderstood the mechanism,
> so am triple checking at the moment.  I probably should have put
> the link in the patch header, by the way.
>
> [3] https://brian2.readthedocs.io/en/stable/user/computation.html


If it is truly for local use, then -march=native is great, but not every
architecture supports "-march=native", so you may need to add work arounds
for archs that don't have that gcc option.

Here is the result of some reading of the gcc manual page

-march=native & -mtune=native is valid in gcc for the following Debian
archs:

arm*
mips*
s390x
i386/amd64/x32

but not these

HPPA
M68k
riscv64
power*/ppc* (no -march at all, but does have -mcpu=native and -mtune=native)
alpha (no -march at all, but does have -mcpu=native and -mtune=native)
sh4 (no -march at all, no -mcpu either)
sparc64 (no -march at all, but does have -mcpu=native and -mtune=native)
ia64 (no -march, has -mtune but not -mtune=native)


>
> In any way, I guess putting more regular compiler flags at first
> should give a far safer outcome than current architecture
> dependent selection.
>
> Thanks for your review!  :)
>

You are welcome

-- 
Michael R. Crusoe

Re: [help] porting brian on MIPS and POWER, or dropping support

[Étienne Mollier]

Hi Nilesh,

Nilesh Patra, on 2021-01-30 22:57:55 +0530:
> Unfortunately, ssh'ing on "mips64el" machines is resulting in timeouts
> somehow - no idea there. I can however ssh to mips(32)el machines.
> 
> That being said, I built it on a ppc64el porter box, and the build works
> fine there. Attaching the log for reference. It probably didn't cross build
> for you, as per earlier mail.
> I suggest that you upload this then, and we simply file removal bugs in
> case it still doesn't work on mips64 (although I smell another false
> positive here)
> 
> Let me know what you think about this.

It looks like I'm getting bitten by the virtualisation layers,
wouldn't be the first time nor the last.  I finish the check
that I triggered to remove risks of baseline violations, run a
routine-update to wrap up the package and will upload; it looks
like it might be worth it.

Many thanks for your time and your additional checks,  :)
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [help] porting brian on MIPS and POWER, or dropping support

[Étienne Mollier]

Hi Michael,

Michael Crusoe, on 2021-01-30 16:51:27 +0100:
> On Sat, 30 Jan 2021 at 14:33, Étienne Mollier 
> wrote:
> The package brian[1] has a mechanism based on the GSL which does
> > some sort of compilation just-in-time.  The default set of build
> > flags works rather well on all flavors of amd64 architecture
> > with or without extensions, however not all of the specified
> > options by default are supported on all architectures.
> >
> 
> Hmm.. this sounds like a recipe for a tough time :-D
> 
> Personally I put no extra effort for mips :-)

I thought so.  Surprizingly, the mipsel 32 bits variant turned
out to work normally, contrary to the 64 bits ISA (at least in
Qemu context).  I opened Bug#981397[2] to proceed to the removal
to buy us some time.

[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=981397

> > [1] https://salsa.debian.org/med-team/brian/-/tree/master
> 
> https://salsa.debian.org/med-team/brian/-/blob/master/debian/patches/gsl-compiler-arg.patch#L29
> 
> -march=native is not allowed as it violates the architecture baseline,
> unless there is an alternative binary or library built without it.

I would generally agree, but in this case I understood the code
was compiled (in the background) on the end user's machine[3].
I don't exclude that I could have misunderstood the mechanism,
so am triple checking at the moment.  I probably should have put
the link in the patch header, by the way.

[3] https://brian2.readthedocs.io/en/stable/user/computation.html

In any way, I guess putting more regular compiler flags at first
should give a far safer outcome than current architecture
dependent selection.

Thanks for your review!  :)
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [help] porting brian on MIPS and POWER, or dropping support

[Michael Crusoe]

On Sat, 30 Jan 2021 at 14:33, Étienne Mollier 
wrote:

> Hi Michael, and people at ease with architecture ports,
>

Hello!

The package brian[1] has a mechanism based on the GSL which does
> some sort of compilation just-in-time.  The default set of build
> flags works rather well on all flavors of amd64 architecture
> with or without extensions, however not all of the specified
> options by default are supported on all architectures.
>

Hmm.. this sounds like a recipe for a tough time :-D

Personally I put no extra effort for mips :-)


> [1] https://salsa.debian.org/med-team/brian/-/tree/master


https://salsa.debian.org/med-team/brian/-/blob/master/debian/patches/gsl-compiler-arg.patch#L29

-march=native is not allowed as it violates the architecture baseline,
unless there is an alternative binary or library built without it.

-- 
Michael R. Crusoe

Re: [help] porting brian on MIPS and POWER, or dropping support

[Étienne Mollier]

Hi Nilesh,

Nilesh Patra, on 2021-01-30 20:44:38 +0530:
> On Sat, 30 Jan, 2021, 8:14 pm Étienne Mollier, 
> wrote:
> > Étienne Mollier, on 2021-01-30 14:33:41 +0100:
> > >   * I curiously couldn't locate mipsel builds in my backups, but
> > > from what I recall I got results very similar to mips64el.
> >
> > The build goes through on mipsel (but not mips64el)!  I did not
> > recall correctly it seems; it might be a clue about what could
> > be wrong.
> >
> 
> That sounds unusual. It's usually the other way round - can it be a false
> positive?
> Should I try this on a mips64 porter box?

Thank you for your offer of testing it on real hardware.  :)

I agree this is very unusual, but that is what I am witnessing
with qemu-user-static at the time indeed.  If you are curious
and have the energy to investigate this, I suppose it would be
at least worth an attempt on real hardware, sure!

Have a nice day,  :)
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [help] porting brian on MIPS and POWER, or dropping support

[Nilesh Patra]

On Sat, 30 Jan, 2021, 8:14 pm Étienne Mollier, 
wrote:

> Hi again,
>
> Nevermind that:
>
> Étienne Mollier, on 2021-01-30 14:33:41 +0100:
> >   * I curiously couldn't locate mipsel builds in my backups, but
> > from what I recall I got results very similar to mips64el.
>
> The build goes through on mipsel (but not mips64el)!  I did not
> recall correctly it seems; it might be a clue about what could
> be wrong.
>

That sounds unusual. It's usually the other way round - can it be a false
positive?
Should I try this on a mips64 porter box?

Nilesh

>

Re: [help] porting brian on MIPS and POWER, or dropping support

[Étienne Mollier]

Hi Andreas,

Andreas Tille, on 2021-01-30 15:31:57 +0100:
> IMHO we should focus on getting the package into testing for those
> architectures that are frequently used (which means temporary removal of
> existing older package versions on architectures where the current
> version might fail to build).
> 
> Once this is done and we have time we can try to care for the rest.

True.  I'm aware of a couple of packages which are currently out
of Testing due to missing architecture build, and I believe I
delayed request for removal for long enough.  Will start with
brian to buy us some time.

Thanks for insisting on that aspect,
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [help] porting brian on MIPS and POWER, or dropping support

[Étienne Mollier]

Hi again,

Nevermind that:

Étienne Mollier, on 2021-01-30 14:33:41 +0100:
>   * I curiously couldn't locate mipsel builds in my backups, but
> from what I recall I got results very similar to mips64el.

The build goes through on mipsel (but not mips64el)!  I did not
recall correctly it seems; it might be a clue about what could
be wrong.

Cheers,  :)
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [help] porting brian on MIPS and POWER, or dropping support

[Andreas Tille]

Hi Étienne,

On Sat, Jan 30, 2021 at 02:33:39PM +0100, Étienne Mollier wrote:
> ...

Thanks a lot for your detailed analysis.
 
> I feel it might be possible to get the software working as
> intended on at least ppc64el, but time is running out with the
> freeze, and brian is still out of Testing to day.  I'm pinging
> in case you might have an idea, but if not, I consider filing a
> request for removal on these architectures by tomorrow.

IMHO we should focus on getting the package into testing for those
architectures that are frequently used (which means temporary removal of
existing older package versions on architectures where the current
version might fail to build).

Once this is done and we have time we can try to care for the rest.

As always thanks a lot for your work on this

Andreas.

-- 
http://fam-tille.de

[help] porting brian on MIPS and POWER, or dropping support

[Étienne Mollier]

Hi Michael, and people at ease with architecture ports,

The package brian[1] has a mechanism based on the GSL which does
some sort of compilation just-in-time.  The default set of build
flags works rather well on all flavors of amd64 architecture
with or without extensions, however not all of the specified
options by default are supported on all architectures.

[1] https://salsa.debian.org/med-team/brian/-/tree/master

The buildd logs[2] give a rather precise picture of the
situation with the existing package fortunately.  Architectures
affected by unrecognized build options are showing the following
symptom:

Skipped: GSL support for numpy has not been implemented yet

[2] https://buildd.debian.org/status/package.php?p=brian

I introduced a patch to try addressing the issue on Debian's
official architectures and on riscv64; build issues different
symptoms on mips, with missing symbols, but reducing
optimization levels seemed to help moving forward.  With these
changes, I got the following results so far:

  * still FTBFS on ppc64el, I see the following build time test
step; the issue looks like it might just be a matter of
doing a small adjustment, but I don't know which one:

==
ERROR: brian2.tests.test_GSL.test_GSL_stateupdater_basic
--
Traceback (most recent call last):
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/core/network.py",
 line 897, in before_run
obj.before_run(run_namespace)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/input/spikegeneratorgroup.py",
 line 200, in before_run
raise ValueError('Using a dt of %s, some neurons of '
ValueError: Using a dt of , 
some neurons of SpikeGeneratorGroup "spikegeneratorgroup" spike more than once 
during a time step.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in 
runTest
self.test(*self.arg)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/tests/test_GSL.py",
 line 19, in wrapped
func()
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/tests/test_GSL.py",
 line 57, in test_GSL_stateupdater_basic
net.run(100*ms)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/core/base.py",
 line 278, in device_override_decorated_function
return func(*args, **kwds)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/units/fundamentalunits.py",
 line 2434, in new_f
result = f(*args, **kwds)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/core/network.py",
 line 1008, in run
self.before_run(namespace)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/core/base.py",
 line 278, in device_override_decorated_function
return func(*args, **kwds)
  File 
"/tmp/autopkgtest.t0d9nq/build.nMH/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/core/network.py",
 line 899, in before_run
raise BrianObjectException("An error occurred when preparing an 
object.", obj) from ex
brian2.core.base.BrianObjectException: Error encountered with object 
named "spikegeneratorgroup".
Object was created here (most recent call only, full details in debug 
log):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in 
runTest
self.test(*self.arg)

  * the test_GSL_stateupdater_basic works on on mips64el, but
rereading the build, there are so many warnings about run
time errors such as below, that I would be wary introducing
the package on that architecture, so it'd be probably better
to not lose much more time here:

[...]
WARNING
/tmp/autopkgtest.Z9EqJI/build.QFw/src/debian/tmp/usr/lib/python3.9/dist-packages/brian2/units/fundamentalunits.py:2186:
 RuntimeWarning: invalid value encountered in square
  deviations = np.nansum((np.log10(floatreps) - 1)**2, axis=0)
 [py.warnings]
WARNING
/usr/lib/python3/dist-packages/numpy/lib/nanfunctions.py:102: RuntimeWarning: 
invalid value encountered in isnan
  mask = np.isnan(a)
 [py.warnings]
[...]

  * I curiously couldn't locate mipsel builds in my backups, but
from what I recall I got results very similar to mips64el.

  * the riscv64 build goes through!  (yay! :)

I feel it might be possible to get 

Re: Reminders about using SIMDe

[Michael R. Crusoe]

On Wed, 27 Jan 2021 at 10:59, Nilesh Patra  wrote:

> Hi Michael
>
> On Wed, 27 Jan 2021 at 14:51, Michael R. Crusoe  wrote:
>
>> Hello,
>>
>> After scheduling a binary rebuild[0] of packages using libsimde-dev
>>
>
> First off, thanks a lot for doing this!
>
>
>> I noticed that there are a few new packages using SIMDe (yay!) but aren't
>> listed on https://wiki.debian.org/SIMDEverywhere
>>
>
> It is at my end, I added simde support to plast, ngmlr and scrappie but
> completely missed adding them to the wiki, sorry about that.
> I actually wanted to know answers of a couple of questions before I open a
> PR for them upstream:
>
> * In the simde wiki, point no.3, it is written to remove -msse3,
> -march=native et al flags, however in a few PRs that you opened up, this[1]
> for example, I do not see this change
>

Yeah, for Debian we need to remove them, but I don't want to enforce that
upstream.


>it is however appended with a -D__SIM_SSE3 - does this allow the
> -msse3 flag to work well? And am I expected to do the same stuff in the
> upstream build system?
>

That is a package specific flag. Most PRs are simpler than that one. This
particular flag was removed later in the PR, so it is no longer needed:
https://github.com/stamatak/standard-RAxML/pull/50/files#diff-1a92a17afd64b6b3dd4626377cd6bed402fbc85276bbcf60581a330fa6936187L5934

The PR is mainly to start the conversation with upstream. Some will be
interested and will adapt and accept the PR. Some upstream maintainers will
ignore it.


> * There is no need to add in arguments: "-DSIMDE_ENABLE_OPENMP
> -fopenmp-simd" into the upstream build system right?
>

That's a gcc specific invocation. If they already have code to
differentiate between gcc, clang, icc, and other compilers, then slipping
it into the gcc section makes sense. Otherwise it can be left out unless
the project explicitly only supports gcc.


> Just to be sure that I'm doing it the right way, could you please review
> the PR I sent for plast[2] (sent this PR last month)?
> (Building the upstream repository locally, it looked okay)
>

Sure!

I apologize if these questions might somehow sound stupid, but I do not
> know SIMDe's working well enough to take a judgement without being clear.
>

Not stupid at all, they are good questions. Feel free to update the wiki
with what you learn.


>
> [1]:
> https://github.com/stamatak/standard-RAxML/pull/50/files#diff-8bf5a567e3d917cc040b7bf12f499844b30961ed930b421b717d25039efdd94aR6
> [2]: https://github.com/PLAST-software/plast-library/pull/8
>
> Thanks a lot,
> Nilesh
>

Re: [RFS] python-cutadapt 3.2-2

[Nilesh Patra]

On Sat, 30 Jan 2021 at 17:21, Étienne Mollier 
wrote:

> Hi Michael,
>
> Michael Crusoe, on 2021-01-30 12:29:07 +0100:
> > You are faster than me! Thanks for fixing this. I granted you upload
> > permissions. Cheers!
>
> Thanks for the DM grants!  The upload is in the pipeline
>

Thanks for this, this was on my table for quite sometime :-)


> FYI, I'm wrapping up a note on the brian package, which FTBFS on
> various architectures including ppc64el, mipsel and mips64el.  I
> found a way enable riscv64 so am confident it might be possible
> to get it to run on officially supported architectures as well,
> but I lack the architecture ports experience to tell for sure.
>

I just granted you DM access for brian.
Thanks for this as well!

Nilesh

Re: [RFS] python-cutadapt 3.2-2

[Étienne Mollier]

Hi Michael,

Michael Crusoe, on 2021-01-30 12:29:07 +0100:
> You are faster than me! Thanks for fixing this. I granted you upload
> permissions. Cheers!

Thanks for the DM grants!  The upload is in the pipeline (but I
realise I forgot to wait for the DM grants acknowledgment, so my
first attempt might get rejected, but don't worry about it.)

FYI, I'm wrapping up a note on the brian package, which FTBFS on
various architectures including ppc64el, mipsel and mips64el.  I
found a way enable riscv64 so am confident it might be possible
to get it to run on officially supported architectures as well,
but I lack the architecture ports experience to tell for sure.
More details later today.

Cheers!  :)
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [RFS] python-cutadapt 3.2-2

[Michael Crusoe]

Hey Étienne!

You are faster than me! Thanks for fixing this. I granted you upload
permissions. Cheers!

On Sat, 30 Jan 2021 at 11:51, Étienne Mollier 
wrote:

> Good day,
>
> I enabled i386 support on python-cutadapt 3.2-2.  This should
> allow the package to migrate to Testing.  Changes are available
> on Salsa[1].
>
> [1] https://salsa.debian.org/med-team/python-cutadapt/
>
> Please consider upload or DM grants.
>
> Have a nice day,  :)
> --
> Étienne Mollier 
> Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
> Sent from /dev/pts/2, please excuse my verbosity.
>


-- 
Michael R. Crusoe
Co-founder & Lead,
Common Workflow Language project 
https://orcid.org/-0002-2961-9670
m...@commonwl.org

[RFS] python-cutadapt 3.2-2

[Étienne Mollier]

Good day,

I enabled i386 support on python-cutadapt 3.2-2.  This should
allow the package to migrate to Testing.  Changes are available
on Salsa[1].

[1] https://salsa.debian.org/med-team/python-cutadapt/

Please consider upload or DM grants.

Have a nice day,  :)
-- 
Étienne Mollier 
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.


signature.asc
Description: PGP signature