Re: Aw: Re: Help stabilising mini-gmp

[Vincent Lefevre]

On 2016-12-02 12:18:25 +0100, Wolf Lammen wrote:
> Gesendet: Freitag, 02. Dezember 2016 um 01:30 Uhr
> Von: "Vincent Lefevre" 
> An: "Nelson H. F. Beebe" 
> Cc: gmp-devel@gmplib.org
> Betreff: Re: Help stabilising mini-gmp
[...]
> > I'm not talking about a zero shift count, but a shift of the value 0
> > with an arbitrary shift count, e.g. (uint64_t) 0 << 64. This is
> > undefined behavior, but I wonder why. When mapped to a hardware
> > instruction, does the result depend on the platform?
> 
> I wonder whether the possibly multiple encodings of 0 is the reason.
> We are used to two's complement encoding of integers, but the C/C++
> standard allows other (from the today's point of view exotic)
> encodings like for example one's complement. And here 0 can have the
> encoding all bits clear or all bits set. Now left shift  ...

No, this is not the reason. For instance, 0 << 1 is well-defined
and the result is 0. So, on such a processor, the compiler would
have to make sure that 0 is represented as a positive 0 (all bits
clear) before the hardware shift is done. Then the fact that this
should work with valid shift counts implies that this will still
work with out-of-range shift counts: the register with all bits
clear will always give a value with all bits clear, whatever the
shift count.

-- 
Vincent Lefèvre  - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Aw: Re: Help stabilising mini-gmp

[Wolf Lammen]

Gesendet: Freitag, 02. Dezember 2016 um 01:30 Uhr
Von: "Vincent Lefevre" 
An: "Nelson H. F. Beebe" 
Cc: gmp-devel@gmplib.org
Betreff: Re: Help stabilising mini-gmp
On 2016-12-01 16:42:39 -0700, Nelson H. F. Beebe wrote:
> Vincent Lefevre  asks in response to my lengthy
> posting earlier today about where the behavior of bit-shift operations
> allows undefined behavior in C and C++:
>
> >> ...
> >> Is this true even when the shifted value is zero? For some time,
> >> I've wondered why this was also undefined for zero (in MPFR, we
> >> had code where the shift count could be out of range only for
> >> the value 0).
> >> ...
>
> No, zero shifts should be perfectly legal, because they just amount to
> a no-op, and thus a copy of source to destination. The hardware
> manuals that I checked today all permit a zero shift count: it would
> take wasteful extra chip circuitry to add a check for such a case.

> I'm not talking about a zero shift count, but a shift of the value 0
> with an arbitrary shift count, e.g. (uint64_t) 0 << 64. This is
> undefined behavior, but I wonder why. When mapped to a hardware
> instruction, does the result depend on the platform?

I wonder whether the possibly multiple encodings of 0 is the reason. We are 
used to
two's complement encoding of integers, but the C/C++ standard allows other (from
the today's point of view exotic) encodings like for example one's complement. 
And
here 0 can have the encoding all bits clear or all bits set. Now left shift 

...

regards

Wolf Lammen

___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Nelson H. F. Beebe]

Vincent clarifies about my response on about zero shift counts:

>> ...
>> I'm not talking about a zero shift count, but a shift of the value 0
>> with an arbitrary shift count, e.g. (uint64_t) 0 << 64. This is
>> undefined behavior, but I wonder why. When mapped to a hardware
>> instruction, does the result depend on the platform?
>> ...

It isn't the issue of a zero value being shifted, but rather, the
range of allowable shift values.  The ISO 1999 C Standard's relevent
restriction is:

>> ...
>> If the value of the right operand is negative or is greater than or
   ^^
>> equal to the width of the promoted left operand, the behavior is
   ^^^
>> undefined.
>> ...

That is, with an n-bit word, shift counts from 0 to n - 1 are legal,
but negative shift counts, and counts of n or larger, are undefined
behavior.

The reason is that hardware implementations do different things: as my
first message today pointed out, some encode the shift count in the
instruction in an n-bit field, and that field can only hold values
from 0 to n - 1.  Requesting a shift of n bits would very likely mean
that the stored shift count would be the low-order n bits, a value
that is 0.  For shifts larger than n, the effective shift would be
encoded as n mod (2**n), so n + 1 would become 1, n + 2 would become
2, and so on.

Such shifts would be extremely surprising to a programmer who did not
understand what is really happening, because humans tend naturally to
think of exact (infinite precision) integer arithmetic: surely a
33-bit left shift of a 32-bit word should just clear that word to
zero!  No, it does not on many systems: it instead becomes a 1-bit
left shift.

Thus, the ISO Standards committee properly defines n (and higher) bit
shifts to be undefined.

In an ideal world, all compilers would at least have a option to do
compile-time tests of constant shifts, and force run-time tests of
variable shifts, so that such behavior could be caught, diagnosed, and
repaired.

A few compilers now do just that.  Recent gcc compilers provide this
info-node documentation:

>> ...
>> '-fsanitize=undefined'
>>  Enable UndefinedBehaviorSanitizer, a fast undefined behavior
>>  detector.  Various computations will be instrumented to detect
>>  undefined behavior at runtime.  Current suboptions are:
>> 
>>  '-fsanitize=shift'
>>   This option enables checking that the result of a shift
>>   operation is not undefined.  Note that what exactly is
>>   considered undefined differs slightly between C and C++, as
>>   well as between ISO C90 and C99, etc.
>>  ... and many more ...
>> ...
 
Recent clang compilers provide

>> ...
>>  -fsanitize=undefined: UndefinedBehaviorSanitizer, a fast and
>>  compatible undefined behavior checker.
>>  [from http://clang.llvm.org/docs/UsersManual.html]
>> 
>>  -fsanitize=shift: Shift operators where the amount shifted is
>>  greater or equal to the promoted bit-width of the left hand
>>  side or less than zero, or where the left hand side is
>>  negative. For a signed left shift, also checks for signed
>>  overflow in C, and for unsigned overflow in C++. You can use
>>  -fsanitize=shift-base or -fsanitize=shift-exponent to check
>>  only left-hand side or right-hand side of shift operation,
>>  respectively.
>>  [from 
>> http://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#ubsan-checks]
>> ...

---
- Nelson H. F. BeebeTel: +1 801 581 5254  -
- University of UtahFAX: +1 801 581 4148  -
- Department of Mathematics, 110 LCBInternet e-mail: be...@math.utah.edu  -
- 155 S 1400 E RM 233   be...@acm.org  be...@computer.org -
- Salt Lake City, UT 84112-0090, USAURL: http://www.math.utah.edu/~beebe/ -
---
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Vincent Lefevre]

On 2016-12-01 18:15:54 -0700, Nelson H. F. Beebe wrote:
> Vincent clarifies about my response on about zero shift counts:
> 
> >> ...
> >> I'm not talking about a zero shift count, but a shift of the value 0
> >> with an arbitrary shift count, e.g. (uint64_t) 0 << 64. This is
> >> undefined behavior, but I wonder why. When mapped to a hardware
> >> instruction, does the result depend on the platform?
> >> ...
> 
> It isn't the issue of a zero value being shifted, but rather, the
> range of allowable shift values.

Yes, but *why* also for zero?

> The ISO 1999 C Standard's relevent restriction is:
> 
> >> ...
> >> If the value of the right operand is negative or is greater than or
>^^
> >> equal to the width of the promoted left operand, the behavior is
>^^^
> >> undefined.
> >> ...
> 
> That is, with an n-bit word, shift counts from 0 to n - 1 are legal,
> but negative shift counts, and counts of n or larger, are undefined
> behavior.
> 
> The reason is that hardware implementations do different things:

I agree for nonzero values, but I repeat: What about 0?

With your examples, 0 shifted by an arbitrary shift count always
gives 0. So, I don't see any valid reason why the C standard chose
to make this undefined behavior in the particular case of 0. This
makes code like:

  n = number of leading 0's of u;
  v = u << n;

undefined for u = 0.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Vincent Lefevre]

On 2016-12-01 16:42:39 -0700, Nelson H. F. Beebe wrote:
> Vincent Lefevre  asks in response to my lengthy
> posting earlier today about where the behavior of bit-shift operations
> allows undefined behavior in C and C++:
> 
> >> ...
> >> Is this true even when the shifted value is zero? For some time,
> >> I've wondered why this was also undefined for zero (in MPFR, we
> >> had code where the shift count could be out of range only for
> >> the value 0).
> >> ...
> 
> No, zero shifts should be perfectly legal, because they just amount to
> a no-op, and thus a copy of source to destination.  The hardware
> manuals that I checked today all permit a zero shift count: it would
> take wasteful extra chip circuitry to add a check for such a case.

I'm not talking about a zero shift count, but a shift of the value 0
with an arbitrary shift count, e.g. (uint64_t) 0 << 64. This is
undefined behavior, but I wonder why. When mapped to a hardware
instruction, does the result depend on the platform?

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Nelson H. F. Beebe]

Vincent Lefevre  asks in response to my lengthy
posting earlier today about where the behavior of bit-shift operations
allows undefined behavior in C and C++:

>> ...
>> Is this true even when the shifted value is zero? For some time,
>> I've wondered why this was also undefined for zero (in MPFR, we
>> had code where the shift count could be out of range only for
>> the value 0).
>> ...

No, zero shifts should be perfectly legal, because they just amount to
a no-op, and thus a copy of source to destination.  The hardware
manuals that I checked today all permit a zero shift count: it would
take wasteful extra chip circuitry to add a check for such a case.

Also, mathematically, excluding zero in a number system is always a
bad idea: I often ask colleagues with children just learning to speak:
will you teach them to count 1, 2, 3, ..., or 0, 1, 2, 3.  The former
was a mistake that humans made for millenia, until about 1200, when
zero made it to Europe from India in Fibonacci's Liber Abbaci.  Even
the 800 years since then have not improved the arithmetical behavior
of most humans.

The 1999 ISO C Standard makes no mention of a zero shift-count
exception.  The 2003 ISO C++ Standard has essentially equivalent
descriptions.

I cannot see any way that a prohibition on zero shift counts could be
added to any programming language: consider code that unpacks a
bitfield from an opaque (to the programmer, but not to the compiler)
data structure: new releases of software could change the bitfield
location, and the shift counts needed to unpack it.

Harbison & Steele's 5th edition of ``C: A Reference Manual'' on page
232 report about shift operators:

>> The right operand may be 0, in which case no shift occurs, and the
>> result value is identical to the value of the converted left
>> operand.

Their five editions (1984 to 2002) with that title are deservedly
famous for the care in which they treat portability, behavioral, and
Standards-conformance issues in C.

The ISO/IEC 1539-1 Fortran standard in its specification of the
compiler instrinsic function for (unsigned aka logical) bit shifting
says:

>> ...
>> 13.14.50 ISHFT (I, SHIFT)
>>   Description.Performs a logical shift.
>>   Class.  Elemental function.
>>   Arguments.
>>   I   shall be of type integer.
>>   SHIFT   shall be of type integer.  The absolute value 
>>   of SHIFT shall be less than or equal to
>>   BIT-SIZE (I).
>>   Result Characteristics. Same  as I.
>>   Result Value.   The result has the value obtained by
>>   shifting the bits of I by SHIFT
>>   positions.  If SHIFT is positive, the
>>   shift is to the left; if SHIFT is
>>   negative, the shift is to the right;
>>   and if SHIFT is zero, no shift is
>>   performed.  Bits shifted out from the
>>   left or from the right, as
>>   appropriate, are lost.  Zeros are
>>   shifted in from the opposite end.
>>   The model for the interpretation of
>>   an integer value as a sequence of
>>   bits is in 13.5.7.
>> 
>>   Example. ISHFT  (3,1)  has  the  result  6.
>> ...

Notice that it explicitly permits zero shift counts.

---
- Nelson H. F. BeebeTel: +1 801 581 5254  -
- University of UtahFAX: +1 801 581 4148  -
- Department of Mathematics, 110 LCBInternet e-mail: be...@math.utah.edu  -
- 155 S 1400 E RM 233   be...@acm.org  be...@computer.org -
- Salt Lake City, UT 84112-0090, USAURL: http://www.math.utah.edu/~beebe/ -
---
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Vincent Lefevre]

On 2016-11-25 18:39:28 +0100, Torbjörn Granlund wrote:
> I assume "undefined" means that the computer will not explode, or even
> cause any less dramatic security issues, or even terminate execution.

Compilers use the fact that undefined behavior must not occur to
do some optimizations, like removing what appears as dead code
(but actually isn't). So, this can yield crashes and security issues.
See for instance the invalid bug:

  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30475
  "assert(int+100 > int) optimized away"

and the following article:

  https://lwn.net/Articles/575563/

So, with some codes that intended to detect buffer overflow or
integer overflow with code like the above one, the check was no
longer performed.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Vincent Lefevre]

On 2016-12-01 08:57:12 -0700, Nelson H. F. Beebe wrote:
> Notice that it does NOT prohibit left-shifting negative values: it
> just says that behavior with negative SHIFT COUNTS is undefined.
> 
> The reason for that is that low-level expressions in C have always
> been intended, and assumed, to have straightforward mappings into
> hardware instructions.  The meaning of a negative shift has always
> been dependent on the machine's instruction set.

Is this true even when the shifted value is zero? For some time,
I've wondered why this was also undefined for zero (in MPFR, we
had code where the shift count could be out of range only for
the value 0).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Nelson H. F. Beebe]

Niels M{\"o}ller and Torbj{\"o}rn Granlund write on Sunday 27-Nov-2016:

>> ...
>>   But it's surprising to me too that also non-overflowing left shift of
>>   negative values is undefined. So it seems generally unsafe to use shift
>>   on signed types, except possibly for constants.
>>   
>> It is pretty silly.  By leaving a useful operation undefined, one cannot
>> use it anywhere if one cares about perfect portability.  Here we have an
>> operation which in practce will work anywhere made somewhat useless by
>> the C/C++ committees.
>> ...

I believe there is a misunderstanding of the C/C++ standards.

Here is verbatim text lifted from Technical Corrigendum 3 (2007)
of the ISO/IEC 9899 standard (aka C99):

>> ...
>>  6.5.7 Bitwise shift operators
>>  Syntax
>> 1 shift-expression:
>>   additive-expression
>>   shift-expression << additive-expression
>>   shift-expression >> additive-expression
>>  Constraints
>> 2Each of the operands shall have integer type.
>>  Semantics
>> 3The integer promotions are performed on each of the operands. The type 
>> of the result is
>>  that of the promoted left operand. If the value of the right operand is 
>> negative or is
>>  greater than or equal to the width of the promoted left operand, the 
>> behavior is undefined.
>> 
>> ...

Notice that it does NOT prohibit left-shifting negative values: it
just says that behavior with negative SHIFT COUNTS is undefined.

The reason for that is that low-level expressions in C have always
been intended, and assumed, to have straightforward mappings into
hardware instructions.  The meaning of a negative shift has always
been dependent on the machine's instruction set.

For example, the famous IBM System/360 architecture from 1964,
stil marketed as the IBM z9000 architecture, says in its
Principles of Operation manual:

>> ...
>> The second-operand address is not used to address data; its
>> rightmost six bits indicate the number of bit positions to be
>> shifted.  The remainder of the address is ignored.
>> ...

In particular, consider positive and negative 4-bit shifts with
these encodings in two's-complement arithmetic:

  4  0x0004   -4  0xfffc

For the negative shift, the count is picked up as 0x3c == 60.  In
other words, a shift of -n bits is really a shift or 64 - n bits.

For the signed right shift operation it prescribes this:

>> ...
>> 1. A right shift of one bit position is equivalent to division
>> by 2 with rounding downward. When an even number is shifted
>> right one position, the result is equivalent to dividing the
>> number by 2. When an odd number is shifted right one position,
>> the result is equivalent to dividing the next lower number by
>> 2. For example, +5 shifted right by one bit position yields
>> +2, whereas -5 yields -3.
>> 
>> 2. For SHIFT RIGHT SINGLE (SRA), shift amounts from 31 to 63
>> cause the entire numeric part to be shifted out of the
>> register, leaving a result of -1 or zero, depending on whether
>> or not the initial contents were negative. For SHIFT RIGHT
>> SINGLE (SRAG), a shift amount of 63 causes the same effect.
>> ...

>From the DEC VAX architecture handbook, the sign of the shift
count determines the direction: left for positive, move for zero,
right for negative. It says that on overflow in a left shift, the
result is the low-order bits of the true result.

On the DEC PDP-10, action is, like the VAX, according to the sign
of the shift.

The MIPS R1000 architecture manual says that the shift count
operand is greater that 31, or less than 0, the shift is by that
value MOD 32.

On Hewlett-Packard, PA-RISC, the shift count is an unsigned 5-bit
field in the instruction word.

On Sun SPARC, the shift count is taken from the 5 or 6
lower-order bits of the count operand; a compile-time constant
shift can be embedded in the instruction word.

The Motorola M68000 arithmetic left and right shifts operate only
one bit at a time, so a compiler must generate an n-bit shift
with an n-iteration loop (perhaps unrolled when n is a
compile-time constant).

The Intel x86 and x86-64 architectures use the low-order bits of
the shift-count operand to determine the actual shift.

These behavioral differences across various systems (all of which
have had multiple C compilers), forced the C Standard to
prescribe negative shift counts as undefined behavior.  In all
cases, the generated assembly code will produce REPRODUCIBLE
results on a single architecture, but those results may DIFFER
across architectures.

---
- Nelson H. F. BeebeTel: +1 801 581 5254  -
- University of UtahFAX: +1 801 581 4148  -
- Department of Mathematics, 110 LCBInternet e-mail: be...@math.utah.edu  -
- 155 S 1400 E RM 233   be...@acm.org  be...@comput

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  Bit it's surprising to me too that also non-overflowing left shift of
  negative values is undefined. So it seems generally unsafe to use shift
  on signed types, except possibly for constants.
  
It is pretty silly.  By leaving a useful operation undefined, one cannot
use it anywhere if one cares about perfect portability.  Here we have an
operation which in practce will work anywhere made somewhat useless by
the C/C++ committees.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Marc Glisse  writes:

  There is something strange with
  https://gmplib.org/devel/tm/gmp/date.html . When I click to get the
  logs for ivydeb32v9.gmplib.org-dyn-noasm-ubsan, the build log is from
  26/11, but the check log is from 21/11...

Several generations of runs are kept for check/failure.  This is made to
avoid missing intermittent (e.g. seed triggered) failures.

Search for "DATE:" to find newer versions.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Fri, 25 Nov 2016, Marc Glisse wrote:


On Mon, 21 Nov 2016, Torbjörn Granlund wrote:


Marc Glisse  writes:

 On Sun, 20 Nov 2016, Niels Möller wrote:

 > It would make sense to test both gmp and mini-gmp with
 > -fsanitize=undefined.

 If we are not already doing it, yes, I highly recommend it.

It is now running for ivydeb64v9 and ivydeb32v9.

Two errors were triggered for the former, one in mini-gmp (presumably
fixed by nisse) and one on cxx/t-ops2z.  Please take a look, glisse.


Fixed. It was unhappy about (-13) << 2. I am a bit surprised it doesn't 
complain about (-13) >> 2 on the next line, we'll see if it ever becomes 
an issue.


There is something strange with https://gmplib.org/devel/tm/gmp/date.html 
. When I click to get the logs for ivydeb32v9.gmplib.org-dyn-noasm-ubsan, 
the build log is from 26/11, but the check log is from 21/11...



--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> I assume "undefined" means that the computer will not explode, or even
> cause any less dramatic security issues, or even terminate execution.

I think a typical operation with "undefined" meaning is dereferencing a
NULL pointer. Which on most current systems will terminate execution.

A system where left shift can crash seems a bit more unlikely. But I
guess it's conceivable with a processor and C compiler featuring an
"arithmetic left shift" instruction which traps on signed overflow.
Which would be compliant with the C standard.

Bit it's surprising to me too that also non-overflowing left shift of
negative values is undefined. So it seems generally unsafe to use shift
on signed types, except possibly for constants.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Fri, 25 Nov 2016, Torbjörn Granlund wrote:


Marc Glisse  writes:

 Surprisingly, shifting negative numbers left is undefined, while
 shifting them right it implementation-defined.

I fail to appreciate the difference between these definitions of
undefinedness.

I assume "undefined" means that the computer will not explode, or even
cause any less dramatic security issues, or even terminate execution.
It's undefined only in that the result can be any bit pattern.  This bit
pattern might vary bewteen any two computers, with used compiler and
between two executions.


Actually, yes, "undefined behavior" means that the computer can explode, 
execution can be terminated (what happens with the sanitizer), etc.



And "implementation-defined" means that the bit pattern can be anything,
varying between computers (or compiler on one computer).


"Implementation-defined value" means that every compiler must return a 
value, and document what that value is (in practice, arithmetic shift).


--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Marc Glisse  writes:

  Surprisingly, shifting negative numbers left is undefined, while
  shifting them right it implementation-defined.
  
I fail to appreciate the difference between these definitions of
undefinedness.

I assume "undefined" means that the computer will not explode, or even
cause any less dramatic security issues, or even terminate execution.
It's undefined only in that the result can be any bit pattern.  This bit
pattern might vary bewteen any two computers, with used compiler and
between two executions.

And "implementation-defined" means that the bit pattern can be anything,
varying between computers (or compiler on one computer).

So it seems the only difference is that "implementation-defined" means
that two executions (compiled in the same way on a given computer) will
yield the same arbitrary bit pattern.  How useful.  :-)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Fri, 25 Nov 2016, Torbjörn Granlund wrote:


Marc Glisse  writes:

 Fixed. It was unhappy about (-13) << 2. I am a bit surprised it
 doesn't complain about (-13) >> 2 on the next line, we'll see if it
 ever becomes an issue.

This must be a compiler problem.  At least in C, shifting negative
integers is undefined.  It is a pity that it is, but the standard say
so.

I am less sure about C++.  Some googling did not provide a definitive
answer.

But if left shift of negative integers is undefined, surely right shift
is too!


Surprisingly, shifting negative numbers left is undefined, while
shifting them right it implementation-defined.


The value of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are 
zero-filled. If E1 has an unsigned type, the value of the result is E1 × 2^E2 , 
reduced modulo one more than the maximum value representable in the result type. 
Otherwise, if E1 has a signed type and non-negative value, and E1 × 2^E2 is 
representable in the corresponding unsigned type of the result type, then that value, 
converted to the result type, is the resulting value; otherwise, the behavior is 
undefined.

The value of E1 >> E2 is E1 right-shifted E2 bit positions. If E1 has an 
unsigned type or if E1 has a signed type and a non-negative value, the value of the 
result is the integral part of the quotient of E1/2^E2 . If E1 has a signed type and 
a negative value, the resulting value is implementation-defined.

--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Marc Glisse  writes:

  Fixed. It was unhappy about (-13) << 2. I am a bit surprised it
  doesn't complain about (-13) >> 2 on the next line, we'll see if it
  ever becomes an issue.

This must be a compiler problem.  At least in C, shifting negative
integers is undefined.  It is a pity that it is, but the standard say
so.

I am less sure about C++.  Some googling did not provide a definitive
answer.

But if left shift of negative integers is undefined, surely right shift
is too!

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Mon, 21 Nov 2016, Torbjörn Granlund wrote:


Marc Glisse  writes:

 On Sun, 20 Nov 2016, Niels Möller wrote:

 > It would make sense to test both gmp and mini-gmp with
 > -fsanitize=undefined.

 If we are not already doing it, yes, I highly recommend it.

It is now running for ivydeb64v9 and ivydeb32v9.

Two errors were triggered for the former, one in mini-gmp (presumably
fixed by nisse) and one on cxx/t-ops2z.  Please take a look, glisse.


Fixed. It was unhappy about (-13) << 2. I am a bit surprised it doesn't 
complain about (-13) >> 2 on the next line, we'll see if it ever becomes 
an issue.


--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  For mini-gmp, I think we should get good code coverage with maximum size
  of 10 limbs, maybe even smaller. But it nevertheless makes sense to me
  to include tests on sizes of cryptographic interest (say, up to the
  order of 100 limbs). Not because we expect problems in that range, but
  because that's a relevant use case, and we want to catch also unexpected
  problems...
  
Indeed.  So perhaps a little larger than 128 limbs, with the
cryptography argument.

  Current testsuite probably isn't very consistent in the size range. If
  we want to set a time budget of around 0.5s per test on a reasonably
  fast machine, I don't have any good intuition on how to trade that in
  terms of sizes and repetition count.
  
It would be good to stay well under 0.5s on average for modern hardware,
else our emulated and old machines will suffer.  A reasonably goal is to
cover the code with the default seed and not run much more tests than
that.  End users who run the tests check for miscompilations, we check
library correctness with our (seedy!) onslaught.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  t...@gmplib.org (Torbjörn Granlund) writes:
  
  > Venerable dupont (alphaev56 gentoo) seems to pass mini-gmp now.  Let's
  > see if the previously consistently failing mini-gmp t-cmp_d now
  > consistently passes; then I think we can assume this to be fixed.
  
  Are there any of the changes which could if it now passes? The
  LD_LIBRARY_PATH fixes?
  
No idea.  I don't think I will try to analyse this since it is an
obsolete platform.  (I think testing on obsolete platforms has some
value, but I won't spend too much time on them.)

I'll keep dupont running for a day or two now.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> Venerable dupont (alphaev56 gentoo) seems to pass mini-gmp now.  Let's
> see if the previously consistently failing mini-gmp t-cmp_d now
> consistently passes; then I think we can assume this to be fixed.

Are there any of the changes which could if it now passes? The
LD_LIBRARY_PATH fixes?

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Venerable dupont (alphaev56 gentoo) seems to pass mini-gmp now.  Let's
see if the previously consistently failing mini-gmp t-cmp_d now
consistently passes; then I think we can assume this to be fixed.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> If hugely varying operand sizes are motivated (which is the case for
> "big"-GMP with its many operand size dependent algorithm choices) one
> could make operand size selection more complex to allow good coverage
> and less time fluctuations.

For mini-gmp, I think we should get good code coverage with maximum size
of 10 limbs, maybe even smaller. But it nevertheless makes sense to me
to include tests on sizes of cryptographic interest (say, up to the
order of 100 limbs). Not because we expect problems in that range, but
because that's a relevant use case, and we want to catch also unexpected
problems...

Current testsuite probably isn't very consistent in the size range. If
we want to set a time budget of around 0.5s per test on a reasonably
fast machine, I don't have any good intuition on how to trade that in
terms of sizes and repetition count.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

mini-gmp testing (was: Re: Help stabilising mini-gmp)

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> We should at least make sure the algorithms' corner cases are exercised,
> e.g., that large quotient are generated for Euclid's algorithm and that
> remainders of +-epsilon are used for division.

The mini-gmp testsuite is a bit hairy because it uses both gmp and
mini-gmp. This works fine with linking, thanks to the name mangling of
gmp symbols, but we can't include both gmp.h and mini-gmp.h in the same
compilation unit, and we have to types called mpz_t which in general
aren't compatible.

Most of the current tests works by invoking a function which uses the
real gmp to produce random inputs and a reference answer. E.g., (a, b,
a+b), or (a, b, gcd(a,b)). These are then translated between gmp mpz_t
and mini-gmp mpz_t via hex representation.

I think this is reasonable for simple things like add and mul, but maybe
not for the more complex things.

In particular, for division and gcd(ext), where the result is easy to
validate using multiplication and addition, there are different ways to
do things. We need to call into gmp to generate inputs or pieces of
inputs (because mini-gmp lacks mpz_urandom and mpz_rrandom). And we
obviously want to do the operation under test using mini-gmp. But we can
choose freely whether or not the code to produce inputs and validate
outputs should use gmp or mini-gmp.

One opposite way to organize the tests would be to have the main part of
the test programs use the real gmp, i.e., include gmp.h rather than
mini-gmp.h. And then do wrapper functions for the functions under test,
e.g, a mini_mpz_gcd which accepts gmp mpz_t as inputs, converts to hex,
passes on to a separate compilation unit which uses mini-gmp.h rather
than gmp.h to convert the hex inputs to mini-gmp mpz_t, call the
corresponding mini-gmp function, and convert back.

That should make it easier to copy testcases from the real gmp. The main
drawback I see is that testcases get less clear if there's a hairy
wrapper between the the testcode and the function it's testing. Complex
wrappers around generation of test inputs and validation of results
somehow suits my taste better, but I have no good rational motivation
for that.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

"Marco Bodrato"  writes:

  I tried the test on a 18-years old 32-bits CPU I'm currently using. A
  single 8000-bit Miller-Rabin round took 20 seconds with mini-gmp.
  That's why I reduced the size. With 2000-bit operands I assume the worst
  execution time will be reduced by a factor 18.
  
  The hypothesis "its time varies hugely" is surely true. Before the
  changeset 17099, on my machine, fluctuations between 2 and 60 seconds
  where the norm. Of course a seed that require many MR rounds on big
  operands is unlikely...
  
I see.  I am less worried now.  Let's mark the bug as done!

If hugely varying operand sizes are motivated (which is the case for
"big"-GMP with its many operand size dependent algorithm choices) one
could make operand size selection more complex to allow good coverage
and less time fluctuations.  One trick is to make random sizes within a
series of, fixed intervals.  I did that for mpz/t-mul.c some years ago.

I have some vague plans of writing a new test framework where that would
be a fundamental mechanims.  One other feature is allowing truly huge
operands, something we currently don't test.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marco Bodrato]

Ciao,

Il Lun, 21 Novembre 2016 6:47 pm, Niels Möller ha scritto:
> t...@gmplib.org (Torbjörn Granlund) writes:
>
>> Of the remaining https://gmplib.org/devel/mini-gmp-status.html issues I
>> worry most about #2.  Marco adjusted the parameters to make it faster,
>> but I remain unconvinced that g5.gmplib.org-dyn:32 really needed 400
>> seconds for the test with old parameters.  I suspect the code can hang
>> for certain seeds.  (We cannot tell the seed because of #1.)
>
> If it happens again, the seed should be printed out.
> https://gmplib.org/repo/gmp/rev/5abbd164e2a3

I tried the test on a 18-years old 32-bits CPU I'm currently using. A
single 8000-bit Miller-Rabin round took 20 seconds with mini-gmp.
That's why I reduced the size. With 2000-bit operands I assume the worst
execution time will be reduced by a factor 18.

The hypothesis "its time varies hugely" is surely true. Before the
changeset 17099, on my machine, fluctuations between 2 and 60 seconds
where the norm. Of course a seed that require many MR rounds on big
operands is unlikely...

Best regards,
mb

-- 
http://bodrato.it/toom/

___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  ni...@lysator.liu.se (Niels Möller) writes:
  
  > I'll try to log in to alpha-gentoo and reproduce.
  
  Plain make check (i.e., main gmp testsuite) fails in the mpn subdir with
  
FAIL: t-get_d
=

Warning, IEEE denorm tests skipped due to SIGFPE (exp=-1075)
FAIL t-get_d (exit status: 136)
  
  And make check-mini-gmp fails with
  
/home/nisse/hack/gmp/mini-gmp/tests/run-tests: line 65: 21336 Floating
point exceptionLD_LIBRARY_PATH="$TEST_LD_LIBRARY_PATH"
DYLD_LIBRARY_PATH="$TEST_DYLD_LIBRARY_PATH" "$1" $testflags
FAIL: t-double
/home/nisse/hack/gmp/mini-gmp/tests/run-tests: line 65: 21353 Floating
point exceptionLD_LIBRARY_PATH="$TEST_LD_LIBRARY_PATH"
DYLD_LIBRARY_PATH="$TEST_DYLD_LIBRARY_PATH" "$1" $testflags
FAIL: t-cmp_d
  
  So for now, I'd say this isn't mini-gmp's fault. What kind of floating
  point does alpha support? I noticed that CFLAGS included -mieee, perhaps
  that's a mode not implemented properly in qemu?
  
Sorry, I should have given some background.

Yes, under qemu-user we see these failures.  They do not happen on
hardware.  (I started looking into that in order to report it to qemu,
but decided against it as getting bugs fixed in qemu is an excruciating
exercise, even if one sends them a simple patch.)

On real hardware, only mini-gmp fails.  Under qemu-system, nothing
fails, not even mini-gmp's t-get_d...  There is no method in the
madness.  :-)

Unfortunately, both our alpha systems and offline and even when on-line
they might be painful to work with due to their slowness.  (If you want
to play with one now, I can switch it on manually.)


-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

ni...@lysator.liu.se (Niels Möller) writes:

> I'll try to log in to alpha-gentoo and reproduce.

Plain make check (i.e., main gmp testsuite) fails in the mpn subdir with

  FAIL: t-get_d
  =
  
  Warning, IEEE denorm tests skipped due to SIGFPE (exp=-1075)
  FAIL t-get_d (exit status: 136)

And make check-mini-gmp fails with

  /home/nisse/hack/gmp/mini-gmp/tests/run-tests: line 65: 21336 Floating
  point exceptionLD_LIBRARY_PATH="$TEST_LD_LIBRARY_PATH"
  DYLD_LIBRARY_PATH="$TEST_DYLD_LIBRARY_PATH" "$1" $testflags
  FAIL: t-double
  /home/nisse/hack/gmp/mini-gmp/tests/run-tests: line 65: 21353 Floating
  point exceptionLD_LIBRARY_PATH="$TEST_LD_LIBRARY_PATH"
  DYLD_LIBRARY_PATH="$TEST_DYLD_LIBRARY_PATH" "$1" $testflags
  FAIL: t-cmp_d

So for now, I'd say this isn't mini-gmp's fault. What kind of floating
point does alpha support? I noticed that CFLAGS included -mieee, perhaps
that's a mode not implemented properly in qemu?

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  If it happens again, the seed should be printed out.
  https://gmplib.org/repo/gmp/rev/5abbd164e2a3
  
Yep.  Unless the smaller operands affects the behaviour.

  > #15 is strange, I haven't tried to understand why these libgcc link
  > errors happens for just certain similar configs.
  
  The direct links to examples are dead.

They were quite garbled, now hopefully fixed.

  Do these system normally have
  LD_LIBRARY_PATH set? If so, maybe something wrong when gmp:s .lib is
  prepended to the path.
  
echo $LD_LIBRARY_PATH 
/usr/local/lib

What confused me was that the -fake test variant (which runs through
many CPUs for a fat build) didn't fail.  Now I see why; they don't
include mini-gmp.

I havn't followed the logic of the local test wrapper, but presumably it
doesn't prepend (or append) correctly to LD_LIBRARY_PATH.

  When the dust has settled, we'll have to think about whether we should
  make a "mini-gmp release" with bug-fixes, or just send out an
  announcement.
  
Agreed.

Let's follow this plan:

1. Fix any issues remaining which we think are are worth fixing (i.e.,
   we don't need to work around every bug elsewhere).

2. Make sure the testsuite is up to scratch wrt hard-to-test algorithms
   (for example those I mentioned yesterday)

3. Then let the test machinery run for a few weeks; this ought to
   trigger any remaining arithmetic bugs.  (Please recall that 6 days
   are required for a full cycle as things are set up these days.)

4. Perhaps release 6.1.2 with the now hopefully bug-free mini-gmp.  (Or
   somehow release it separately (whatever that means) or just shout
   about this and direct people to the repo.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> Of the remaining https://gmplib.org/devel/mini-gmp-status.html issues I
> worry most about #2.  Marco adjusted the parameters to make it faster,
> but I remain unconvinced that g5.gmplib.org-dyn:32 really needed 400
> seconds for the test with old parameters.  I suspect the code can hang
> for certain seeds.  (We cannot tell the seed because of #1.)

If it happens again, the seed should be printed out.
https://gmplib.org/repo/gmp/rev/5abbd164e2a3

> I am not sure #8 (alpha/dupont) is for real.

I'll try to log in to alpha-gentoo and reproduce.

> #15 is strange, I haven't tried to understand why these libgcc link
> errors happens for just certain similar configs.

The direct links to examples are dead. Do these system normally have
LD_LIBRARY_PATH set? If so, maybe something wrong when gmp:s .lib is
prepended to the path.

> I am glad the "red" bugs had one cause (although a serious one).  It
> would have been much worse with many errors of this kind.  (Well, the
> results for end users are same same, but it matters for how to assess
> code quality.)

When the dust has settled, we'll have to think about whether we should
make a "mini-gmp release" with bug-fixes, or just send out an
announcement.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  t...@gmplib.org (Torbjörn Granlund) writes:
  
  > 2. The latest Gentoo doesn't support -fno-sanitize-recover.  I suppose
  >it works without it?
  
  As I understand it, the point of -fno-sanitize-recover is to make
  programs crash on the spot when there's some undefined operation. Not
  sure what the behaviour is without that, if the program might complete
  normally with just logging of problems, or if detected problems still
  makes it exit with non-zero status.
  
Yes, it just prints some complaints.  This is not good for us, so now
ivydeb64v9 and ivydeb32v9 with newer compilers will run some tests with
these options.

So where are we now, wrt mini-gmp?

We're on more stable grounds, I think, now.  Less panic, more time for
other things.  But please let us finish this within some weeks.

Of the remaining https://gmplib.org/devel/mini-gmp-status.html issues I
worry most about #2.  Marco adjusted the parameters to make it faster,
but I remain unconvinced that g5.gmplib.org-dyn:32 really needed 400
seconds for the test with old parameters.  I suspect the code can hang
for certain seeds.  (We cannot tell the seed because of #1.)

I suspect we can just wait and see if it happens again (for the new,
smaller parameters).

#5 went away, I suspect the other t-str fixes solved it.

I am not sure #8 (alpha/dupont) is for real.

#15 is strange, I haven't tried to understand why these libgcc link
errors happens for just certain similar configs.

I am glad the "red" bugs had one cause (although a serious one).  It
would have been much worse with many errors of this kind.  (Well, the
results for end users are same same, but it matters for how to assess
code quality.)


-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> 2. The latest Gentoo doesn't support -fno-sanitize-recover.  I suppose
>it works without it?

As I understand it, the point of -fno-sanitize-recover is to make
programs crash on the spot when there's some undefined operation. Not
sure what the behaviour is without that, if the program might complete
normally with just logging of problems, or if detected problems still
makes it exit with non-zero status.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Marc Glisse  writes:

  On Sun, 20 Nov 2016, Niels Möller wrote:
  
  > It would make sense to test both gmp and mini-gmp with
  > -fsanitize=undefined.
  
  If we are not already doing it, yes, I highly recommend it.

It is now running for ivydeb64v9 and ivydeb32v9.

Two errors were triggered for the former, one in mini-gmp (presumably
fixed by nisse) and one on cxx/t-ops2z.  Please take a look, glisse.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  Anyway, I can now repro locally, by running 
  
make check CFLAGS="-O -Wall -g -fsanitize=undefined -fno-sanitize-recover"
  
  in the mini-gmp/tests source directory. This fails with
  
I spent a good number of hours trying to accompish this.

1. NetBSD 7.0.1 apparently does not have the required libraries (I
   wasted time installing the latest version).  Or at least they don't
   build properly, and googling don't help.

2. The latest Gentoo doesn't support -fno-sanitize-recover.  I suppose
   it works without it?

(I generalised the main nightbuild script to handle this stuff, making
the system "noasm" redundant.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  For mini-gmp, I hope running the testsuite with a large number of random
  seeds should give good coverage, so we don't need any additional
  framework.
  
We should at least make sure the algorithms' corner cases are exercised,
e.g., that large quotient are generated for Euclid's algorithm and that
remainders of +-epsilon are used for division.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

"Marco Bodrato"  writes:

> Should we implement (or adapt) something like the tests/devel/try thing to
> widely test the basic operations in mini-gmp?

I think devel/try is mainly for testing assembly code, to find register
clobbering bugs and the like.

For mini-gmp, I hope running the testsuite with a large number of random
seeds should give good coverage, so we don't need any additional
framework.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I've checked in an update to the html file. But running www-update fails,
  
[nisse@shell /var/hg]$ ./www-update 
abort: Permission denied: '/var/hg/www/devel/mini-gmp-status.html'
  
Please try again now.

  Before doing a ww-update, it would also be nice to check how it looks in
  a browser. Is the easiest way to do that to have a local checkout? I did
  this editing on shell where I haven't even tried to run a web browser.
  
I suppose downloading and using file:///... would be one way.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> Good!  You're welcome to update the mini-gmp-status.html repo file and
> run the web update script.

I've checked in an update to the html file. But running www-update fails,

  [nisse@shell /var/hg]$ ./www-update 
  abort: Permission denied: '/var/hg/www/devel/mini-gmp-status.html'

Before doing a ww-update, it would also be nice to check how it looks in
a browser. Is the easiest way to do that to have a local checkout? I did
this editing on shell where I haven't even tried to run a web browser.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

  > It would make sense to test both gmp and mini-gmp with
  > -fsanitize=undefined.
  
  If we are not already doing it, yes, I highly recommend it.

I also highly recommend it and the system noasm should hopefully obey
our recommendations in its next run.  ;-)

(The hardware martin.gmplib.org which is where "shell", "gshell", ivy*,
*-debv*, all runs, is getting its 6 cores busy 24 hours per day with all
the testing.  Not good, something needs to be done.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Sun, 20 Nov 2016, Niels Möller wrote:

It would make sense to test both gmp and mini-gmp with 
-fsanitize=undefined.


If we are not already doing it, yes, I highly recommend it.

--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

  > If the code becomes undefined by gcc's transformations, that it's a
  > compiler bug.
  
  That's not at all what I said, the code doesn't become undefined.

I didn't say you said that.  :-)

  "Reminder: computing INT_MIN-1 is forbidden. If you are not doing
  that, you can ignore this message. I am giving you this warning
  because you are computing X-1 and I decided to key the launch of
  nuclear missiles to the value X==INT_MIN. Have a safe day!"
  
:-D

Now even I get it...

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

Marc Glisse  writes:

> After inlining, there are subtractions. check_si is called at least
> once with oi = si + c (c is ±1). gcc simplifies the test si > si - 1
> to true, and warns that this optimization may break your program if
> you rely on wrapping.

I suspect this test program relies on it. With c == 1, si takes the
values 2, 4, 8, ..., (1<<62), (-1<<63), and I guess the check gcc warns
about is intended to detect overflow and act as a stop condition.

And my adding of debug printf might have pushed the size of the function
over some threshold so that it's no longer inlined, comparison not
optimized away, and then the test succeeded.

> The usefulness of such a warning is debatable, and we tend to drop
> some of them from gcc when we think nobody will notice.

Not sure. There are two dangers: Programs relying on undefined behavior,
and optimization based on the assumption that there will never be any
undefined behaviour.

Anyway, I can now repro locally, by running 

  make check CFLAGS="-O -Wall -g -fsanitize=undefined -fno-sanitize-recover"

in the mini-gmp/tests source directory. This fails with

  t-signed.c:93:8: runtime error: signed integer overflow: -1 + 
-9223372036854775808 cannot be represented in type 'long int'
  FAIL: t-signed

It would make sense to test both gmp and mini-gmp with
-fsanitize=undefined.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Sun, 20 Nov 2016, Torbjörn Granlund wrote:


Marc Glisse  writes:

 After inlining, there are subtractions. check_si is called at least
 once with oi = si + c (c is ±1). gcc simplifies the test si > si - 1

Inlining of check_si?


Of all the functions. (note that I am just speculating, but the warning 
does not surprise me)



If the code becomes undefined by gcc's transformations, that it's a
compiler bug.


That's not at all what I said, the code doesn't become undefined. If it 
was fine, it remains fine. If it was broken, the brokenness may become 
more obvious. Let me rephrase the warning:


"Reminder: computing INT_MIN-1 is forbidden. If you are not doing that, 
you can ignore this message. I am giving you this warning because you are 
computing X-1 and I decided to key the launch of nuclear missiles to the 
value X==INT_MIN. Have a safe day!"


(the computation of X-1 is in try_op_si, but the warning points at the 
location where the compiler inserts the trigger for the nuclear missiles)


--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Marc Glisse  writes:

  After inlining, there are subtractions. check_si is called at least
  once with oi = si + c (c is ±1). gcc simplifies the test si > si - 1

Inlining of check_si?

If the code becomes undefined by gcc's transformations, that it's a
compiler bug.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I next tried adding -Wall to the command line flags, and I see
  
gcc -std=gnu99 -m64  -mtune=power7 -O3 -Wall -I../.. -c
/home/nisse/hack/gmp/mini-gmp/tests/t-signed.c -o t-signed.o
/home/nisse/hack/gmp/mini-gmp/tests/t-signed.c: In function 'testmain':
/home/nisse/hack/gmp/mini-gmp/tests/t-signed.c:45:26: warning: assuming
signed overflow does not occur when assuming that (X - c) > X is always
false [-Wstrict-overflow]
   if ((si < oi ? -1 : si > oi) != c)
  ^
/home/nisse/hack/gmp/mini-gmp/tests/t-signed.c:45:21: warning: assuming
signed overflow does not occur when assuming that (X + c) >= X is always
true [-Wstrict-overflow]
   if ((si < oi ? -1 : si > oi) != c)
   ^
  
  I don't quite understand neither what the testcase is doing, nor what the
  warning means, since there's no subtraction in there.
  
It looks like a confused compiler.  Perhaps the test failure is caused
by the same confusion?

ppc64eb.sys (and ppc64el.sys) run the same environment, but under full
system emulation.  They provide gdb.  But these systems are slow (slower
single cpu performance, and no SMP).

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Sun, 20 Nov 2016, Niels Möller wrote:


ni...@lysator.liu.se (Niels Möller) writes:


It seems "trivially" reproducible on ppc64 though, both real metal
versions and fake ones like ppceb-debv8 (and ppcel-debv8).


I'll try to debug (if no one else beats me to it), but not today.


I've logged in to ppceb-debv8, and it's easy to reproduce. No gdb
installed, so I'm attempting printf debugging. And then I noticed that
the test succeeds if adding a few debug printouts...

I next tried adding -Wall to the command line flags, and I see

 gcc -std=gnu99 -m64  -mtune=power7 -O3 -Wall -I../.. -c
 /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c -o t-signed.o
 /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c: In function 'testmain':
 /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c:45:26: warning: assuming
 signed overflow does not occur when assuming that (X - c) > X is always
 false [-Wstrict-overflow]
if ((si < oi ? -1 : si > oi) != c)
   ^
 /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c:45:21: warning: assuming
 signed overflow does not occur when assuming that (X + c) >= X is always
 true [-Wstrict-overflow]
if ((si < oi ? -1 : si > oi) != c)
^

I don't quite understand neither what the testcase is doing, nor what the
warning means, since there's no subtraction in there.


After inlining, there are subtractions. check_si is called at least once 
with oi = si + c (c is ±1). gcc simplifies the test si > si - 1 to true, 
and warns that this optimization may break your program if you rely on 
wrapping. The usefulness of such a warning is debatable, and we tend to 
drop some of them from gcc when we think nobody will notice.


--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

ni...@lysator.liu.se (Niels Möller) writes:

>> It seems "trivially" reproducible on ppc64 though, both real metal
>> versions and fake ones like ppceb-debv8 (and ppcel-debv8).
>
> I'll try to debug (if no one else beats me to it), but not today.

I've logged in to ppceb-debv8, and it's easy to reproduce. No gdb
installed, so I'm attempting printf debugging. And then I noticed that
the test succeeds if adding a few debug printouts...

I next tried adding -Wall to the command line flags, and I see

  gcc -std=gnu99 -m64  -mtune=power7 -O3 -Wall -I../.. -c
  /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c -o t-signed.o
  /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c: In function 'testmain':
  /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c:45:26: warning: assuming
  signed overflow does not occur when assuming that (X - c) > X is always
  false [-Wstrict-overflow]
 if ((si < oi ? -1 : si > oi) != c)
^
  /home/nisse/hack/gmp/mini-gmp/tests/t-signed.c:45:21: warning: assuming
  signed overflow does not occur when assuming that (X + c) >= X is always
  true [-Wstrict-overflow]
 if ((si < oi ? -1 : si > oi) != c)
 ^

I don't quite understand neither what the testcase is doing, nor what the
warning means, since there's no subtraction in there.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I just pushed an LD_LIBRARY_PATH improvement, we'll see if it reduces
  the number of failures.
  
The power-aix aka gcc111 failures are gone.  The
sky.gmplib.org-dyn-fat-fake:64.txt is still there until sky runs again
(Tuesday).

There used to be over 500 failures due to the LD_LIBRARY_PATH/CFLAGS
problems, then various hacks brought it down.  Your proper fix seems to
solve the problems.

We're now down to 88 expected failures.  A mere 43 are related to
mini-gmp, some 10 to "hardened" GNU/Linux, the rest are long-standing
clang bugs.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I think the three red arithmetic bugs (t-gcd, t-powm and t-div) were all
  caused by the bug in mpn_invert_3by2, so hopefully fixed now. Or have
  you seen any new failures for those?
  
Good!  You're welcome to update the mini-gmp-status.html repo file and
run the web update script.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> There are currently 10 outstanding mini-gmp issues according to
> , will be 9 if your
> LD_LIBRARY_PATH change is effective.  Is it up-to-date?

I think the three red arithmetic bugs (t-gcd, t-powm and t-div) were all
caused by the bug in mpn_invert_3by2, so hopefully fixed now. Or have
you seen any new failures for those?

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I just pushed an LD_LIBRARY_PATH improvement, we'll see if it reduces
  the number of failures.
  
We should see already tomorrow morning (altough it takes 6 days for all
configs to run without manual intervention).

There are currently 10 outstanding mini-gmp issues according to
, will be 9 if your
LD_LIBRARY_PATH change is effective.  Is it up-to-date?

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> ni...@lysator.liu.se (Niels Möller) writes:
>   I don't think so. But I think the right way is to simply add $(CFLAGS)
>   to the linker command line. That's the usual way to use it, right? And
>   omitting it in the linking rule in mini-gmp/tests/Makefile is a bug.
>   
> Since CFLAGS includes -Iblah it seemed like a uglier choice.

Changed to pass CFLAGS when linking (according to GNU standards), and
move -I flags to CPPFLAGS where they belong.

> It seems "trivially" reproducible on ppc64 though, both real metal
> versions and fake ones like ppceb-debv8 (and ppcel-debv8).

I'll try to debug (if no one else beats me to it), but not today.

>   In file included from gmp/mini-gmp/tests/testutils.c:24:0:
>   gmp/mini-gmp/tests/../mini-gmp.c: In function 'mpz_set_d':
>   gmp/mini-gmp/tests/../mini-gmp.c:1647:3: internal compiler error:
>   Aborted
>  if (x != x || x == x * 0.5)
>  ^~
>
> That's the LD_LIBRARY_PATH bug, I think.

That could explain it (I was first thinking that "ICE" meant a failed
assert in the compiler code, but I guess any SIGSEGV or similar shows
the ICE message?). 

I just pushed an LD_LIBRARY_PATH improvement, we'll see if it reduces
the number of failures.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

Marc Glisse  writes:

> That's a common hack (that I am ok with), but I don't think I would
> call it "the right way"...

At least, it's the GNU way. According to
https://www.gnu.org/prep/standards/standards.html#Command-Variables: 

  CFLAGS should be used in every invocation of the C compiler, both those
  which do compilation and those which do linking. 

> (btw, some compilers warn about the trailing comma in enum hex_random_op)

I tend to use trailing comma both in enum definitions and in array
initializers, when they span more than one line, to aid cut and paste. I
can't recall I've seen any warnings about that.

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

Marc Glisse  writes:

  > The first failure was also a bit interesting, an internal compiler error
  > with gcc-6.
  >
  > In file included from gmp/mini-gmp/tests/testutils.c:24:0:
  > gmp/mini-gmp/tests/../mini-gmp.c: In function 'mpz_set_d':
  > gmp/mini-gmp/tests/../mini-gmp.c:1647:3: internal compiler error:
  > Aborted
  >   if (x != x || x == x * 0.5)
  >   ^~
  >
  > 
https://gmplib.org/devel/tm/gmp/build/failure/sky.gmplib.org-dyn-fat-fake:64.txt
  
  Strange that it only shows up there, the command line does not have
  anything specific to skylake (no -march or -mtune). I can't reproduce
  on another machine.
  
Not only there, /ppceb-debv8.gmplib.org-dyn:mode32.txt too.

I suppose the mpfr version matters, or whatnot.

(It will not happen on these two machine if you cut out the compilation
command, since then you'll miss the LD_LIBRARY_PATH.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  To make sure I understand the issue, this is to get flags like -m32
  passed on to the linker?
  
Almost.  It passes it to the compiler used for invoking the linker,
which then munges it and DTRT.

  > I am not sure there are no side-effects conflicting with the author's
  > intended Makefile behaviour.
  
  I don't think so. But I think the right way is to simply add $(CFLAGS)
  to the linker command line. That's the usual way to use it, right? And
  omitting it in the linking rule in mini-gmp/tests/Makefile is a bug.
  
Since CFLAGS includes -Iblah it seemed like a uglier choice.

  > There is also a problem with t-signed on ppc64 machines.
  
  I noticed one of those failures, and it was't trivially reproducible on
  x86_64. Needs some investigation.
  
It seems "trivially" reproducible on ppc64 though, both real metal
versions and fake ones like ppceb-debv8 (and ppcel-debv8).

  The first failure was also a bit interesting, an internal compiler error
  with gcc-6.
  
  In file included from gmp/mini-gmp/tests/testutils.c:24:0:
  gmp/mini-gmp/tests/../mini-gmp.c: In function 'mpz_set_d':
  gmp/mini-gmp/tests/../mini-gmp.c:1647:3: internal compiler error:
  Aborted
 if (x != x || x == x * 0.5)
 ^~

That's the LD_LIBRARY_PATH bug, I think.

  
https://gmplib.org/devel/tm/gmp/build/failure/sky.gmplib.org-dyn-fat-fake:64.txt
  
  Begs for a proper gcc bug report, I guess. I fail to log in to sky, via
  nshell, at the moment ("ssh -p  localhost" on shell fails with
  connection refused). And this failure was on the non-virtualized host
  sky, right, not on one of the vms?
  
I've updated the instructions and added an alias in ssh_config.  (The
port number had changed with the update of the tunneling mechanism.
Sorry about that!)

Indeed sky itself.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marc Glisse]

On Sat, 19 Nov 2016, Niels Möller wrote:


I don't think so. But I think the right way is to simply add $(CFLAGS)
to the linker command line. That's the usual way to use it, right?


That's a common hack (that I am ok with), but I don't think I would call 
it "the right way"...



The first failure was also a bit interesting, an internal compiler error
with gcc-6.

In file included from gmp/mini-gmp/tests/testutils.c:24:0:
gmp/mini-gmp/tests/../mini-gmp.c: In function 'mpz_set_d':
gmp/mini-gmp/tests/../mini-gmp.c:1647:3: internal compiler error:
Aborted
  if (x != x || x == x * 0.5)
  ^~

https://gmplib.org/devel/tm/gmp/build/failure/sky.gmplib.org-dyn-fat-fake:64.txt


Strange that it only shows up there, the command line does not have 
anything specific to skylake (no -march or -mtune). I can't reproduce on 
another machine.


(btw, some compilers warn about the trailing comma in enum hex_random_op)

--
Marc Glisse
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> I made that and another change to now get the test to run on the
> majority of our systems.  The other change was this one:
>
>   https://gmplib.org/repo/gmp/rev/88fead579828
>
> This makes sure linking knows the ABI. 

To make sure I understand the issue, this is to get flags like -m32
passed on to the linker?

> I am not sure there are no side-effects conflicting with the author's
> intended Makefile behaviour.

I don't think so. But I think the right way is to simply add $(CFLAGS)
to the linker command line. That's the usual way to use it, right? And
omitting it in the linking rule in mini-gmp/tests/Makefile is a bug.

> Another problem with the Makefile.am check-mini-gmp code has surfaced.
> It overrides LD_LIBRARY_PATH (and DYLD_LIBRARY_PATH) for the entire
> mini-gmp build.  That includes gcc, which as you know is linked to GMP.
> The result is compilation problems on at least one platform.

Ah, that's an interesting side effect. To do something more specific to
the tests, how should that be named? Should we adopt the EMULATOR
variable (from automake, not sure if it's still in the current version)?
Then one would set

  EMULATOR="LD_LIBRARY_PATH=... DYLD_LIBRARY_PATH=..." \
  $(MAKE) ...
 
when running mini-gmp tests. I think EMULATOR is already supported by
the version of run-tests script which is in mini-gmp, it runs the test
programs ("$1" is the executable to be run) using 

  if [ -z "$EMULATOR" ] || head -1 "$1" | grep '^#!' > /dev/null; then
"$1" $testflags
  else
$EMULATOR "$1" $testflags
  fi

> There is also a problem with t-signed on ppc64 machines.

I noticed one of those failures, and it was't trivially reproducible on
x86_64. Needs some investigation.

The first failure was also a bit interesting, an internal compiler error
with gcc-6.

In file included from gmp/mini-gmp/tests/testutils.c:24:0:
gmp/mini-gmp/tests/../mini-gmp.c: In function 'mpz_set_d':
gmp/mini-gmp/tests/../mini-gmp.c:1647:3: internal compiler error:
Aborted
   if (x != x || x == x * 0.5)
   ^~

https://gmplib.org/devel/tm/gmp/build/failure/sky.gmplib.org-dyn-fat-fake:64.txt

Begs for a proper gcc bug report, I guess. I fail to log in to sky, via
nshell, at the moment ("ssh -p  localhost" on shell fails with
connection refused). And this failure was on the non-virtualized host
sky, right, not on one of the vms?

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

t...@gmplib.org (Torbjörn Granlund) writes:

--- a/Makefile.am   Wed Nov 16 23:07:02 2016 +0100
+++ b/Makefile.am   Thu Nov 17 07:12:08 2016 +0100
@@ -433,7 +433,7 @@ check-mini-gmp:
MINI_GMP_DIR="$$abs_srcdir/mini-gmp" \
LDFLAGS="-L../../.libs" \
LIBS="-lgmp -lm" \
-   CC="$(CC_FOR_BUILD)" EXTRA_CFLAGS="-g -I../.." check
+   CC="$(CC)" EXTRA_CFLAGS="$(CFLAGS) -g -I../.." check
 
  I will put this in now.  (I'd like to move back to getting some dev work
  done.  That needs a cleaner test reporting state.)

I made that and another change to now get the test to run on the
majority of our systems.  The other change was this one:

  https://gmplib.org/repo/gmp/rev/88fead579828

This makes sure linking knows the ABI.  I am not sure there are no
side-effects conflicting with the author's intended Makefile behaviour.

With these changes, it was feasible to mark the remaining mini-gmp
falures as "expected" to get a manageable report page again.  (I also
marked the PIC/static issue with which I am supposed to work as
expected; that issue now affects a handful machines as more machines
have been upgraded to newer Debian stretch versions.)

Another problem with the Makefile.am check-mini-gmp code has surfaced.
It overrides LD_LIBRARY_PATH (and DYLD_LIBRARY_PATH) for the entire
mini-gmp build.  That includes gcc, which as you know is linked to GMP.
The result is compilation problems on at least one platform.

The authors need to make sure only the test binary execution get a
modified LD_LIBRARY_PATH.

There is also a problem with t-signed on ppc64 machines.

I updated https://gmplib.org/devel/mini-gmp-status.html; please keep it
up-to-date with your changes.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I think this small change should fix that, but I can't test it at the
  moment.
  
  --- a/Makefile.am Wed Nov 16 23:07:02 2016 +0100
  +++ b/Makefile.am Thu Nov 17 07:12:08 2016 +0100
  @@ -433,7 +433,7 @@ check-mini-gmp:
MINI_GMP_DIR="$$abs_srcdir/mini-gmp" \
LDFLAGS="-L../../.libs" \
LIBS="-lgmp -lm" \
  - CC="$(CC_FOR_BUILD)" EXTRA_CFLAGS="-g -I../.." check
  + CC="$(CC)" EXTRA_CFLAGS="$(CFLAGS) -g -I../.." check
   
I will put this in now.  (I'd like to move back to getting some dev work
done.  That needs a cleaner test reporting state.)


-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  - CC="$(CC_FOR_BUILD)" EXTRA_CFLAGS="-g -I../.." check
  + CC="$(CC)" EXTRA_CFLAGS="$(CFLAGS) -g -I../.." check
   
That looks like an improvement.

Perhaps one will need to play with linker flags too, I am not sure.

Once this is fully resolved, around 530 of the current 535 failures
should clear.  :-)

I set up the testclient.sh script to look for GNU make (as "make",
"gmake", "gnumake") and then bother with check-mini-gmp iff GNU make is
found.  GMP itself is built with the "make" command which might or might
be GNU make.

The result of this logic is that mini-gmp is silently ignored without
GNU make.  One needs to grep in the result log file to tell.  Perhaps
this should be improved.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> I re-enabled it again, expect hundreds of failures due to the ABI
> adherence issue.  It's not easy to suppress those.

I think this small change should fix that, but I can't test it at the
moment.

--- a/Makefile.am   Wed Nov 16 23:07:02 2016 +0100
+++ b/Makefile.am   Thu Nov 17 07:12:08 2016 +0100
@@ -433,7 +433,7 @@ check-mini-gmp:
MINI_GMP_DIR="$$abs_srcdir/mini-gmp" \
LDFLAGS="-L../../.libs" \
LIBS="-lgmp -lm" \
-   CC="$(CC_FOR_BUILD)" EXTRA_CFLAGS="-g -I../.." check
+   CC="$(CC)" EXTRA_CFLAGS="$(CFLAGS) -g -I../.." check
 
 clean-mini-gmp:
if [ -d mini-gmp/tests ] ; then \


Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marco Bodrato]

Ciao,

Il Mer, 16 Novembre 2016 11:57 pm, Niels Möller ha scritto:
> t...@gmplib.org (Torbjörn Granlund) writes:
>
>> ni...@lysator.liu.se (Niels Möller) writes:
>>   It could also use some units test of its own; result is easy to
>>   validate.
>>
>> Perhaps better tests are needed.
>
> There already were unit tests for mpn_invert_3by2
> (mini-gmp/tests/t-invert). But we didn't get sufficient coverage in
> inputs until testing was enabled in gmp night builds.

Should we implement (or adapt) something like the tests/devel/try thing to
widely test the basic operations in mini-gmp?

Regards,
m

-- 
http://bodrato.it/papers/

___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

t...@gmplib.org (Torbjörn Granlund) writes:

  I will save lots of time by simply disable mini-gmp testing for now.

I re-enabled it again, expect hundreds of failures due to the ABI
adherence issue.  It's not easy to suppress those.

It should at least be slightly fewer now with testclient.sh supporting
split make programs for GMP and mini-gmp.  Assuming I got that shell
script magic right.  (So far, so good; only ~60 ABI failures at the
moment.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> ni...@lysator.liu.se (Niels Möller) writes:
>
>   I've tracked this one down to incorrect results from mpn_invert_3by2.
>   I'll investigate, and in the process try to document how it works, and
>   maybe rewrite it (it's a bit too hairy for its own good).

Bug found. The details: Problem was the computation of the 2/1 inverse
done by mpn_invert_3by2 (later adjusted to be a 3/2 inverse).

This was implemented as follows:

1. Compute the (approximate) high half limb of the inverse, using a plain 
division.

2. Adjust it to become a 3/2 inverse on half limbs, i.e, with 64 bit
   limbs, that would be floor((2^96-1) / u1) - 2^32.

3. Use this for a 3/2 division (on half-limbs) to compute the low half
   of the inverse. Similar logic used on full limbs for gmp's
   invert_4by2 (div_qr_2.c).

In the 3/2 division algorithm, candidate remainder is computed as

  r1   <-- (u1 - q1 d1) (mod B)
  {t1, t0} <-- d0 q1
  {r1, r0} <-- ({r1, u0} - {t1, t0} - {d1, d0}) (mod B^2)
  q1   <-- (q1 + 1) (mod B)

Multiplication and the extra subtraction of {d1, d0} must be done,
before incrementing q1, because incrementing might overflow. That's no
problem for the rest of the algorithm, but it would break the important
high half of the product d0 q1.

Now when working with half limbs, q1 fills only half a word so this can
be simplified by incrementing q1 first. The problem was that the
variable wasn't typed as a full limb (mp_limb_t) but as an unsigned,
which on 64-bit archs typically is exactly half a limb. And bug
triggered when low half of the inverse is 2^32 - 1.

> I am very worried about this, as well as past and lingering arithmetic
> bugs in mini-gmp.  Important lessons need to be learned, and full
> disclosure to our user needs to be made.

I think a bug like this ought to have been caught by asserts in the
division code. So that using an incorrect inverse results in a crash
rather than silent miscomputation.

> I have not looked at the mini-gmp sources, except very briefly.  I
> understood the project as a plain, safe implementation of core functions
> of GMP.

It's not always obvious where to draw the line. The objective is to be
at most 10 times slower than real gmp, and then I think division
corresponding to gmp:s _pi1 schoolbook division is reasonable. And the
way to do it is pretty well understood. The "novelty" is to compute the
inverse using a single-limb divide instruction for the high half, and a
3/2 for the low half.

> If we still believe in the idea of mini-gmp as part of a known robust
> library as GMP, I suggest that we do the following: Do not write any new
> code such as mpn_invert_3by2.  Only grab code from GMP, extracting the
> most fundamental algorithms.  Sometimes this will require editing to
> simplify the code; this work needs to be done carefully and
> thoughtfully.  Perhaps this is boring, but I am sure that our users
> prefer the boring property of correct computation results over never so
> surprising miscalculations!

To this, I'd like to add that we should use asserts whenever practical.
E.g., mpn_div_qr_pi1 only uses assert to check input requirements, but
it should be pretty easy to verify that each quotient used is sane
(correct or one off in the way that is expected). In mini-gmp, I think
it is reasonable to spend a % or two of the cpu resources on sanity
checks.

>   It could also use some units test of its own; result is easy to
>   validate.
>   
> Perhaps better tests are needed.

There already were unit tests for mpn_invert_3by2
(mini-gmp/tests/t-invert). But we didn't get sufficient coverage in
inputs until testing was enabled in gmp night builds.

And the t-div and t-powm failures I reproduced yesterday seem to also
have been caused by the broken inverse.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I've tracked this one down to incorrect results from mpn_invert_3by2.
  I'll investigate, and in the process try to document how it works, and
  maybe rewrite it (it's a bit too hairy for its own good).
  
I am very worried about this, as well as past and lingering arithmetic
bugs in mini-gmp.  Important lessons need to be learned, and full
disclosure to our user needs to be made.

I have not looked at the mini-gmp sources, except very briefly.  I
understood the project as a plain, safe implementation of core functions
of GMP.  Now, looking back, we know that the initial implementation was
very buggy, and clearly the code is still not great.  I am very far from
convinced that fixing the new miscalculation bugs will result in stable
implementation.

If we still believe in the idea of mini-gmp as part of a known robust
library as GMP, I suggest that we do the following: Do not write any new
code such as mpn_invert_3by2.  Only grab code from GMP, extracting the
most fundamental algorithms.  Sometimes this will require editing to
simplify the code; this work needs to be done carefully and
thoughtfully.  Perhaps this is boring, but I am sure that our users
prefer the boring property of correct computation results over never so
surprising miscalculations!

Under no circumstances should mini-gmp be a platform for trying out
cool, new algorithms.

  It could also use some units test of its own; result is easy to
  validate.
  
Perhaps better tests are needed.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> 5. The t-div test intermittently fails on at least
>sky.gmplib.org-k10-dyn-clang-3.6-clang++-3.6:64 (note that it uses
>clang, which means the likelihood of a compiler bug being the culprit
>is perhaps high).

I've tracked this one down to incorrect results from mpn_invert_3by2.
I'll investigate, and in the process try to document how it works, and
maybe rewrite it (it's a bit too hairy for its own good).

It could also use some units test of its own; result is easy to
validate.

With some luck, this could be the root cause also for the gcd and powm
failures (the powm failure I've seen involved a negative exponent, and
hence an initial modular inversion).

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> https://gmplib.org/devel/tm/gmp/check/failure/bwldeb64v7.gmplib.org-dyn-fat:64.txt

That's t-powm. I can reproduce that one locally.

  GMP_CHECK_RANDOMIZE=1479273572 ./t-powm

> So we might have 3 pending arithmetic bugs in mini-gmp (unless the gcd
> one is a clang bug).

I left

  while GMP_CHECK_RANDOMIZE=0 ./t-gcd ; do : ; done

running over night, with no failures (64-bit x86_64, gcc-6). 

But I can reproduce the problem with the data from
shell.gmplib.org-stat-clang35-clang++35-fat:32.txt, 

   GMP_CHECK_RANDOMIZE=1479112190 ./t-gcd

So this one isn't clang only.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> I realise that mini-gmp's requirement of GNU make hurts GMP's build
> environment compatibility testing; until now, we have tried to use plain
> 'make' on almost all systems.  If mini-gmp now forces the entire build
> to use GNU make, then that's bad.

I agree. Is there any easy way to use GNU make only for make
check-mini-gmp? Either in test scripts, or automatically in that target
(the make used for the target itself doens't matter, it's the recursive
make, currently using $(MAKE), that really needs GNU make.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

t...@gmplib.org (Torbjörn Granlund) writes:

  I realise that mini-gmp's requirement of GNU make hurts GMP's build
  environment compatibility testing; until now, we have tried to use plain
  'make' on almost all systems.  If mini-gmp now forces the entire build
  to use GNU make, then that's bad.

I rewrote parts of the testclient.sh script to accomodate this.

I also cleaned out the (finally more than 400) failures.  Not all were
due to the ABI clash, some were related to system make vs GNU make.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

I realise that mini-gmp's requirement of GNU make hurts GMP's build
environment compatibility testing; until now, we have tried to use plain
'make' on almost all systems.  If mini-gmp now forces the entire build
to use GNU make, then that's bad.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

There seem to be more purely arithmetic bugs in mini-gmp.  The latest is
this one:

https://gmplib.org/devel/tm/gmp/check/failure/bwldeb64v7.gmplib.org-dyn-fat:64.txt

So we might have 3 pending arithmetic bugs in mini-gmp (unless the gcd
one is a clang bug).

(This is why cleaning out 200 failures takes time, there are serious
failures lurking in their midst.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

OK, now the real havoc struck, more than 200 unexpected failures due to
mini-gmp.  I have no idea why they happened now and not sooner.

I clicked on a few, and thet all fail due to the already known ABI
incompatibility.

I will save lots of time by simply disable mini-gmp testing for now.
(Cleaning up the current failures will take sveral hours.)


-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

t...@gmplib.org (Torbjörn Granlund) writes:

  There is now some inconsistencies between the reporting status and the
  expected_* files.  This is evidenced by green lines in the expected
  section of e.g., .
  
I spotted the problem.  The   is for the html file
(devel/mini-gmp-status.html).  The expected files are plain text files
matching build reporting files, no html going on there.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

There is now some inconsistencies between the reporting status and the
expected_* files.  This is evidenced by green lines in the expected
section of e.g., .

[There is also something wrong with your pushes, not causing mail
notification to be made.  Make sure you trust the proper files in ~hg;
you should be getting an error/warning about this from the hg commands.]

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  Should we also drop the "mini-gmp/" prefix from files named in that
  ChangeLog?
  
That would be a bit neater.


-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> I will create a separate ChangeLog in mini-gmp's subdirectory and move
> mini-gmp entries into it.

Should we also drop the "mini-gmp/" prefix from files named in that
ChangeLog?

> I think we should also consider making mini-gmp a separate GNU
> project. Doing that might be good for its visibility, and also make
> users realise that it is in fact a separate project with different
> development principles and quality requirements.

What would such a separation mean in practice? We could do separate
releases (release announcements are always good for visibility) and a
separate version number. But I think I'd prefer to keep it in the the
main gmp repo, and I hope we'll sort out the testing issues in a week or
two so it can benefit from gmp's testing infrastructure.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> ni...@lysator.liu.se (Niels Möller) writes:
>
>   I tried to mark the mini-gmp t-limbs section with , but saving my
>   changes from emacs failed, 
>   
> My bad, please try now.

Worked better now.

>   How, more precisely? Just delete the corresponding files? E.g,
>   
> shell.gmplib.org-dyn-gcc49-g++49:64.txt
> shell.gmplib.org-dyn-gcc49-g++49:64.txt.gz
>   
> Yes, if permission allows it.  :-)

Done, 6 files deleted.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

ni...@lysator.liu.se (Niels Möller) writes:

  I tried to mark the mini-gmp t-limbs section with , but saving my
  changes from emacs failed, 
  
My bad, please try now.

  > If you check in a fix for any of these problems, please strike the
  > entire item by using   Please also clean up the
  > ~gmp/gmp/check/failure report files.
  
  How, more precisely? Just delete the corresponding files? E.g,
  
shell.gmplib.org-dyn-gcc49-g++49:64.txt
shell.gmplib.org-dyn-gcc49-g++49:64.txt.gz
  
Yes, if permission allows it.  :-)

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> For those issues which result in a check/failure entry, I have marked
> them as "expected" (by means of ~gmp/gmp/expected_check_failures).  I
> did that to make sure these failures do not hide any problems with GMP
> itself.

I tried to mark the mini-gmp t-limbs section with , but saving my
changes from emacs failed, 

  Cannot write backup file; backing up in /home/nisse/%backup%~
  basic-save-buffer-2: Doing chmod: operation not permitted, 
/home/gmp/gmp/expected_check_failures

> If you check in a fix for any of these problems, please strike the
> entire item by using   Please also clean up the
> ~gmp/gmp/check/failure report files.

How, more precisely? Just delete the corresponding files? E.g,

  shell.gmplib.org-dyn-gcc49-g++49:64.txt
  shell.gmplib.org-dyn-gcc49-g++49:64.txt.gz

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

t...@gmplib.org (Torbjörn Granlund) writes:

  6. The t-limbs test triggers an assert.  It looks like any config could
 trigger this, but thus far it has only happened on
 shell.gmplib.org-dyn-gcc49-g++49:64, ivydeb32v7.gmplib.org-stat-fat:32.
  
To save me time, I checked in a trivial change to a mini-gmp's Makefile
allowing the separate build of tests.  This gives the test system a
chance to bump off t-limbs which fails for a new config each night with
the same bogus assert.  (This mechanism exist to allow testing for
configs with broken environments.  It is usually used selectively.)

I will create a separate ChangeLog in mini-gmp's subdirectory and move
mini-gmp entries into it.  I think we should also consider making
mini-gmp a separate GNU project.  Doing that might be good for its
visibility, and also make users realise that it is in fact a separate
project with different development principles and quality requirements.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

"Marco Bodrato"  writes:

  Il Lun, 14 Novembre 2016 3:49 pm, Torbjörn Granlund ha scritto:
  > 2. There are inexplicable timeouts (or possibly excessive memory use)
  >triggered by t-pprime_p for some of our systems:
  
  >Perhaps the test is simply too slow, or its time varies hugely, or it
  >hits an infinite loop.
  
  I reduced the size of operands and I hope this one (only) was healed.
  
I strongly doubt it was merely slowness.  The system g5.gmplib.org is
not slow, it has a 1.800 GHz PPC970.  This single test needed more than
600s there (for some unknown seed).

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Niels Möller]

t...@gmplib.org (Torbjörn Granlund) writes:

> I created a web page for tracking the mini-gmp problems:
>
>   https://gmplib.org/devel/mini-gmp-status.html

From that page:

  (P2) The seeding triggered by GMP_CHECK_RANDOMIZE is apparently based on
  seconds since epoch (with some funny skew). This is not good enough, as
  tests which run at the same time will get the same seed. 

Seed is currently time(NULL) + getpid(), so a loop like

  while GMP_CHECK_RANDOMIZE=0 ./t-foo ; do : ; done

will typically use an incrementing sequence of seeds, and not repeat
until the pid space wraps around.

Main gmp tests seem to use /dev/urandom if available, falling back to
gettimeofday if available, otherwise only the time function (no pid).

For mini-gmp tests, it was my intention that tests should be able to run
standalone from the gmp tree (assuming gmp and GNU make is installed on
the system, just "make -C tests check" in a mini-gmp source tree should
work). So it shouldn't depend on any configure tests, and ideally should
use only standard C functions, or the most mainstream functions outside
of the standard. (And depending on getpid is not ideal... but a
significant improvement over only time(NULL)).

Attempting to open /dev/urandom seems very reasonable to me. If we do
that, the remaining question is how elaborate a fallback we need if
opening fails.

If really needed, we could also propagate certain gmp configure values
in the make check-mini-gmp target, e.g., HAVE_GETPID and
HAVE_GETTIMEOFDAY.

/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677.
Internet email is subject to wholesale government surveillance.
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Torbjörn Granlund]

I created a web page for tracking the mini-gmp problems:

  https://gmplib.org/devel/mini-gmp-status.html

I'd like to stress that mini-gmp problems do not affect the quality of
GMP itself.  GMP uses mini-gmp for some bootstrapping tasks, but that's
known to work properly for both 32-bit and 64-bit machines.

For comitters:

For those issues which result in a check/failure entry, I have marked
them as "expected" (by means of ~gmp/gmp/expected_check_failures).  I
did that to make sure these failures do not hide any problems with GMP
itself.

If you check in a fix for any of these problems, please strike the
entire item by using   Please also clean up the
~gmp/gmp/check/failure report files.

-- 
Torbjörn
Please encrypt, key id 0xC8601622
___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel

Re: Help stabilising mini-gmp

[Marco Bodrato]

Ciao,

Il Lun, 14 Novembre 2016 3:49 pm, Torbjörn Granlund ha scritto:
> 2. There are inexplicable timeouts (or possibly excessive memory use)
>triggered by t-pprime_p for some of our systems:

>Perhaps the test is simply too slow, or its time varies hugely, or it
>hits an infinite loop.

I reduced the size of operands and I hope this one (only) was healed.

Best regards,
mb

-- 
http://bodrato.it/papers/

___
gmp-devel mailing list
gmp-devel@gmplib.org
https://gmplib.org/mailman/listinfo/gmp-devel