Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Emilio Pozuelo Monfort]

On Sat, 7 Apr 2018 10:51:27 +0200 Emilio Pozuelo Monfort  
wrote:
> On Tue, 09 Jan 2018 20:07:53 + Ludovico Cavedon  
> wrote:
> > Hi,
> > 
> >  I have an update on this: I have a patch for upstream review at
> > https://github.com/ntop/nDPI/pull/506.
> > It fixes this issue, but unittests still fail on s90x (and I guess on the
> > other big endian archs), so no new upload for now, until I debug that.
> 
> Got any more information on those test failures, e.g. an upstream bug report?

Ping? Any news here? I would suggest to update the package with the fix that you
already have, and if s390x still fails then that can be looked at afterwards.

Emilio

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Emilio Pozuelo Monfort]

On Tue, 09 Jan 2018 20:07:53 + Ludovico Cavedon  wrote:
> Hi,
> 
>  I have an update on this: I have a patch for upstream review at
> https://github.com/ntop/nDPI/pull/506.
> It fixes this issue, but unittests still fail on s90x (and I guess on the
> other big endian archs), so no new upload for now, until I debug that.

Got any more information on those test failures, e.g. an upstream bug report?

Emilio

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Ludovico Cavedon]

Hi,

 I have an update on this: I have a patch for upstream review at
https://github.com/ntop/nDPI/pull/506.
It fixes this issue, but unittests still fail on s90x (and I guess on the
other big endian archs), so no new upload for now, until I debug that.

Ludovico

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Ludovico Cavedon]

Thank you for looking into this.

On Tue, Jan 2, 2018 at 12:15 PM Aaron M. Ucko  wrote:

> > It so happens that I was having a brief look already :)
>
> Great, thanks!
>
> > The hang occurs inside the above while loop. Notice that the value
> > loaded into "label" inside the loop never changes and this is the only
> > variable the loop condition depends on. Therefore, if the initial loop
> > condition is true, the program will loop forever.
>
> So I see.  Perhaps the intent was to update mpls after bumping ip_offset.
>

Yes, there is clearly something missing.
Probably
mpls = (struct ndpi_mpls_header *) [ip_offset];
is missing from inside the loop.

I will check on this before the end of the week, follow up with upstream,
and patch.

Thanks,
Ludovico

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Aurelien Jarno]

On 2018-01-02 10:57, Aaron M. Ucko wrote:
> Source: ndpi
> Version: 2.2-1
> Severity: serious
> Tags: upstream
> Justification: fails to build from source (but built successfully in the past)
> User: debian-m...@lists.debian.org
> Usertags: mips
> 
> Builds of ndpi for mips, s390x, and the non-release architectures
> powerpc and ppc64 all failed because running the test suite hit the
> autobuilders' inactivity timeouts.  These timeouts are generous enough
> (600 minutes on ppc64, 150 minutes on the other three architectures)
> that hitting them generally indicates that something managed to hang
> or spin indefinitely.  However, I see that the hppa build ran for a
> long time before terminating on its own (albeit with test suite
> errors), so you may simply need to add progress indicators for the
> sake of slow architectures.  (These are inactivity timeouts, so any
> output to stdout or stderr resets them.)

s390x and ppc64 are definitely not slow architectures. The common
between all these architectures is that there are big endian.

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Aaron M. Ucko]

> It so happens that I was having a brief look already :)

Great, thanks!

> The hang occurs inside the above while loop. Notice that the value
> loaded into "label" inside the loop never changes and this is the only
> variable the loop condition depends on. Therefore, if the initial loop
> condition is true, the program will loop forever.

So I see.  Perhaps the intent was to update mpls after bumping ip_offset.

> Firstly, the order of fields in a bitfield is ABI dependent so any
> architecture could legitimately rearrange all the fields giving wrong
> results. I also don't really understand what the programmer intended to
> happen when invoking ntohl on a bitfield.

Excellent question, though it's ironic that these failures are occurring
on architectures for which ntohl is a no-op. ;-) Please note that the
autobuilders for m68k and powerpcspe, the (non-release) big-endian
architectures on which this ndpi version nominally built fine, both run
with nocheck in DEB_BUILD_OPTIONS, so we shouldn't read too much into
those successes.

-- 
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[James Cowgill]

Hi,

On 02/01/18 15:57, Aaron M. Ucko wrote:
> Source: ndpi
> Version: 2.2-1
> Severity: serious
> Tags: upstream
> Justification: fails to build from source (but built successfully in the past)
> User: debian-m...@lists.debian.org
> Usertags: mips
> 
> Builds of ndpi for mips, s390x, and the non-release architectures
> powerpc and ppc64 all failed because running the test suite hit the
> autobuilders' inactivity timeouts.  These timeouts are generous enough
> (600 minutes on ppc64, 150 minutes on the other three architectures)
> that hitting them generally indicates that something managed to hang
> or spin indefinitely.  However, I see that the hppa build ran for a
> long time before terminating on its own (albeit with test suite
> errors), so you may simply need to add progress indicators for the
> sake of slow architectures.  (These are inactivity timeouts, so any
> output to stdout or stderr resets them.)
> 
> Could you please take a look?

It so happens that I was having a brief look already :)

The ndpiReader application is the one that hangs.

Extract from example/ndpi_util.c:750
>   case MPLS_UNI:
>   case MPLS_MULTI:
> mpls = (struct ndpi_mpls_header *) [ip_offset];
> label = ntohl(mpls->label);
> /* label = ntohl(*((u_int32_t*)[ip_offset])); */
> workflow->stats.mpls_count++;
> type = ETH_P_IP, ip_offset += 4;
>
> while((label & 0x100) != 0x100) {
>   ip_offset += 4;
>   label = ntohl(mpls->label);
> }

The hang occurs inside the above while loop. Notice that the value
loaded into "label" inside the loop never changes and this is the only
variable the loop condition depends on. Therefore, if the initial loop
condition is true, the program will loop forever.

The reason why this happens only on big-endian may be related to the
definition of ndpi_mpls_header:
> PACK_ON
> struct ndpi_mpls_header
> {
>   u_int32_t label:20, exp:3, s:1, ttl:8;
> } PACK_OFF;

Firstly, the order of fields in a bitfield is ABI dependent so any
architecture could legitimately rearrange all the fields giving wrong
results. I also don't really understand what the programmer intended to
happen when invoking ntohl on a bitfield. Probably teh code needs
rewriting to not use ndpi_mpls_header.

James



signature.asc
Description: OpenPGP digital signature

Bug#886133: ndpi: FTBFS on mips, s390x, powerpc, and ppc64: tests time out

[Aaron M. Ucko]

Source: ndpi
Version: 2.2-1
Severity: serious
Tags: upstream
Justification: fails to build from source (but built successfully in the past)
User: debian-m...@lists.debian.org
Usertags: mips

Builds of ndpi for mips, s390x, and the non-release architectures
powerpc and ppc64 all failed because running the test suite hit the
autobuilders' inactivity timeouts.  These timeouts are generous enough
(600 minutes on ppc64, 150 minutes on the other three architectures)
that hitting them generally indicates that something managed to hang
or spin indefinitely.  However, I see that the hppa build ran for a
long time before terminating on its own (albeit with test suite
errors), so you may simply need to add progress indicators for the
sake of slow architectures.  (These are inactivity timeouts, so any
output to stdout or stderr resets them.)

Could you please take a look?

Thanks!

-- 
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu