Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Olly Betts]

Control: tag -1 + fixed-upstream patch

On Mon, Mar 13, 2017 at 04:24:01PM +1300, Olly Betts wrote:
> I think this is worthwhile addressing in xapian-core.

I've pushed a fix for this upstream:

https://trac.xapian.org/changeset/9ed4dec1cb51c241cf7eb0b927f0e3122c2676a2/git

Once the release is out I'll push this to unstable.

Cheers,
Olly

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Olly Betts]

Control: tag -1 -moreinfo
Control: tag -1 +upstream

This came up on #debian-devel and jrtc27 (James Clarke) spotted the underlying
cause - from "man getaddrinfo":

| If  hints.ai_flags includes the AI_ADDRCONFIG flag, then IPv4 addresses
| are returned in the list pointed to by res only if the local system has
| at  least  one IPv4 address configured, and IPv6 addresses are returned
| only if the local system has at least one IPv6 address configured.  The
| loopback  address is not considered for this case as valid as a config‐
| ured address.

The last sentence about loopback isn't in POSIX, but is in RFC3493.  And
this is the problem here - inside the chroot, pbuilder has removed all but
loopback, so getaddrinfo() won't return an IPv4 address as there's no IPv4
address configured (not considering loopback).  By similar logic, it won't
return an IPv6 address either.

Practically, you wouldn't actually use Xapian's remote backend on a box
with only loopback - you'd have to run the server on the same box, which
makes little sense.  But this setup is likely to be seen in testsuites for
both xapian-core, and for things which build on it and use the remote
backend, and disabling networking except localhost is sane for
autobuilders, so I think this is worthwhile addressing in xapian-core.

Cheers,
Olly

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Olly Betts]

On Thu, Feb 16, 2017 at 03:21:01PM -0800, Daniel Schepler wrote:
> On Thu, Feb 16, 2017 at 1:37 PM, Olly Betts  wrote:
> > One thought - is /etc/gai.conf present in the chroot with any uncommented
> > content?
> 
> Nope, it's just the normal template /etc/gai.conf file with everything
> commented.
> 
> I think pbuilder does blank out /etc/resolv.conf in the chroot to
> prevent that from confusing things.  Is it possible that for some
> reason, the code now requires a DNS server to be available even to
> bind to a numeric IP address?

I just tried temporarily emptying /etc/resolv.conf on my dev box and
"./apitest -bremotetcp_glass" still works (using HEAD of RELEASE/1.4
branch, which hasn't changed in any relevant way since 1.4.3).

Cheers,
Olly

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Daniel Schepler]

On Thu, Feb 16, 2017 at 1:37 PM, Olly Betts  wrote:
> One thought - is /etc/gai.conf present in the chroot with any uncommented
> content?

Nope, it's just the normal template /etc/gai.conf file with everything
commented.

I think pbuilder does blank out /etc/resolv.conf in the chroot to
prevent that from confusing things.  Is it possible that for some
reason, the code now requires a DNS server to be available even to
bind to a numeric IP address?
-- 
Daniel

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Olly Betts]

On Thu, Feb 16, 2017 at 09:19:35AM -0800, Daniel Schepler wrote:
> > Daniel: What version of pbuilder are you using?  If it's more than a
> > year old, then upgrading will probably fix this.  If not, then either
> > that fix has regressed in pbuilder regression, or else something more
> > complex is going on.
> 
> I was still able to reproduce it with the latest pbuilder version
> (0.228.4); and even though my previous pbuilder version was somewhat
> out of date, it was still new enough to have the code setting up an
> "lo" interface.  So, as you said, something else must be going on
> here.

OK, thanks for checking with the latest.

> Incidentally, the build did work until recently - the last version I
> have with a successful build is 1.2.23-1, and I think it started
> failing with the 1.4.* series.

As I said above:

| The reproducible build used to work but it looks like it regressed in
| 1.3.4-1:
| 
| https://tests.reproducible-builds.org/debian/history/xapian-core.html
| 
| That's very suspicious as that's the first version packaged for Debian
| which used getaddrinfo() (1.2.x used gethostbyname(), etc), and the
| history shows that 1.2.x continued to build OK at the same time.  So
| that seems to point firmly to this not being a pbuilder regression.
| 
| But I've no idea what the cause is.  My only thoughts are that perhaps
| there's only IPv6 loopback (but then 1.2.x wouldn't work), or that
| getaddrinfo() is interacting badly with the restricted networking.

One thought - is /etc/gai.conf present in the chroot with any uncommented
content?

Cheers,
Olly

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Daniel Schepler]

On Sun, Jan 29, 2017 at 9:06 PM, Olly Betts  wrote:
> On Sun, Jan 29, 2017 at 06:00:06PM -0400, David Bremner wrote:
>> Daniel Schepler  writes:
>>
>> > Source: xapian-core
>> > Version: 1.4.3-1
>> > Severity: normal
>>
>> I'm not the xapian maintainer, but I consider this not a bug. Using
>> localhost networking for testing is common, and not forbidden by Debian
>> policy afaik.
>
> Policy's wording isn't entirely clear here - from §4.9:
>
> | For packages in the main archive, no required targets may attempt
> | network access.
>
> My reading (and I assume what is intended) is that this is not meant to
> include localhost, but localhost is the loopback *network* interface, so
> I can see how people could interpret it to exclude that too.
>
> But I think having localhost available is valuable, and if the latter
> interpretation was being argued for, I think policy needs clarifying.
> As David points out, having localhost available allows testsuites to test
> network functionality.  If we can't run such tests we could miss
> Debian-specific patches introducing regressions which the testsuite would
> have caught if we'd been able to run it.
>
> But pbuilder with USENETWORK=no should actually still provide lo.
> It didn't originally but that was reported and addressed:
>
> https://bugs.debian.org/753944
>
> Daniel: What version of pbuilder are you using?  If it's more than a
> year old, then upgrading will probably fix this.  If not, then either
> that fix has regressed in pbuilder regression, or else something more
> complex is going on.

I was still able to reproduce it with the latest pbuilder version
(0.228.4); and even though my previous pbuilder version was somewhat
out of date, it was still new enough to have the code setting up an
"lo" interface.  So, as you said, something else must be going on
here.

Incidentally, the build did work until recently - the last version I
have with a successful build is 1.2.23-1, and I think it started
failing with the 1.4.* series.
-- 
Daniel

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Olly Betts]

On Tue, Jan 31, 2017 at 10:09:57PM +0100, Georg Faerber wrote:
> > Daniel: What version of pbuilder are you using?  If it's more than a
> > year old, then upgrading will probably fix this.  If not, then either
> > that fix has regressed in pbuilder regression, or else something more
> > complex is going on.
> 
> This is happening here [1] as well, which is a recent pbuilder version,
> I guess. I'm wondering if this is a bug in pbuilder or a wrapper around.
> 
> Cheers,
> Georg
> (who debugged for some hours a similar problem over here [2]...)

Did you reach any useful conclusions?

FWIW, the code that appears to fail is:

struct addrinfo hints;
memset(, 0, sizeof(struct addrinfo));
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE | AI_ADDRCONFIG | AI_NUMERICSERV;
hints.ai_protocol = 0;
hints.ai_canonname = NULL;
hints.ai_addr = NULL;
hints.ai_next = NULL;

const char * node = host.empty() ? NULL : host.c_str();
struct addrinfo *result;
int s = getaddrinfo(node, str(port).c_str(), , );
 
if (s != 0) {
throw Xapian::NetworkError("Couldn't resolve host " + host,
   eai_to_xapian(s));
}

Where host is "127.0.0.1" and port is "1239".  It seems the error is:

EAI_NONAME
  The  node  or service is not known; or both node and service are
  NULL; or AI_NUMERICSERV was specified in hints.ai_flags and ser‐
  vice was not a numeric port-number string.

The reproducible build used to work but it looks like it regressed in
1.3.4-1:

https://tests.reproducible-builds.org/debian/history/xapian-core.html

That's very suspicious as that's the first version packaged for Debian
which used getaddrinfo() (1.2.x used gethostbyname(), etc), and the
history shows that 1.2.x continued to build OK at the same time.  So
that seems to point firmly to this not being a pbuilder regression.

But I've no idea what the cause is.  My only thoughts are that perhaps
there's only IPv6 loopback (but then 1.2.x wouldn't work), or that
getaddrinfo() is interacting badly with the restricted networking.

Cheers,
Olly

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Georg Faerber]

Hi Olly,

> Daniel: What version of pbuilder are you using?  If it's more than a
> year old, then upgrading will probably fix this.  If not, then either
> that fix has regressed in pbuilder regression, or else something more
> complex is going on.

This is happening here [1] as well, which is a recent pbuilder version,
I guess. I'm wondering if this is a bug in pbuilder or a wrapper around.

Cheers,
Georg
(who debugged for some hours a similar problem over here [2]...)


[1] 
https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/xapian-core_1.4.3-1.rbuild.log
[2] 
https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/schleuder_3.0.1-1.rbuild.log


signature.asc
Description: Digital signature

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Olly Betts]

On Sun, Jan 29, 2017 at 06:00:06PM -0400, David Bremner wrote:
> Daniel Schepler  writes:
> 
> > Source: xapian-core
> > Version: 1.4.3-1
> > Severity: normal
> 
> I'm not the xapian maintainer, but I consider this not a bug. Using
> localhost networking for testing is common, and not forbidden by Debian
> policy afaik.

Policy's wording isn't entirely clear here - from §4.9:

| For packages in the main archive, no required targets may attempt
| network access.

My reading (and I assume what is intended) is that this is not meant to
include localhost, but localhost is the loopback *network* interface, so
I can see how people could interpret it to exclude that too.

But I think having localhost available is valuable, and if the latter
interpretation was being argued for, I think policy needs clarifying.
As David points out, having localhost available allows testsuites to test
network functionality.  If we can't run such tests we could miss
Debian-specific patches introducing regressions which the testsuite would
have caught if we'd been able to run it.

But pbuilder with USENETWORK=no should actually still provide lo.
It didn't originally but that was reported and addressed:

https://bugs.debian.org/753944

Daniel: What version of pbuilder are you using?  If it's more than a
year old, then upgrading will probably fix this.  If not, then either
that fix has regressed in pbuilder regression, or else something more
complex is going on.

Cheers,
Olly

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[David Bremner]

Daniel Schepler  writes:

> Source: xapian-core
> Version: 1.4.3-1
> Severity: normal
>

I'm not the xapian maintainer, but I consider this not a bug. Using
localhost networking for testing is common, and not forbidden by Debian
policy afaik.

d

Bug#853107: xapian-core: FTBFS under pbuilder with USENETWORK=no

[Daniel Schepler]

Source: xapian-core
Version: 1.4.3-1
Severity: normal

>From my pbuilder build log:

...
Running test: qpmemoryleak1... EXCEPTION std::string Failed to get
'Listening...' from command '../bin/xapian-tcpsrv --one-shot
--interface 127.0.0.1 --port 1239 -t30 --writable .chert/dbw'
(output: Starting writable serv
er on host 127.0.0.1, port 1239
NetworkError: Couldn't resolve host 127.0.0.1 (Name or service not known)
)
Running test: closedb4... EXCEPTION std::string Failed to get
'Listening...' from command '../bin/xapian-tcpsrv --one-shot
--interface 127.0.0.1 --port 1239 -t30 --writable .chert/dbw'
(output: Starting writable server on
host 127.0.0.1, port 1239
NetworkError: Couldn't resolve host 127.0.0.1 (Name or service not known)
)
Running test: closedb10... EXCEPTION std::string Failed to get
'Listening...' from command '../bin/xapian-tcpsrv --one-shot
--interface 127.0.0.1 --port 1239 -t30 --writable .chert/dbw'
(output: Starting writable server o
n host 127.0.0.1, port 1239
NetworkError: Couldn't resolve host 127.0.0.1 (Name or service not known)
)
Running test: writeread1... EXCEPTION std::string Failed to get
'Listening...' from command '../bin/xapian-tcpsrv --one-shot
--interface 127.0.0.1 --port 1239 -t30 --writable .chert/dbw'
(output: Starting writable server
on host 127.0.0.1, port 1239
NetworkError: Couldn't resolve host 127.0.0.1 (Name or service not known)
)
Running test: valuestats1... EXCEPTION std::string Failed to get
'Listening...' from command '../bin/xapian-tcpsrv --one-shot
--interface 127.0.0.1 --port 1239 -t30 --writable .chert/dbw'
(output: Starting writable server
on host 127.0.0.1, port 1239
NetworkError: Couldn't resolve host 127.0.0.1 (Name or service not known)
)
Running test: valueweightsource5... SKIPPED
./apitest backend remotetcp_chert: 1 tests passed, 286 failed, 2 skipped.
./apitest total: 2441 tests passed, 572 failed, 41 skipped.
FAIL: apitest
Running test: exception1... ok
Running test: refcnt1... ok
Running test: refcnt2... ok
...
PASS: unittest

1 of 4 tests failed
Please report to https://xapian.org/bugs

Makefile:1003: recipe for target 'check-TESTS' failed
make[5]: *** [check-TESTS] Error 1
make[5]: Leaving directory '/build/xapian-core-1.4.3/build/tests'
Makefile:1126: recipe for target 'check-am' failed
make[4]: *** [check-am] Error 2
make[4]: Leaving directory '/build/xapian-core-1.4.3/build/tests'
Makefile:1129: recipe for target 'check' failed
make[3]: *** [check] Error 2
make[3]: Leaving directory '/build/xapian-core-1.4.3/build/tests'
Makefile:3209: recipe for target 'check-recursive' failed
make[2]: *** [check-recursive] Error 1
make[2]: Leaving directory '/build/xapian-core-1.4.3/build'
Makefile:3497: recipe for target 'check' failed
make[1]: *** [check] Error 2
make[1]: Leaving directory '/build/xapian-core-1.4.3/build'
debian/rules:167: recipe for target 'build-stamp' failed
make: *** [build-stamp] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
-- 
Daniel Schepler