Preventing network access during nose doctest ?

2014-05-12 Thread Olivier Berger 
Hi.

I'm trying to fix #739222 where tests fail (- FTBFS) during execution
of nose's doctest plugin on something like :
 import rdflib

 g = rdflib.Graph()
 result = g.parse(http://www.w3.org/2000/10/swap/test/meet/white.rdf;)

 print(graph has %s statements. % len(g))
graph has 19 statements.

I'm puzzled : I'm invoking the tests run with :
PYBUILD_SYSTEM=custom \
PYBUILD_TEST_ARGS={interpreter} run_tests.py \
http_proxy= https_proxy= \
dh_auto_test --buildsystem=pybuild

where run_tests.py will invoke nose with --with-doctest, but even though
the HTTP proxy variables being set, they don't seem to be preventing
urllib2 to try to access the file.

So there doesn't seem to be a way to notice until the network is really
brought down :-/

Instructions at https://wiki.debian.org/Python/Pybuild suggest that the
variables help, but probably only for pip etc. ?


In any case, an obvious fix here could be to prevent running doctest at
all.

But there might be a smarter way, to fake the existence of such a page
(retrieved through urllib2) so as the tests can be run offline ?

If anyone has suggestions...

Many thanks in advance.

Best regards,
-- 
Olivier BERGER 
http://www-public.telecom-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut Mines-Telecom, Telecom SudParis, Evry (France)


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87siofqim6@inf-8660.int-evry.fr

Re: Preventing network access during nose doctest ?

2014-05-12 Thread Yaroslav Halchenko 


On Mon, 12 May 2014, Olivier Berger wrote:

 Hi.

 I'm trying to fix #739222 where tests fail (- FTBFS) during execution
 of nose's doctest plugin on something like :
  import rdflib

  g = rdflib.Graph()
  result = g.parse(http://www.w3.org/2000/10/swap/test/meet/white.rdf;)

  print(graph has %s statements. % len(g))
 graph has 19 statements.

 I'm puzzled : I'm invoking the tests run with :
   PYBUILD_SYSTEM=custom \
 PYBUILD_TEST_ARGS={interpreter} run_tests.py \
   http_proxy= https_proxy= \
   dh_auto_test --buildsystem=pybuild

 where run_tests.py will invoke nose with --with-doctest, but even though
 the HTTP proxy variables being set, they don't seem to be preventing
 urllib2 to try to access the file.

should work AFAICT:

$ python -c 'import rdflib; g = rdflib.Graph(); result = 
g.parse(http://www.w3.org/2000/10/swap/test/meet/white.rdf;); print(graph has 
%s statements. % len(g))'
graph has 19 statements.

$ http_proxy=http://127.0.0.1:9/ python -c 'import rdflib; g = rdflib.Graph(); 
result = g.parse(http://www.w3.org/2000/10/swap/test/meet/white.rdf;); 
print(graph has %s statements. % len(g))'
Traceback (most recent call last):
  File string, line 1, in module
  File /usr/lib/python2.7/dist-packages/rdflib/Graph.py, line 886, in parse
source = self.prepare_input_source(source, publicID)
  File /usr/lib/python2.7/dist-packages/rdflib/Graph.py, line 692, in 
prepare_input_source
input_source = URLInputSource(location)
  File /usr/lib/python2.7/dist-packages/rdflib/URLInputSource.py, line 21, in 
__init__
file = urlopen(req)
  File /usr/lib/python2.7/urllib2.py, line 127, in urlopen
return _opener.open(url, data, timeout)
  File /usr/lib/python2.7/urllib2.py, line 404, in open
response = self._open(req, data)
  File /usr/lib/python2.7/urllib2.py, line 422, in _open
'_open', req)
  File /usr/lib/python2.7/urllib2.py, line 382, in _call_chain
result = func(*args)
  File /usr/lib/python2.7/urllib2.py, line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
  File /usr/lib/python2.7/urllib2.py, line 1184, in do_open
raise URLError(err)
urllib2.URLError: urlopen error [Errno 111] Connection refused

make sure you exported them in debian/rules:
export  http_proxy=http://127.0.0.1:9/

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140512124702.gh8...@onerussian.com

Re: Preventing network access during nose doctest ?

2014-05-12 Thread Piotr Ożarowski 
[Olivier Berger, 2014-05-12]
 I'm trying to fix #739222 where tests fail (- FTBFS) during execution
 of nose's doctest plugin on something like :
  import rdflib
 
  g = rdflib.Graph()
  result = g.parse(http://www.w3.org/2000/10/swap/test/meet/white.rdf;)
 
  print(graph has %s statements. % len(g))
 graph has 19 statements.
 
 I'm puzzled : I'm invoking the tests run with :
   PYBUILD_SYSTEM=custom \
 PYBUILD_TEST_ARGS={interpreter} run_tests.py \
   http_proxy= https_proxy= \
   dh_auto_test --buildsystem=pybuild
 
 where run_tests.py will invoke nose with --with-doctest, but even though
 the HTTP proxy variables being set, they don't seem to be preventing
 urllib2 to try to access the file.

remove http_proxy= https_proxy= line and pybuild will try to block
internet connection (and you will be able to figure out which test
tries to access network and fix/disable it)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140512125355.ga4...@sts0.p1otr.com

Re: Preventing network access during nose doctest ?

2014-05-12 Thread Barry Warsaw 
On May 12, 2014, at 02:57 PM, Nicolas Dandrimont wrote:

Here, you're setting an empty http_proxy variable, which means don't use a
proxy. What you really want is to set the proxy to something that errors out,
e.g. http://127.0.0.1:9/ (the discard port on localhost).

And then, you'll need to disable the tests that need internet access, or
modify them so that they stop needing it.

Note that some test suites do legitimate http/https access during their
tests.  By legitimate I mean, they set up their own localhost special-port
service and vend files out of them.  You could call these integration tests,
but still, test suites do this and setting the proxies to the discard port
will break them.

I agree that *external* access must be blocked, but cases such as the above
must also be allowed.  Unfortunately I don't know of a better way then
unsetting the proxy envars for the test suite.

-Barry


signature.asc
Description: PGP signature

Re: Preventing network access during nose doctest ?

2014-05-12 Thread Jakub Wilk 

* Barry Warsaw ba...@debian.org, 2014-05-12, 09:41:
Note that some test suites do legitimate http/https access during their 
tests.  By legitimate I mean, they set up their own localhost 
special-port service and vend files out of them.


AFAICS urllib honours the no_proxy environment variable.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140512135255.ga...@jwilk.net

Re: Preventing network access during nose doctest ?

2014-05-12 Thread Olivier Berger 
Nicolas Dandrimont ol...@debian.org writes:

 * Olivier Berger olivier.ber...@telecom-sudparis.eu [2014-05-12 14:36:17 
 +0200]:

 Hi.
 
 I'm trying to fix #739222 where tests fail (- FTBFS) during execution
 of nose's doctest plugin on something like :
  import rdflib
 
  g = rdflib.Graph()
  result = 
 g.parse(http://www.w3.org/2000/10/swap/test/meet/white.rdf;)
 
  print(graph has %s statements. % len(g))
 graph has 19 statements.
 
 I'm puzzled : I'm invoking the tests run with :
  PYBUILD_SYSTEM=custom \
 PYBUILD_TEST_ARGS={interpreter} run_tests.py \
  http_proxy= https_proxy= \
  dh_auto_test --buildsystem=pybuild
 
 where run_tests.py will invoke nose with --with-doctest, but even though
 the HTTP proxy variables being set, they don't seem to be preventing
 urllib2 to try to access the file.

Thanks everyone for the hints.


 Hi,

 Here, you're setting an empty http_proxy variable, which means don't use a
 proxy. What you really want is to set the proxy to something that errors out,
 e.g. http://127.0.0.1:9/ (the discard port on localhost).

 And then, you'll need to disable the tests that need internet access, or 
 modify
 them so that they stop needing it.


Indeed I was overly confused : I had previously manually set them to
empty values to workaround the proxy setting made by pybuild internally,
so that the build would effectively access the network... and then I
forgot why I had done that... and of course I hadn't documented that ;)

Sorry about the bothering.

But, then there's still the option of maybe mocking the tests so that we
can proxy locally a copy of the document being fetched... but I'll let
that for another day ;)

Best regards,
-- 
Olivier BERGER 
http://www-public.telecom-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut Mines-Telecom, Telecom SudParis, Evry (France)


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87ppjjqf8k@inf-8660.int-evry.fr