Re: [Freeipa-devel] Unit tests failing on F18
On 10/18/2012 12:04 AM, Rob Crittenden wrote: Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... Thanks! Martin I built upon your patch and have something that seems to work at least somewhat. I'm getting some unexpected test failures when running the entire suite but no NSS shutdown errors. I haven't had a chance to really investigate everything yet, sending this out as a work-in-progress in case you want to take a look. rob Yeah, this is great! I tested a fresh build+install on Fedora 18 with your patch and all tests succeeded. So as for F18, I am inclined to ACK the patch as is. I am just not sure that this will work on platforms with Python version 2.7, xmlrpclib is different there. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Unit tests failing on F18
Martin Kosek wrote: On 10/18/2012 12:04 AM, Rob Crittenden wrote: Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... Thanks! Martin I built upon your patch and have something that seems to work at least somewhat. I'm getting some unexpected test failures when running the entire suite but no NSS shutdown errors. I haven't had a chance to really investigate everything yet, sending this out as a work-in-progress in case you want to take a look. rob Yeah, this is great! I tested a fresh build+install on Fedora 18 with your patch and all tests succeeded. So as for F18, I am inclined to ACK the patch as is. I am just not sure that this will work on platforms with Python version 2.7, xmlrpclib is different there. Martin Here is my first crack at fixing that too. It requires a bunch of run time juggling though. rob From c53e283986f2b00db53e28009829ba09d62930aa Mon Sep 17 00:00:00 2001 From: Rob Crittenden rcrit...@redhat.com Date: Wed, 17 Oct 2012 16:58:54 -0400 Subject: [PATCH] Close connection after each request, avoid NSS shutdown problem. The unit tests were failing when executed against an Apache server in F-18 due to dangling references causing NSS shutdown to fail. --- ipalib/rpc.py | 30 +- ipapython/nsslib.py | 6 ++ 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/ipalib/rpc.py b/ipalib/rpc.py index e97536d9de5c455d3ff58c081fca37f16d087370..8389396e0e23623b5edb60d634041949f95711ce 100644 --- a/ipalib/rpc.py +++ b/ipalib/rpc.py @@ -257,16 +257,24 @@ class SSLTransport(LanguageAwareTransport): # If we an existing connection exists using the same NSS database # there is no need to re-initialize. Pass thsi into the NSS # connection creator. +if sys.version_info (2, 6): +if self._connection and host == self._connection[0]: +return self._connection[1] + dbdir = '/etc/pki/nssdb' no_init = self.__nss_initialized(dbdir) -(major, minor, micro, releaselevel, serial) = sys.version_info -if major == 2 and minor 7: +if sys.version_info (2, 7): conn = NSSHTTPS(host, 443, dbdir=dbdir, no_init=no_init) else: conn = NSSConnection(host, 443, dbdir=dbdir, no_init=no_init) self.dbdir=dbdir + conn.connect() -return conn +if sys.version_info (2, 7): +return conn +else: +self._connection = host, conn +return self._connection[1] class KerbTransport(SSLTransport): @@ -331,6 +339,13 @@ class KerbTransport(SSLTransport): return (host, extra_headers, x509) + +def
Re: [Freeipa-devel] Unit tests failing on F18
Martin Kosek wrote:
Hello,
I was investigating global unit test failure on Fedora 18 for most of today, I
would like to share results I found so far.
Unit test and its related scripts on F18 now reports NSS BUSY exception, just
like this one:
# ./make-testcert
Traceback (most recent call last):
File ./make-testcert, line 134, in module
sys.exit(makecert(reqdir))
File ./make-testcert, line 111, in makecert
add=True)
File ./make-testcert, line 68, in run
result = self.execute(method, *args, **options)
File /root/freeipa-master2/ipalib/backend.py, line 146, in execute
raise error #pylint: disable=E0702
ipalib.errors.NetworkError: cannot connect to
'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053]
(SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.
Something In F18 must have changed, this worked before... But leaked
NSSConnection objects without proper close() now ends with the exception above.
In case of make-testcert script, the exception is raised because the script
does the following procedure:
1) connect, do one command
2) disconnect
3) connect, do second command
However, during disconnect, NSSConnection is leaked which makes NSS very
uncomfortable during second connection atempt (and nss_shutdown()). I managed
to fix this issue with attached patch. ./make-testcert or ./make-test
tests/test_xmlrpc/test_group_plugin.py works fine now.
But global ./make-test still fails, I think there is some remaining
NSSConnection leak, I suspect there is something wrong with how we use our
context (threading.local object). It looses a connection or some other thread
invoked in ldap2 module may be kicking in, here is my debug output:
CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec
Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in
/root/.ipa/.dmpw
Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok
tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT
150714476: GET languages
CONTEXT[xmlclient] = None
The connection is in the context, but then something happens and it is gone.
Then, unit tests try to connect again and NSS fails.
I would be really glad if somebody with a knowledge of NSS or how threads in
Python/IPA work could give me some advice...
Thanks!
Martin
I built upon your patch and have something that seems to work at least
somewhat. I'm getting some unexpected test failures when running the
entire suite but no NSS shutdown errors. I haven't had a chance to
really investigate everything yet, sending this out as a
work-in-progress in case you want to take a look.
rob
From ad3f420395c8e4fc24d9ab3aa6f53641f188efc6 Mon Sep 17 00:00:00 2001
From: Rob Crittenden rcrit...@redhat.com
Date: Wed, 17 Oct 2012 16:58:54 -0400
Subject: [PATCH] candidate for fixing test execution against httpd
---
ipalib/rpc.py | 22 +++---
ipapython/nsslib.py | 6 ++
2 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/ipalib/rpc.py b/ipalib/rpc.py
index e97536d..6de45cf 100644
--- a/ipalib/rpc.py
+++ b/ipalib/rpc.py
@@ -257,6 +257,9 @@ class SSLTransport(LanguageAwareTransport):
# If we an existing connection exists using the same NSS database
# there is no need to re-initialize. Pass thsi into the NSS
# connection creator.
+if self._connection and host == self._connection[0]:
+return self._connection[1]
+
dbdir = '/etc/pki/nssdb'
no_init = self.__nss_initialized(dbdir)
(major, minor, micro, releaselevel, serial) = sys.version_info
@@ -265,8 +268,10 @@ class SSLTransport(LanguageAwareTransport):
else:
conn = NSSConnection(host, 443, dbdir=dbdir, no_init=no_init)
self.dbdir=dbdir
+
conn.connect()
-return conn
+self._connection = host, conn
+return self._connection[1]
class KerbTransport(SSLTransport):
@@ -331,6 +336,13 @@ class KerbTransport(SSLTransport):
return (host, extra_headers, x509)
+
+def single_request(self, host, handler, request_body, verbose=0):
+try:
+return SSLTransport.single_request(self, host, handler, request_body, verbose)
+finally:
+self.close()
+
def parse_response(self, response):
session_cookie = response.getheader('Set-Cookie')
if session_cookie:
@@ -371,7 +383,8 @@ class xmlclient(Connectible):
if not hasattr(self.conn, '_ServerProxy__transport'):
return None
-if type(self.conn._ServerProxy__transport) in (KerbTransport, DelegatedKerbTransport):
+if (isinstance(self.conn._ServerProxy__transport, KerbTransport) or
+isinstance(self.conn._ServerProxy__transport, DelegatedKerbTransport)):
scheme = https
else:
scheme = http
@@ -493,7 +506,10 @@ class xmlclient(Connectible):
return serverproxy
Re: [Freeipa-devel] Unit tests failing on F18
On 10/12/2012 06:16 PM, John Dennis wrote: On 10/12/2012 11:20 AM, Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... O.K. I'll take a look at it. I seem to recall Rob looked into something similar a couple of days ago. Rob, do you have any additional information to share? Great, with your NSS+Python knowledge this should be walk in the park :-) Any luck with investigation of this issue? Thanks, Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Unit tests failing on F18
On 10/12/2012 06:21 PM, Rob Crittenden wrote: John Dennis wrote: On 10/12/2012 11:20 AM, Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... O.K. I'll take a look at it. I seem to recall Rob looked into something similar a couple of days ago. Rob, do you have any additional information to share? Nothing to add. I looked at this briefly yesterday but nothing jumped out at me. I think this may be unrelated to the previous problem I investigated. It very well could be we are not closing a connection somewhere. rob Yes, we are not. I fixed it in one place (patch in my original e-mail). With the patch, one could run at least some parts of the unit test. I assume we are simply not treating initializing and shutting down of NSS correctly in some places (also with regards to threads), but I have not found a fix yet. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Unit tests failing on F18
Martin Kosek wrote: On 10/12/2012 06:21 PM, Rob Crittenden wrote: John Dennis wrote: On 10/12/2012 11:20 AM, Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... O.K. I'll take a look at it. I seem to recall Rob looked into something similar a couple of days ago. Rob, do you have any additional information to share? Nothing to add. I looked at this briefly yesterday but nothing jumped out at me. I think this may be unrelated to the previous problem I investigated. It very well could be we are not closing a connection somewhere. rob Yes, we are not. I fixed it in one place (patch in my original e-mail). With the patch, one could run at least some parts of the unit test. I assume we are simply not treating initializing and shutting down of NSS correctly in some places (also with regards to threads), but I have not found a fix yet. If you modify tests/test_xmlrpc/xmlrpc_test.py and remove request.destroy_context() from tearDown then the tests will pass. But this shouldn't be necessary, we really want to clean up the context between requests. I'm guessing that we're being affected by an external change. It could be in a number of places: httplib, xmlrpclib, NSS, etc. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Unit tests failing on F18
On 10/15/2012 02:27 AM, Martin Kosek wrote: On 10/12/2012 06:16 PM, John Dennis wrote: O.K. I'll take a look at it. I seem to recall Rob looked into something similar a couple of days ago. Rob, do you have any additional information to share? Great, with your NSS+Python knowledge this should be walk in the park :-) Any luck with investigation of this issue? Not yet, it's next on my to-do list. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] Unit tests failing on F18
Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... Thanks! Martin diff --git a/ipalib/rpc.py b/ipalib/rpc.py index e97536d..f19d221 100644 --- a/ipalib/rpc.py +++ b/ipalib/rpc.py @@ -250,6 +250,9 @@ class SSLTransport(LanguageAwareTransport): return False def make_connection(self, host): +if self._connection and host == self._connection[0]: +return self._connection[1] + host, self._extra_headers, x509 = self.get_host_info(host) # Python 2.7 changed the internal class used in xmlrpclib from # HTTP to HTTPConnection. We need to use the proper subclass @@ -265,8 +268,10 @@ class SSLTransport(LanguageAwareTransport): else: conn = NSSConnection(host, 443, dbdir=dbdir, no_init=no_init) self.dbdir=dbdir + conn.connect() -return conn +self._connection = host, conn +return self._connection[1] class KerbTransport(SSLTransport): @@ -493,7 +498,10 @@ class xmlclient(Connectible): return serverproxy def destroy_connection(self): -pass +conn = getattr(context, self.id, None) +if conn is not None: +conn = conn.conn._ServerProxy__transport +conn.close() def forward(self, name, *args, **kw): ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Unit tests failing on F18
On 10/12/2012 11:20 AM, Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... O.K. I'll take a look at it. I seem to recall Rob looked into something similar a couple of days ago. Rob, do you have any additional information to share? -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Unit tests failing on F18
John Dennis wrote: On 10/12/2012 11:20 AM, Martin Kosek wrote: Hello, I was investigating global unit test failure on Fedora 18 for most of today, I would like to share results I found so far. Unit test and its related scripts on F18 now reports NSS BUSY exception, just like this one: # ./make-testcert Traceback (most recent call last): File ./make-testcert, line 134, in module sys.exit(makecert(reqdir)) File ./make-testcert, line 111, in makecert add=True) File ./make-testcert, line 68, in run result = self.execute(method, *args, **options) File /root/freeipa-master2/ipalib/backend.py, line 146, in execute raise error #pylint: disable=E0702 ipalib.errors.NetworkError: cannot connect to 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053] (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. Something In F18 must have changed, this worked before... But leaked NSSConnection objects without proper close() now ends with the exception above. In case of make-testcert script, the exception is raised because the script does the following procedure: 1) connect, do one command 2) disconnect 3) connect, do second command However, during disconnect, NSSConnection is leaked which makes NSS very uncomfortable during second connection atempt (and nss_shutdown()). I managed to fix this issue with attached patch. ./make-testcert or ./make-test tests/test_xmlrpc/test_group_plugin.py works fine now. But global ./make-test still fails, I think there is some remaining NSSConnection leak, I suspect there is something wrong with how we use our context (threading.local object). It looses a connection or some other thread invoked in ldap2 module may be kicking in, here is my debug output: CONTEXT[xmlclient] = ipalib.request.Connection object at 0x9a1f5ec Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in /root/.ipa/.dmpw Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT 150714476: GET languages CONTEXT[xmlclient] = None The connection is in the context, but then something happens and it is gone. Then, unit tests try to connect again and NSS fails. I would be really glad if somebody with a knowledge of NSS or how threads in Python/IPA work could give me some advice... O.K. I'll take a look at it. I seem to recall Rob looked into something similar a couple of days ago. Rob, do you have any additional information to share? Nothing to add. I looked at this briefly yesterday but nothing jumped out at me. I think this may be unrelated to the previous problem I investigated. It very well could be we are not closing a connection somewhere. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
