Hello,
tl;dr; I want to add forms based auth in ipalib's rpc client and use
method 2.e) for API configuration. Will be used in otp tests (ipa-4-1
upstream). Want to know if you are OK with it. PoC implementation attached.
== Background ==
I want to call FreeIPA API from our Web UI CI tests.
Uses case are:
* speed up data preparation (the biggest time consumer in Web UI tests)
* prepare data or do stuff which could not be done in Web UI (little bit
part of https://fedorahosted.org/freeipa/ticket/4772)
For this task I want to reuse ipalib's rpcclient to avoid yet another
RPC client implementation.
Plan is to use it in OTP web ui tests.
== ipalib limitations ==
Please note that web ui tests could be run as a normal user on a machine
which is not a FreeIPA client, i.e. on developers machine with local
browser to observe the progress.
=== 1 Forms based authentication ===
ipalib doesn't support forms-based auth and kerberos might not be
available. I have investigated the option to add forms auth and managed
to write a PoC patch(attached) which enables it. Question is whether we
want to allow it in ipalib even for non-test uses cases. I think it
could be beneficial for some users.
=== 2 API/environment finalization in ipalib/__init__.py for tests ===
Tests which are run by make-test are initialized with
os.environ['IPA_UNIT_TEST_MODE'] = 'cli_test'
in such case, ipalib automatically finalizes API. Unfortunately the
default configuration is unusable if run on a machine without IPA
configuration files (~/.ipa/default.conf, /etc/ipa/default.conf, ...).
We can't even supply the values afterwards because api.env is 'set-once'
object.
There are several approaches which could be taken:
Proper fixes:
a) Finish the idea, mentioned in Registry, of storing plugin
registrations to allow multiple API creation with all plugins and
therefore custom instances of API. Would be probably quite a lot of work
because of wide use of global api object. I.e., various test can use
their own instance of api.
b) Remove the API initialization in ipalib/__init__.py. Initialize API
only in tests which use it. Without a) It might suffer from the same
issue as d)
c) Remove 'set-once' constraint from Env object to allow config change
after API initialization
Workarounds:
d) Change the os.env variable to a different than 'cli_test' value
before api import in uidriver.py to prevent the automatic
initialization. Init api in Web UI tests. Might cause issues if other
tests were run and initialized api before Web ui tests.
e) override values in Env object by a hack:
object.__setattr__(api.env, key, val)
api.env._Env__d[key] = val
Advantage is that it's self-contained in Web UI tests. Must be changed
back at the end of test not to interfere with other tests.
f) require IPA configuration file even on non-ipa client machines. Easy,
but not very pleasant for user - all required configuration values are
already present in webui test config file or in env variables ->
multiple files has to be changed when switching between testing
instances(frequent use case).
g) other
From longer perspective I would like to see a) and b), maybe c). But I
would like to use this feature in otptests which should land in 4.1 ->
a) is not an option because of its complexity. Therefore, atm, I would
like to go with e) or c)
Thanks
--
Petr Vobornik
From 3c1a140b80b87693a6deb76dc7ebeb4ef479c560 Mon Sep 17 00:00:00 2001
From: Petr Vobornik <[email protected]>
Date: Fri, 12 Dec 2014 16:18:34 +0100
Subject: [PATCH] webui-tests: support direct IPA API calls
---
ipatests/test_webui/ui_driver.py | 122 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 121 insertions(+), 1 deletion(-)
diff --git a/ipatests/test_webui/ui_driver.py b/ipatests/test_webui/ui_driver.py
index bf78fcb5798f5f17f2d76650da0c0327bb4c6d71..bde337c1ee2ac69833263eca988945c27d2d3c4a 100644
--- a/ipatests/test_webui/ui_driver.py
+++ b/ipatests/test_webui/ui_driver.py
@@ -52,8 +52,11 @@ try:
NO_YAML = False
except ImportError:
NO_YAML = True
-from urllib2 import URLError
+from urllib2 import URLError, urlparse
from ipaplatform.paths import paths
+from ipapython import certdb, ipautil
+from ipapython.ipautil import run
+from ipalib import x509, api
ENV_MAP = {
'MASTER': 'ipa_server',
@@ -117,10 +120,30 @@ class UI_driver(object):
if NO_SELENIUM:
raise nose.SkipTest('Selenium not installed')
+ @property
+ def api(self):
+ """
+ ipalib API
+ """
+ if not self._api:
+ self.init_api()
+ return self._api
+
+ @property
+ def nss_db(self):
+ """
+ NSS DB for IPA lib rpc client
+ """
+ if not self._nss_db:
+ self._nss_db = self._prepare_nss_dir(self.config['ipa_server'])
+ return self._nss_db
+
def __init__(self, driver=None, config=None):
self.request_timeout = 30
self.driver = driver
self.config = config
+ self._api = None;
+ self._nss_db = None;
if not config:
self.load_config()
@@ -174,6 +197,103 @@ class UI_driver(object):
Test clean up
"""
self.driver.quit()
+ if self._nss_db:
+ self._nss_db.close()
+ self._restore_env()
+
+ def _prepare_nss_dir(self, ipa_server):
+ """
+ Create temporary NSS Database with IPA server CA cert. CA cert is
+ downloaded over HTTP from IPA server.
+ """
+
+ # create new NSSDatabase
+ tmp_db = certdb.NSSDatabase()
+ pwd_file = ipautil.write_tmp_file(ipautil.ipa_generate_password())
+ tmp_db.create_db(pwd_file.name)
+
+ # download and add cert
+ url = urlparse.urlunparse(('http', ipautil.format_netloc(ipa_server),
+ '/ipa/config/ca.crt', '', '', ''))
+ stdout, stderr, rc = run([paths.BIN_WGET, "-O", "-", url])
+ certs = x509.load_certificate_list(stdout, tmp_db.secdir)
+ ca_certs = [cert.der_data for cert in certs]
+ for i, cert in enumerate(ca_certs):
+ tmp_db.add_cert(cert, 'CA certificate %d' % (i + 1), 'C,,')
+
+ return tmp_db
+
+ def init_api(self):
+ """
+ Initialize ipalib API so we can use API commands in UI tests directly.
+ Uses forms-based authentication. It allows Web UI tests to be run a
+ system which is not an IPA client.
+ """
+
+ self._api = api
+ self._env_backup = {}
+
+ def set_env(key, val):
+ self._env_backup[key] = api.env[key]
+ object.__setattr__(api.env, key, val)
+ api.env._Env__d[key] = val
+
+ ipa_server = self.config['ipa_server']
+ set_env('xmlrpc_uri', "https://"; + ipa_server + "/ipa/xml")
+ set_env('jsonrpc_uri', "https://"; + ipa_server + "/ipa/json")
+ set_env('realm', self.config["ipa_realm"])
+ set_env('domain', self.config["ipa_domain"])
+ set_env('basedn', ipautil.realm_to_suffix(self.config["ipa_realm"]))
+
+ #api.bootstrap(
+ # context='webui_tests',
+ # in_server=False,
+ # debug=False,
+ # verbose=2,
+ # xmlrpc_uri="https://"; + ipa_server + "/ipa/json",
+ # # realm, domain and basedn must be set because ipalib produces
+ # # defaults for options for API calls. Default domain and realm:
+ # # EXAMPLE.COM would be used otherwise. It would lead to command
+ # # failures, such as: "RealmMismatch: The realm for the principal
+ # # does not match the realm for this IPA server"
+ # realm=self.config["ipa_realm"],
+ # domain=self.config["ipa_domain"],
+ # basedn=ipautil.realm_to_suffix(self.config["ipa_realm"]),
+ # )
+ #api.finalize()
+ self.reconnect_api(self.config['ipa_admin'], self.config['ipa_password'])
+
+ def _restore_env(self):
+ """
+ Revert changes in API env
+ """
+ if not self._env_backup or self._api:
+ return
+
+ env = self.api.env
+ def restore(key, val):
+ object.__setattr__(env, key, val)
+ env._Env__d[key] = val
+
+ for k,v in self._env_backup:
+ restore(k, v)
+
+ self._env_backup = None
+
+ def reconnect_api(self, user, password):
+ """
+ Reconnect rpcclient as different user with password
+ """
+ self.disconnect_api()
+ self.api.Backend.rpcclient.connect(
+ nss_dir=self.nss_db.secdir,
+ user=user,
+ password=password
+ )
+
+ def disconnect_api(self):
+ if self._api and self.api.Backend.rpcclient.isconnected():
+ self.api.Backend.rpcclient.disconnect()
def get_driver(self):
"""
--
1.9.3
From 29244f8dce7ba7242848ec79d339e2630e518849 Mon Sep 17 00:00:00 2001
From: Petr Vobornik <[email protected]>
Date: Wed, 10 Dec 2014 18:57:51 +0100
Subject: [PATCH] rpc-client: add forms based auth support
---
ipalib/rpc.py | 311 ++++++++++++++++++++++++++++++++++++++--------------------
1 file changed, 202 insertions(+), 109 deletions(-)
diff --git a/ipalib/rpc.py b/ipalib/rpc.py
index 806f6bb9adf004660c9cb285cf31b09a988afa93..8f9836bca84db969f2e5456ab85bf1d9fda559cc 100644
--- a/ipalib/rpc.py
+++ b/ipalib/rpc.py
@@ -38,9 +38,11 @@ import os
import locale
import base64
import urllib
+import urllib2
import json
import socket
-from urllib2 import urlparse
+from urllib2 import urlparse, HTTPError
+import cookielib
from xmlrpclib import (Binary, Fault, DateTime, dumps, loads, ServerProxy,
Transport, ProtocolError, MININT, MAXINT)
@@ -52,7 +54,8 @@ from nss.error import NSPRError
from ipalib.backend import Connectible
from ipalib.constants import LDAP_GENERALIZED_TIME_FORMAT
from ipalib.errors import (public_errors, UnknownError, NetworkError,
- KerberosError, XMLRPCMarshallError, JSONError, ConversionError)
+ KerberosError, XMLRPCMarshallError, JSONError, ConversionError,
+ InvalidSessionPassword)
from ipalib import errors, capabilities
from ipalib.request import context, Connection
from ipalib.util import get_current_principal
@@ -502,7 +505,75 @@ class SSLTransport(LanguageAwareTransport):
return self._connection[1]
-class KerbTransport(SSLTransport):
+class SessionTransport(SSLTransport):
+
+ def get_host_info(self, host):
+ """
+ Adds session cookie
+ """
+ (host, extra_headers, x509) = LanguageAwareTransport.get_host_info(self, host)
+
+ if not isinstance(extra_headers, list):
+ extra_headers = []
+
+ session_cookie = getattr(context, 'session_cookie', None)
+ root_logger.debug("SessionTransport, using session: " + str(session_cookie))
+ if session_cookie:
+ extra_headers.append(('Cookie', session_cookie))
+ return (host, extra_headers, x509)
+
+ def store_session_cookie(self, cookie_header):
+ '''
+ Given the contents of a Set-Cookie header scan the header and
+ extract each cookie contained within until the session cookie
+ is located. Examine the session cookie if the domain and path
+ are specified, if not update the cookie with those values from
+ the request URL. Then write the session cookie into the key
+ store for the principal. If the cookie header is None or the
+ session cookie is not present in the header no action is
+ taken.
+
+ Context Dependencies:
+
+ The per thread context is expected to contain:
+ principal
+ The current pricipal the HTTP request was issued for.
+ request_url
+ The URL of the HTTP request.
+
+ '''
+
+ if cookie_header is None:
+ return
+
+ principal = getattr(context, 'principal', None)
+ request_url = getattr(context, 'request_url', None)
+ root_logger.debug("received Set-Cookie '%s'", cookie_header)
+
+ # Search for the session cookie
+ try:
+ session_cookie = Cookie.get_named_cookie_from_string(cookie_header,
+ COOKIE_NAME, request_url)
+ except Exception, e:
+ root_logger.error("unable to parse cookie header '%s': %s", cookie_header, e)
+ return
+
+ if session_cookie is None:
+ return
+
+ cookie_string = str(session_cookie)
+ root_logger.debug("storing cookie '%s' for principal %s", cookie_string, principal)
+ try:
+ update_persistent_client_session_data(principal, cookie_string)
+ except Exception, e:
+ # Not fatal, we just can't use the session cookie we were sent.
+ pass
+
+ def parse_response(self, response):
+ self.store_session_cookie(response.getheader('Set-Cookie'))
+ return SSLTransport.parse_response(self, response)
+
+class KerbTransport(SessionTransport):
"""
Handles Kerberos Negotiation authentication to an XML-RPC server.
"""
@@ -530,14 +601,10 @@ class KerbTransport(SSLTransport):
Two things can happen here. If we have a session we will add
a cookie for that. If not we will set an Authorization header.
"""
- (host, extra_headers, x509) = SSLTransport.get_host_info(self, host)
-
- if not isinstance(extra_headers, list):
- extra_headers = []
+ (host, extra_headers, x509) = SessionTransport.get_host_info(self, host)
session_cookie = getattr(context, 'session_cookie', None)
if session_cookie:
- extra_headers.append(('Cookie', session_cookie))
return (host, extra_headers, x509)
# Set the remote host principal
@@ -566,61 +633,10 @@ class KerbTransport(SSLTransport):
def single_request(self, host, handler, request_body, verbose=0):
try:
- return SSLTransport.single_request(self, host, handler, request_body, verbose)
+ return SessionTransport.single_request(self, host, handler, request_body, verbose)
finally:
self.close()
- def store_session_cookie(self, cookie_header):
- '''
- Given the contents of a Set-Cookie header scan the header and
- extract each cookie contained within until the session cookie
- is located. Examine the session cookie if the domain and path
- are specified, if not update the cookie with those values from
- the request URL. Then write the session cookie into the key
- store for the principal. If the cookie header is None or the
- session cookie is not present in the header no action is
- taken.
-
- Context Dependencies:
-
- The per thread context is expected to contain:
- principal
- The current pricipal the HTTP request was issued for.
- request_url
- The URL of the HTTP request.
-
- '''
-
- if cookie_header is None:
- return
-
- principal = getattr(context, 'principal', None)
- request_url = getattr(context, 'request_url', None)
- root_logger.debug("received Set-Cookie '%s'", cookie_header)
-
- # Search for the session cookie
- try:
- session_cookie = Cookie.get_named_cookie_from_string(cookie_header,
- COOKIE_NAME, request_url)
- except Exception, e:
- root_logger.error("unable to parse cookie header '%s': %s", cookie_header, e)
- return
-
- if session_cookie is None:
- return
-
- cookie_string = str(session_cookie)
- root_logger.debug("storing cookie '%s' for principal %s", cookie_string, principal)
- try:
- update_persistent_client_session_data(principal, cookie_string)
- except Exception, e:
- # Not fatal, we just can't use the session cookie we were sent.
- pass
-
- def parse_response(self, response):
- self.store_session_cookie(response.getheader('Set-Cookie'))
- return SSLTransport.parse_response(self, response)
-
class DelegatedKerbTransport(KerbTransport):
"""
@@ -758,24 +774,47 @@ class RPCClient(Connectible):
setattr(context, 'session_cookie', session_cookie.http_cookie())
# Form the session URL by substituting the session path into the original URL
- scheme, netloc, path, params, query, fragment = urlparse.urlparse(original_url)
+ return self.get_session_url(original_url)
+
+ def get_session_url(self, url):
+ scheme, netloc, path, params, query, fragment = urlparse.urlparse(url)
path = self.session_path
session_url = urlparse.urlunparse((scheme, netloc, path, params, query, fragment))
-
return session_url
+
+ def destroy_session(self, principal):
+ if hasattr(context, 'session_cookie'):
+ delattr(context, 'session_cookie')
+ try:
+ delete_persistent_client_session_data(principal)
+ except Exception, e:
+ # This shouldn't happen if we have a session but it isn't fatal.
+ pass
+
def create_connection(self, ccache=None, verbose=0, fallback=True,
- delegate=False, nss_dir=None):
- try:
- rpc_uri = self.env[self.env_rpc_uri_key]
+ delegate=False, nss_dir=None, user=None, password=None):
+ """
+ Create connection
+ """
+
+ setattr(context, 'user', user)
+ setattr(context, 'password', password)
+ use_krb = not user or not password
+
+ rpc_uri = self.env[self.env_rpc_uri_key]
+ if use_krb:
principal = get_current_principal()
- setattr(context, 'principal', principal)
+ else:
+ principal = user
+ setattr(context, 'principal', principal)
+ try:
# We have a session cookie, try using the session URI to see if it
# is still valid
if not delegate:
rpc_uri = self.apply_session_cookie(rpc_uri)
except ValueError:
- # No session key, do full Kerberos auth
+ # No session key, do full auth
pass
# This might be dangerous. Use at your own risk!
if nss_dir:
@@ -786,7 +825,9 @@ class RPCClient(Connectible):
kw = dict(allow_none=True, encoding='UTF-8')
kw['verbose'] = verbose
if url.startswith('https://'):
- if delegate:
+ if not use_krb:
+ transport_class = SessionTransport
+ elif delegate:
transport_class = DelegatedKerbTransport
else:
transport_class = KerbTransport
@@ -794,46 +835,39 @@ class RPCClient(Connectible):
transport_class = LanguageAwareTransport
kw['transport'] = transport_class(protocol=self.protocol)
self.log.info('trying %s' % url)
- setattr(context, 'request_url', url)
- serverproxy = self.server_proxy_class(url, **kw)
- if len(urls) == 1:
- # if we have only 1 server and then let the
- # main requester handle any errors. This also means it
- # must handle a 401 but we save a ping.
- return serverproxy
try:
- command = getattr(serverproxy, 'ping')
- try:
- response = command([], {})
- except Fault, e:
- e = decode_fault(e)
- if e.faultCode in errors_by_code:
- error = errors_by_code[e.faultCode]
- raise error(message=e.faultString)
- else:
- raise UnknownError(
- code=e.faultCode,
- error=e.faultString,
- server=url,
- )
- # We don't care about the response, just that we got one
+ if use_krb:
+ setattr(context, 'request_url', url)
+ serverproxy = self.server_proxy_class(url, **kw)
+ if len(urls) == 1:
+ # if we have only 1 server and then let the
+ # main requester handle any errors. This also means it
+ # must handle a 401 but we save a ping.
+ return serverproxy
+ self.krb_auth(serverproxy, url)
+ else:
+ self.forms_auth(url, principal, user, password)
+ # further forms-based communication requires session url
+ if self.session_path not in url:
+ url = self.get_session_url(url)
+ setattr(context, 'request_url', url)
+ serverproxy = self.server_proxy_class(url, **kw)
break
- except KerberosError, krberr:
- # kerberos error on one server is likely on all
- raise errors.KerberosError(major=str(krberr), minor='')
+
except ProtocolError, e:
if hasattr(context, 'session_cookie') and e.errcode == 401:
- # Unauthorized. Remove the session and try again.
- delattr(context, 'session_cookie')
- try:
- delete_persistent_client_session_data(principal)
- except Exception, e:
- # This shouldn't happen if we have a session but it isn't fatal.
- pass
- return self.create_connection(ccache, verbose, fallback, delegate)
+ self.destroy_session(principal)
+ return self.create_connection(ccache, verbose, fallback, delegate, nss_dir, user, password)
if not fallback:
raise
serverproxy = None
+ except HTTPError, e:
+ if e.code == 401:
+ self.destroy_session(principal)
+ if not fallback:
+ raise
+ self.log.info('Connection to %s failed with %s', url, e)
+ serverproxy = None
except Exception, e:
if not fallback:
raise
@@ -846,6 +880,63 @@ class RPCClient(Connectible):
error=', '.join(urls))
return serverproxy
+ def krb_auth(self, serverproxy, url):
+ try:
+ command = getattr(serverproxy, 'ping')
+ try:
+ response = command([], {})
+ except Fault, e:
+ e = decode_fault(e)
+ if e.faultCode in errors_by_code:
+ error = errors_by_code[e.faultCode]
+ raise error(message=e.faultString)
+ else:
+ raise UnknownError(
+ code=e.faultCode,
+ error=e.faultString,
+ server=url,
+ )
+ except KerberosError, krberr:
+ # kerberos error on one server is likely on all
+ raise errors.KerberosError(major=str(krberr), minor='')
+
+ def forms_auth(self, rpc_url, principal, user, password):
+ """
+ Try forms-based authentication.
+ """
+ (scheme, netloc, path, params, query, fragment
+ ) = urlparse.urlparse(rpc_url)
+ login_url = urlparse.urlunparse((scheme, netloc, 'ipa/session/login_password', '', '', ''))
+ self.log.debug('Forms-based authentication for: %s' % rpc_url)
+ self.log.debug('User: %s' % user)
+
+ headers = {
+ 'Content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
+ 'Referer': 'https://%s/ipa/xml' % str(netloc),
+ }
+ data = {
+ 'user': user,
+ 'password': password,
+ }
+ data = urllib.urlencode(data)
+ handlers = [urllib2.HTTPSHandler()]
+ opener = urllib2.build_opener(*handlers)
+ req = urllib2.Request(login_url, data, headers)
+ response = urllib2.urlopen(req)
+ if response.getcode() == 200:
+ self.log.debug("Forms based auth successfull")
+ session_cookie = Cookie.get_named_cookie_from_string(
+ response.info().getheader('Set-Cookie'),
+ COOKIE_NAME,
+ rpc_url
+ )
+ cookie_string = str(session_cookie)
+ update_persistent_client_session_data(principal, cookie_string)
+ ipa_cookie = session_cookie.http_cookie()
+ setattr(context, 'session_cookie', ipa_cookie)
+ self.log.debug("forms_auth session:" + ipa_cookie)
+
+
def destroy_connection(self):
if sys.version_info >= (2, 7):
conn = getattr(context, self.id, None)
@@ -902,19 +993,21 @@ class RPCClient(Connectible):
session_cookie = getattr(context, 'session_cookie', None)
if session_cookie and e.errcode == 401:
# Unauthorized. Remove the session and try again.
- delattr(context, 'session_cookie')
- try:
- principal = getattr(context, 'principal', None)
- delete_persistent_client_session_data(principal)
- except Exception, e:
- # This shouldn't happen if we have a session but it isn't fatal.
- pass
+ self.destroy_session(getattr(context, 'principal', None))
# Create a new serverproxy with the non-session URI. If there
# is an existing connection we need to save the NSS dbdir so
# we can skip an unnecessary NSS_Initialize() and avoid
# NSS_Shutdown issues.
- serverproxy = self.create_connection(os.environ.get('KRB5CCNAME'), self.env.verbose, self.env.fallback, self.env.delegate)
+ serverproxy = self.create_connection(
+ os.environ.get('KRB5CCNAME'),
+ self.env.verbose,
+ self.env.fallback,
+ self.env.delegate,
+ context.nss_dir,
+ getattr(context, 'user'),
+ getattr(context, 'password')
+ )
dbdir = None
current_conn = getattr(context, self.id, None)
--
1.9.3
_______________________________________________
Freeipa-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/freeipa-devel
