Hi Sengork (et. al):
It's been longer than I anticipated in getting back to this issue
and thread (... apologies for going dark while I tended to other
matters).
Continuing where we left off, thank you for the below feedback.
In troubleshooting, I believe the issue lies in the below code
within "vcloud.py". vcloud.py is the same for python2 & python3,
but the below code (and perhaps other "shared code" too)
behaves slightly differently between interpreters:
=================================
vcloud.py code snip ...
=================================
def _get_auth_headers(self):
"""Some providers need different headers than others"""
return {
'Authorization':
"Basic %s"
% base64.b64encode(b('%s:%s' % (self.user_id, self.key))),
'Content-Length': 0,
'Accept': 'application/*+xml'
}
=================================
Specifically, the "base64.b64encode ..." line returns the following
for each interpreter... Notice the byte-string for the python-3 case:
PYTHON 2 (Works):
curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 31763344' -H
'Accept: application/*+xml'
-H 'Authorization: Basic xxxxxyyyyyzzzzzz'
--compress https://services.vcloudexpress.terremark.com:443/api/v0.8/login#
python 3 (Fails):
curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 31862928' -H
'Accept: application/*+xml'
-H 'Authorization: Basic b'"'"'xxxxxyyyyyyyzzzzzz'"'"''
--compress https://services.vcloudexpress.terremark.com:443/api/v0.8/login
When I temporarily substituted my auth string in the above code directly,
it worked for python 3 too.
=========================
Three things to mention:
=========================
(1) I was surprised that that base64 line worked at all (even in Python 2)
since I always thought use of "b" only worked for string literals
(not generated strings).
(2) Not an issue for this specific thread, but even when I hard-coded
my auth string (just to get past the Python-3 related auth issue), sadly
another unrelated (xml parsing) issue arose almost immediately.
Just as a heads up, it was this:
Traceback (most recent call last):
File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1668, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 1
(So I never got my list_images() output.)
(3) Similar to the last point, python-2 gets much further along, but also
eventually experiences an parse-related exception too (as it's
iterating through the list the of images too I believe). Again, as a
heads-up fyi:
File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76, in
__init__
raise Exception(self.parse_error())
Exception: <Element '{http://www.w3.org/1999/xhtml}html' at 0x118dd10>
Thanks,
Noel
Hi Noel, That narrows down the issue to obtaining an authentication token (via
_get_auth_token) which is the very first thing libcloud vCloud driver
needs to do before it can issue subsequent HTTP calls. Note this is
the only time when your base64 encoded "user:pass" will be sent in a
HTTP request. What is failing is the fact API web service is not
accepting your credentials and replying with an authentication token
to be used in subsequent HTTP requests. Couple of things to consider:
- Is the API username equal to the full email address?
- Try reproducing the failure using Python 2.6/2.7
- Verify your base64 encoded string in curl request body matches your
"username:password"
import base64
base64.b64decode("auth string")
- Comment out this section of _get_auth_headers from vcloud.py "
'Accept': 'application/*+xml' " and retry authenticating
- HTTP call is doing the right thing according to Terremark's v0.8 API
doc (a POST to the right URL with right base64 encoding), however I
have noticed Terremark's developer forum has a few older threads
describing similar API auth issues which were fixed by support staff.
Might be worth double checking there is no active issue at the API
end. There was no Providers.VCLOUD until we've introduced vCloud v1.5
support (which is only in trunk right now). Looks like you should not need
VERIFY_SSL_CERT = False as Terremark
seem to have a signed SSL cert @
https://services.vcloudexpress.terremark.com For reference here's the full
request/response body I receive for a
dummy "user:pass" see how it compares: # -------- begin 47775432 request
----------
curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 47775432'
-H 'Authorization: Basic dXNlcjpwYXNz'
https://services.vcloudexpress.terremark.com:443/api/v0.8/login# -------- begin
47775432:47788872 response ----------
HTTP/1.1 401 Unauthorized
Content-Length: 1293
X-Powered-By: ASP.NET Server: Microsoft-IIS/7.0
Date: Sat, 12 May 2012 02:26:10 GMT
Content-Type: text/html
Www-Authenticate: Basic Realm="vCloud Api" <!DOCTYPE html PUBLIC "-//W3C//DTD
XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";;>
<html xmlns="http://www.w3.org/1999/xhtml";;>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>401 - Unauthorized: Access is denied due to invalid credentials.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica,
sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px
2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>401 - Unauthorized: Access is denied due to invalid credentials.</h2>
<h3>You do not have permission to view this directory or page using
the credentials that you supplied.</h3>
</fieldset></div>
</div>
</body>
</html> On 12 May 2012 05:03, Noel Milton Vega <nmv...@computingarchitects.com>
wrote:
>
> PS... Ignore my comment about "conn.list_sizes()" in the below
> code not generating a HTTP call. I looked at the base.py source
> and see that my usage (of the returned class) below is slightly
> off. :)
>
> My original auth problem below still remains. I continue to
> investigate it.
>
> Thanks again,
> -- nmv
>
>
>
>
> >________________________________
> > From: Noel Milton Vega <nmv...@computingarchitects.com>
> >To: "users@libcloud.apache.org" <users@libcloud.apache.org>
> >Sent: Friday, May 11, 2012 2:25 PM
> >Subject: Re: Terremark authentication example using libcloud ...
> >
> >Hello Sengork:
> >
> >Thank you for the reply. I'll formulate this reply in "PARTS" for
> >clarity.
> >
> >=============
> >
> >PART1:
> >=============
> >
> >Thanks for the gentle reminder to include the libcloud
> >version, which I should have originally stated is
> >version "0.9.1" for python3. Obtained via: >>> libcloud.__version__
> >
> >
> >=============
> >
> >PART2:
> >=============
> >
> >Per your example below, I tried running it again, this time by
> >inserting the following prior to the get_driver() call:
> >
> >
> > libcloud.security.VERIFY_SSL_CERT = False
> >
> >Sadly, it did not resolve the issue. Here again, for referential
> >convenience, is the cleaned up code (including my
> >in-line comments where the exception is being raised):
> >
> >> ###############################################
> >> #! /usr/bin/env python3
> >>
> >> import libcloud.compute.types
> >> import libcloud.compute.providers
> >> import libcloud.security
> >> #
> >> TERREMARK_USERNAME = "some@email.address"
> >> TERREMARK_PASSWORD = "some-password"
> >> #
> >> libcloud.security.VERIFY_SSL_CERT = False
> >> # Including this or not doesn't change the exception outcome.
> >
> >> Driver =
> >> libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.TERREMARK)
> >> conn = Driver(TERREMARK_USERNAME, TERREMARK_PASSWORD)
> >>
> >
> >> for aSize in conn.list_sizes(): print(aSize) # No HTTP request generated
> >> here.
Why? Hmmm.
> >
> >> # Oddly, this call to list_sizes() doesn't seem to generate a HTTP
request (i.e. no curl(1)
> >> # output for this). However it does return a Python list with
> >>seemingly
fake/dummy VM
> >> # sizes generated internally by libcloud.
> >
> >>
> >> for anImage in conn.list_images(): print(anImage) # HTTP request is
> >> generated
here, but so is the exception.
> >> # On the other hand, this call to list_images() does generate a HTTP
request (and curl(1) output),
> >> # and causes the exception above to be raised. See below (in this
email) for a cut/paste of
> >> # curl(1) output as well as related python exception.
> >
> >>
> >> [ ... snip ... ]
> >> #########################################
> >The exception, with or without the VERIFY_SSL_CERT flag, is the same and
> >appears as follows:
> >
> >-------------------
> >IN CURL(1) LOG:
> >
> >-------------------
> >
> > # -------- begin 27800016 request ----------
> > curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 27800016'
> > -H 'Authorization: Basic b'"'"'a-60-character-scring'"'"''
> > --compress
> >https://services.vcloudexpress.terremark.com:443/api/v0.8/login
> > # -------- begin 27800016:27800208 response ----------
> > HTTP/1.1 401 Unauthorized
> > Content-Type: text/html
> > Server: Microsoft-IIS/7.0
> > Www-Authenticate: Basic Realm="vCloud Api"
> > X-Powered-By: ASP.NET
> > Date: Fri, 11 May 2012 17:31:52 GMT
> > Content-Length: 1293
> > [ ... snip ... ]
> >
> >
> >--------------------
> >
> >IN Python3 IDE:
> >--------------------
> >
> > xml.etree.ElementTree.ParseError: not well-formed (invalid token): line
> >1, column
1
> > File
> >"/home/nmvega/Dropbox/CodeDEV.d/PYTHON.d/Libcloud-Client-Development.d/terremark.py",
line 28, in <module>
> > for anImage in conn.list_images(): print(anImage)
> > File
> >"/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 474, in list_images
> > for vdc in self.vdcs:
> > File
> >"/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 281, in vdcs
> > self.connection.check_org() # make sure the org is set.
> > File
> >"/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 226, in check_org
> > self._get_auth_token()
> > File
> >"/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 246, in _get_auth_token
> > body = ET.XML(resp.read())
> > File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1329, in XML
> > parser.feed(text)
> > File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1662, in feed
> >
> >
> >
> >
> >=================
> >
> >PART3:
> >=================
> >Finally, just a note that in v0.9.1 of libcloud, the source file
> >/usr/lib/python3.2/site-packages/libcloud/compute/providers.py
> >doesn't have a dict() entry for Providers.VCLOUD, so I couldn't try
> >your included example (... Just for grins -- it would not have worked
> >for TERREMARK anyay, since they use a modified version of the
> >vCloud API). Just a side note that Providers.VCLOUD was not there.
> >
> >
> >Thank you in advance again Sengork, et. al.
> >
> >
> >-- nmv
> >
> >
> >
> >
> >>________________________________
> >> From: Sengor <seng...@gmail.com>
> >>To: users@libcloud.apache.org; Noel Milton Vega
> >><nmv...@computingarchitects.com>
> >>Sent: Thursday, May 10, 2012 11:37 PM
> >>Subject: Re: Terremark authentication example using libcloud ...
> >>
> >>Hi,
> >>
> >>I've not used the Terremark variant of vCloud driver, but this is what
> >>should work:
> >>conn = Driver(TERREMARK_USERNAME, TERREMARK_PASSWORD)
> >>
> >>The driver will perform base64 encoding and joining "user:name" on your
> >>behalf.
> >>
> >>As an example this is what's done for vCloud v1.5:
> >>
> >>from libcloud.compute.types import Provider
> >>from libcloud.compute.providers import get_driver
> >># Do this only if API has self signed SSL certificate
> >>import libcloud.security
> >>libcloud.security.VERIFY_SSL_CERT = False
> >>Driver = get_driver(Provider.VCLOUD)
> >>conn = Driver("user@organisation", "password", host="hostname.com",
> >>api_version="1.5")
> >>conn.list_images()
> >>
> >>You should not need api_version and host for v0.8.
> >>
> >>I suggest that you enable libcloud debugging and observe the raw API
> >>request/responses for any hints as to the cause of the issue.
> >>
> >>$ export LIBCLOUD_DEBUG=/tmp/raw
> >>$ tail -f /tmp/raw
> >>$ python
> >>.....
> >>
> >>Please also let us know which version of libcloud are you using?
> >>
> >>
> >>
> >>
> >>On 11 May 2012 06:00, Noel Milton Vega
> >><nmv...@computingarchitects.com>wrote:
> >>
> >>> Hello friends:
> >>>
> >>> I was wondering if anyone had a Terremark libcloud authentication
> >>> example, similar to the ones indicated here for Amazon EC2 and Rackspace:
> >>>
> >>> http://libcloud.apache.org/getting-started.html
> >>>
> >>> I've tried a few variations of the examples provided in the above link for
> >>> the Terremark case (a vCloud API implementation), but received an "Invalid
> >>> Credentials with the provider" Exception in each case.
> >>>
> >>> So far I've tried passing Authentication using the three (3) variations
> >>> shown below (commented out) for "conn = Driver()".
> >>>
> >>> Thank you in advance!
> >>>
> >>> ###############################################
> >>> #! /usr/bin/env python3
> >>>
> >>> import libcloud.compute.types
> >>> import libcloud.compute.providers
> >>> import base64
> >>> #
> >>> TERREMARK_USERNAME = "some@email.address"
> >>> TERREMARK_PASSWORD = "some-password"
> >>> TERREMARK_B64AUTH = base64.b64encode(bytes(TERREMARK_USERNAME + ":" +
> >>> TERREMARK_PASSWORD, encoding='utf-8'))
> >>> #
> >>> Driver =
> >>> libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.TERREMARK)
> >>>
> >>> #conn = Driver(TERREMARK_USERNAME, TERREMARK_PASSWORD)
> >>> #conn = Driver(TERREMARK_B64AUTH)
> >>> #conn = Driver(TERREMARK_USERNAME + ":" + TERREMARK_PASSWORD)
> >>>
> >>> [ ... snip ... ]
> >>> #########################################
> >>>
> >>
> >>
> >>
> >>--
> >>sengork
> >>
> >>
> >>
> >
> > --
sengork
