Re: [Openstack] Borrow a Swift Implementation
Hey Kyle You can e-mail me directly and I'll set up an account for you. Mvh / Best regards Morten Møller Riis Gigahost ApS m...@gigahost.dk On Jul 29, 2013, at 11:20 AM, Kyle Kelley rgb...@gmail.com wrote: Hey all, Does anyone have an implementation of OpenStack Swift I could use to test my IPython notebook storage manager? It's been tested on Rackspace's CloudFiles, but I want to make sure it works for vanilla OpenStack. I'm using pyrax underneath the hood, but willing to change things to make it more generic. Cheers, Kyle Kelley ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improved browser-based access to Swift
Nice. Just looked through the code. Would like to add this to our solution but it seems to be hardcoded to keystone (we use swauth). I might look into getting it working with swauth though - would be a nice addition. Thanks for sharing! Mvh / Best regards Morten Møller Riis Gigahost ApS m...@gigahost.dk On Jul 18, 2013, at 10:20 PM, Koert van der Veer ko...@cloudvps.com wrote: We've been offering public object store services for roughly half a year now. In the past few months, we received a wide range of responses from our customers. Tech-savvy customers are very happy with this offering, and quite a few are busy migrating their existing storage solutions to our object store. However, we find that several customers, especially those coming from shared hosting solutions, struggle to grasp the abstract ideas behind object storage. They struggle with two things: authentication against keystone, which they consider too complicated, and having a user interface that is totally distinct from the actual object store. To help these users, we've developed two middleware projects: swift_basicauth and better_staticweb. swift_basicauth allows web browsers and general purpose HTTP clients to access object stores, without having to contact keystone. The middleware interprets the authentication, and then fetches a token based on that authentication. With that token, the rest of the request is processed. This enables a wide range of HTTP clients to access the object store. While the primary aim was unlocking the objectstore for web-browsers, we quickly discovered the convenience of using it with curl: curl --user uid:pwd -X PUT https://static.example.net/the_file; -T the_file. The second middleware we developed was named better_staticweb (sorry for the pretentious name). It is similar to static-web, in fact is is mostly compatible. However, web-listings is enabled by default, even for authenticated access (useful in combination with basic auth). It enables the user to visualize his object store as a less-abstract concept. Better_staticweb looks at the http Accept header to determine whether or not to respond with a listing. It still listens to the same meta-headers, but it assumes different defaults. We've gone through quite a bit of testing to guarantee that it doesn't interfere with regular API usage. Both middleware projects are released under the Apache 2.0 licence, and can be found on our github page: https://github.com/CloudVPS/better-staticweb https://github.com/CloudVPS/swift-basicauth -- Koert van der Veer - Senior Developer @ CloudVPS CloudVPS - High Availability Cloud Solutions w: http://www.cloudvps.com/ m: ko...@cloudvps.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Swift tempurl
I'm having trouble getting tempurl to work. I set the X-Account-Meta-Temp-Url-Key metadata on the account. And a GET request shows it's set correctly. I've enabled it on the proxy server in /etc/swift/proxy-server.conf: [DEFAULT] bind_port = 8080 workers = 8 user = swift [pipeline:main] pipeline = healthcheck cache tempurl swauth proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true [filter:tempurl] use = egg:swift#tempurl [filter:swauth] use = egg:swauth#swauth set log_name = swauth super_admin_key = removed default_swift_cluster = gigahost#https://url/v1#http://localhost:8080/v1 [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache memcache_servers = 127.0.0.1:11211 When creating the tempurl's, even using the swift-temp-url program I get: ~ $ curl -i https://url/v1/AUTH_224b1001-2c75-444c-aaef-30af13b9154c/000/206.pdf?temp_url_sig=387d79120a591e1cf6f4d4356f5c0a96fb49d202temp_url_expires=1373438360 HTTP/1.1 401 Unauthorized Server: nginx Date: Wed, 10 Jul 2013 08:21:32 GMT Content-Type: text/plain Content-Length: 35 Connection: keep-alive 401 Unauthorized: Temp URL invalid I've even tried running it locally on the proxy-server to localhost:8080, so I'm sure it's not nginx doing any funny stuff. Does anybody have any suggestions? I'm pretty lost :( Mvh / Best regards Morten Møller Riis Gigahost ApS m...@gigahost.dk ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Swift tempurl
I use swauth, and thus authtoken is not present in my pipeline. This might be
why I'm seeing the problem actually.
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
On Jul 11, 2013, at 4:04 AM, Shrinand Javadekar shrin...@maginatics.com wrote:
I had a similar problem. See [1]. The fix was to add delay_auth_decision = 1
in the proxy-server.conf
[filter:authtoken]
...
delay_auth_decision = 1
-Shri
[1] https://answers.launchpad.net/swift/+question/225614
On Wed, Jul 10, 2013 at 4:43 AM, Morten Møller Riis m...@gigahost.dk wrote:
I've been looking at tempurl.py and found that the problem occurs here:
def _get_key(self, env, account):
Returns the X-Account-Meta-Temp-URL-Key header value for the
account, or None if none is set.
:param env: The WSGI environment for the request.
:param account: Account str.
:returns: X-Account-Meta-Temp-URL-Key str value, or None.
key = None
memcache = env.get('swift.cache')
if memcache:
key = memcache.get('temp-url-key/%s' % account)
if not key:
newenv = make_pre_authed_env(env, 'HEAD', '/v1/' + account,
self.agent)
newenv['CONTENT_LENGTH'] = '0'
newenv['wsgi.input'] = StringIO('')
key = [None]
def _start_response(status, response_headers, exc_info=None):
for h, v in response_headers:
if h.lower() == 'x-account-meta-temp-url-key':
key[0] = v
i = iter(self.app(newenv, _start_response))
self.logger.info()
try:
i.next()
except StopIteration:
pass
key = key[0]
if key and memcache:
memcache.set('temp-url-key/%s' % account, key, timeout=60)
return key
The request get 403 forbidden and thus never gets the key in the first place.
I'm looking at the github repo and the implementation there seems to have
changed.
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
On Jul 10, 2013, at 6:24 PM, Morten Møller Riis m...@gigahost.dk wrote:
I'm having trouble getting tempurl to work.
I set the X-Account-Meta-Temp-Url-Key metadata on the account. And a GET
request shows it's set correctly.
I've enabled it on the proxy server in /etc/swift/proxy-server.conf:
[DEFAULT]
bind_port = 8080
workers = 8
user = swift
[pipeline:main]
pipeline = healthcheck cache tempurl swauth proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:tempurl]
use = egg:swift#tempurl
[filter:swauth]
use = egg:swauth#swauth
set log_name = swauth
super_admin_key = removed
default_swift_cluster = gigahost#https://url/v1#http://localhost:8080/v1
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:11211
When creating the tempurl's, even using the swift-temp-url program I get:
~ $ curl -i
https://url/v1/AUTH_224b1001-2c75-444c-aaef-30af13b9154c/000/206.pdf?temp_url_sig=387d79120a591e1cf6f4d4356f5c0a96fb49d202temp_url_expires=1373438360
HTTP/1.1 401 Unauthorized
Server: nginx
Date: Wed, 10 Jul 2013 08:21:32 GMT
Content-Type: text/plain
Content-Length: 35
Connection: keep-alive
401 Unauthorized: Temp URL invalid
I've even tried running it locally on the proxy-server to localhost:8080, so
I'm sure it's not nginx doing any funny stuff.
Does anybody have any suggestions? I'm pretty lost :(
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Swift tempurl
This turned out to be a known issue with swath.
I upgraded to a newer version and the problem was resolved :)
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
On Jul 11, 2013, at 10:52 AM, Morten Møller Riis m...@gigahost.dk wrote:
I use swauth, and thus authtoken is not present in my pipeline. This might be
why I'm seeing the problem actually.
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
On Jul 11, 2013, at 4:04 AM, Shrinand Javadekar shrin...@maginatics.com
wrote:
I had a similar problem. See [1]. The fix was to add delay_auth_decision = 1
in the proxy-server.conf
[filter:authtoken]
...
delay_auth_decision = 1
-Shri
[1] https://answers.launchpad.net/swift/+question/225614
On Wed, Jul 10, 2013 at 4:43 AM, Morten Møller Riis m...@gigahost.dk wrote:
I've been looking at tempurl.py and found that the problem occurs here:
def _get_key(self, env, account):
Returns the X-Account-Meta-Temp-URL-Key header value for the
account, or None if none is set.
:param env: The WSGI environment for the request.
:param account: Account str.
:returns: X-Account-Meta-Temp-URL-Key str value, or None.
key = None
memcache = env.get('swift.cache')
if memcache:
key = memcache.get('temp-url-key/%s' % account)
if not key:
newenv = make_pre_authed_env(env, 'HEAD', '/v1/' + account,
self.agent)
newenv['CONTENT_LENGTH'] = '0'
newenv['wsgi.input'] = StringIO('')
key = [None]
def _start_response(status, response_headers, exc_info=None):
for h, v in response_headers:
if h.lower() == 'x-account-meta-temp-url-key':
key[0] = v
i = iter(self.app(newenv, _start_response))
self.logger.info()
try:
i.next()
except StopIteration:
pass
key = key[0]
if key and memcache:
memcache.set('temp-url-key/%s' % account, key, timeout=60)
return key
The request get 403 forbidden and thus never gets the key in the first
place. I'm looking at the github repo and the implementation there seems to
have changed.
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
On Jul 10, 2013, at 6:24 PM, Morten Møller Riis m...@gigahost.dk wrote:
I'm having trouble getting tempurl to work.
I set the X-Account-Meta-Temp-Url-Key metadata on the account. And a GET
request shows it's set correctly.
I've enabled it on the proxy server in /etc/swift/proxy-server.conf:
[DEFAULT]
bind_port = 8080
workers = 8
user = swift
[pipeline:main]
pipeline = healthcheck cache tempurl swauth proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:tempurl]
use = egg:swift#tempurl
[filter:swauth]
use = egg:swauth#swauth
set log_name = swauth
super_admin_key = removed
default_swift_cluster = gigahost#https://url/v1#http://localhost:8080/v1
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:11211
When creating the tempurl's, even using the swift-temp-url program I get:
~ $ curl -i
https://url/v1/AUTH_224b1001-2c75-444c-aaef-30af13b9154c/000/206.pdf?temp_url_sig=387d79120a591e1cf6f4d4356f5c0a96fb49d202temp_url_expires=1373438360
HTTP/1.1 401 Unauthorized
Server: nginx
Date: Wed, 10 Jul 2013 08:21:32 GMT
Content-Type: text/plain
Content-Length: 35
Connection: keep-alive
401 Unauthorized: Temp URL invalid
I've even tried running it locally on the proxy-server to localhost:8080,
so I'm sure it's not nginx doing any funny stuff.
Does anybody have any suggestions? I'm pretty lost :(
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
m...@gigahost.dk
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] Metadata in listing
Hi Guys I was wondering if there is any possibility of getting metadata output in the listing when you issue a GET on a container. At the moment it returns eg.: object name10620_1b8b2553c6eb9987ff647d69e3181f9eeb3a43ef.jpg/name hashe453fcd7ff03e9e0e460555e875b1da1/hash bytes9272/bytes content_typeimage/jpeg/content_type last_modified2012-09-20T23:27:34.473230/last_modified /object If I have X-Object-Meta-Something on an object it would be nice to see it here as well. I know I can get it by doing a HEAD request. But this gets heavy for many objects. Any suggestions? Best regards Morten Møller Riis Med venlig hilsen / Best regards Morten Møller Riis m...@gigahost.dk Gigahost Gammeltorv 8, 2. 1457 København K ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Metadata in listing
Thanks Chuck. What I am playing at here is I want to create an rsync like script where I can save permissions, owner uid/gid and mode (maybe even xattrs), so that a restore will work with these. Swift makes this very easy with the object metadata - however, running a sync would require i HEAD request for each file. I was looking to optimize this. Med venlig hilsen / Best regards Morten Møller Riis m...@gigahost.dk Gigahost Gammeltorv 8, 2. 1457 København K On Dec 13, 2012, at 4:40 PM, Chuck Thier cth...@gmail.com wrote: The metadata for objects is stored at the object level, not in the container dbs. Reporting metadata information for container listings would require the server to HEAD every object in the container, which would cause too much work on the backend. -- Chuck On Wed, Dec 12, 2012 at 7:01 AM, Morten Møller Riis m...@gigahost.dk wrote: Hi Guys I was wondering if there is any possibility of getting metadata output in the listing when you issue a GET on a container. At the moment it returns eg.: object name10620_1b8b2553c6eb9987ff647d69e3181f9eeb3a43ef.jpg/name hashe453fcd7ff03e9e0e460555e875b1da1/hash bytes9272/bytes content_typeimage/jpeg/content_type last_modified2012-09-20T23:27:34.473230/last_modified /object If I have X-Object-Meta-Something on an object it would be nice to see it here as well. I know I can get it by doing a HEAD request. But this gets heavy for many objects. Any suggestions? Best regards Morten Møller Riis Med venlig hilsen / Best regards Morten Møller Riis m...@gigahost.dk Gigahost Gammeltorv 8, 2. 1457 København K ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
