Re: [openstack-dev] [glance] Use Launcher/ProcessLauncher in glance

[stuart . mclaren]

I've been looking at the most recent patch for this 
(https://review.openstack.org/#/c/117988/).

I'm wondering what behaviour do folks feel is best in the following scenario?

1) an image download starts (download1)

2) send SIGHUP to glance-api

3) start another download (download2)

4) download1 completes

5) ?

I think there's a few potential options:

5a) The request for 'download2' is never accepted (in which case
the service is 'offline' on the node until all in-flight requests are 
completed).

5b) The request for 'download2' is killed when 'download1' completes
and the service restarts (not much point in new SIGHUP behaviour)

5c) The request for 'download2' is accepted and completes using the
existing process, but in this case the service potentially never restarts if new
requests keep coming in

5d) A 'swift reload' type operation is done where the old processes
are left alone to complete (download1 and download2 complete) but
new parent (and child) processes are spun up to handle new requests
('download3'). The disadvantage here is some extra process accounting
and potentially stray old code running on your system

(See http://docs.openstack.org/developer/swift/admin_guide.html#swift-orphans)

-Stuart


On Mon, Jul 28, 2014 at 8:12 AM, Tailor, Rajesh rajesh.tai...@nttdata.com
wrote:


Hi All,

I have submitted the patch Made provision for glance service to use
Launcher to the community gerrit.
Pl refer: https://review.openstack.org/#/c/110012/

I have also set the workflow to 'work in progress'. I will start working
on writing unit tests for the proposed
changes, after positive feedback for the same.

Could you please give your comments on this.

Could you also please suggest me whether to file a launchpad bug or a
blueprint,  to propose these changes in the glance project ?



Submitting to github.com/openstack/glance-specs would be best. Thanks.




Thanks,
Rajesh Tailor

-Original Message-
From: Tailor, Rajesh [mailto:rajesh.tai...@nttdata.com]
Sent: Wednesday, July 23, 2014 12:13 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [glance] Use Launcher/ProcessLauncher in
glance

Hi Jay,
Thank you for your response.
I will soon submit patch for the same.

Thanks,
Rajesh Tailor

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Tuesday, July 22, 2014 8:07 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [glance] Use Launcher/ProcessLauncher in
glance

On 07/17/2014 03:07 AM, Tailor, Rajesh wrote:
 Hi all,

 Why glance is not using Launcher/ProcessLauncher (oslo-incubator) for
 its wsgi service like it is used in other openstack projects i.e.
 nova, cinder, keystone etc.

Glance uses the same WSGI service launch code as the other OpenStack
project from which that code was copied: Swift.

 As of now when SIGHUP signal is sent to glance-api parent process, it
 calls the callback handler and then throws OSError.

 The OSError is thrown because os.wait system call was interrupted due
 to SIGHUP callback handler.

 As a result of this parent process closes the server socket.

 All the child processes also gets terminated without completing
 existing api requests because the server socket is already closed and
 the service doesn't restart.

 Ideally when SIGHUP signal is received by the glance-api process, it
 should process all the pending requests and then restart the
 glance-api service.

 If (oslo-incubator) Launcher/ProcessLauncher is used in glance then it
 will handle service restart on 'SIGHUP' signal properly.

 Can anyone please let me know what will be the positive/negative
 impact of using Launcher/ProcessLauncher (oslo-incubator) in glance?

Sounds like you've identified at least one good reason to move to
oslo-incubator's Launcher/ProcessLauncher. Feel free to propose patches
which introduce that change to Glance. :)

 Thank You,

 Rajesh Tailor
 __
 Disclaimer:This email and any attachments are sent in strictest
 confidence for the sole use of the addressee and may contain legally
 privileged, confidential, and proprietary data. If you are not the
 intended recipient, please advise the sender by replying promptly to
 this email and then delete and destroy this email and any attachments
 without any further use, copying or forwarding

Please advise your corporate IT department that the above disclaimer on
your emails is annoying, is entirely disregarded by 99.999% of the real
world, has no legal standing or enforcement, and may be a source of
problems with people's mailing list posts being sent into spam boxes.

All the best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
Disclaimer:This email and any attachments 

[openstack-dev] [glance] 0.14.0 client - v2 requests broken

[stuart . mclaren]

All,

The 0.14.0 client breaks all v2 requests when pointed at a glance server
which is running code which is more than a few days old (ie pretty much
any production server).

There's a band-aid patch here: https://review.openstack.org/#/c/120143/
and a longer term patch here: https://review.openstack.org/#/c/120300/

What would folks propose we do to avoid having a broken client for
an extended period of time (days)?

Would merging 120143 and releasing it as 0.14.1 be a reasonable short term
solution which would allow less pressured reviewing of 120300?

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] 0.14.0 client - v2 requests broken

[stuart . mclaren]

On 09/10/2014 11:52 AM, stuart.mclaren at hp.com wrote:

All,

The 0.14.0 client breaks all v2 requests when pointed at a glance server
which is running code which is more than a few days old (ie pretty much
any production server).

There's a band-aid patch here: https://review.openstack.org/#/c/120143/
and a longer term patch here: https://review.openstack.org/#/c/120300/

What would folks propose we do to avoid having a broken client for
an extended period of time (days)?

Would merging 120143 and releasing it as 0.14.1 be a reasonable short term
solution which would allow less pressured reviewing of 120300?




Why do you think landing 120300 will take long? If both patches are
already on gerrit, I'd rather review the real fix.



Is there a reason you think landing 120143 first is better?


120300 is a biggish change (several hundred lines) and I was concerned
it might take a while to go through the review cycle.  However, the
consensus seems to be to go directly with 120300 -- that it shouldn't
take too long to get through. (And its on-demand schema behaviour is
a nice bonus too!)



Flavio


--
@flaper87
Flavio Percoco


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] PTL Non-Candidacy

[stuart . mclaren]

Hi Mark,

Many thanks for your leadership, and keeping glance so enjoyable to work on 
over the last few cycles.

-Stuart


From: Mark Washenberger mark.washenber...@markwash.net
To: OpenStack Development Mailing List
openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Glance] PTL Non-Candidacy
Message-ID:
CAJyP2C_zR5oK-=dno1e5wv-qsimfqplckzfbl+rhg6-7fhd...@mail.gmail.com
Content-Type: text/plain; charset=utf-8

Greetings,

I will not be running for PTL for Glance for the Kilo release.

I want to thank all of the nice folks I've worked with--especially the
attendees and sponsors of the mid-cycle meetups, which I think were a major
success and one of the highlights of the project for me.

Cheers,
markwash
-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.openstack.org/pipermail/openstack-dev/attachments/20140922/85b17570/attachment-0001.html

--


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Glance] delayed delete and user credentials

[stuart . mclaren]

Hi Pete,

See the comments in patch set 5 for scrubber.py and
also this documentation: https://review.openstack.org/#/c/59471/

-Stuart


Message: 2
Date: Thu, 6 Feb 2014 11:56:28 -0700
From: Pete Zaitcev zait...@redhat.com
To: OpenStack Dev OpenStack-dev@lists.openstack.org
Subject: [openstack-dev] [Glance] delayed delete and user credentials
Message-ID: 20140206115628.345bf...@guren.zaitcev.lan
Content-Type: text/plain; charset=US-ASCII

Hi, guys:

I looked briefly at a bug/fix, which looks exceedingly strange to me:
https://review.openstack.org/59689

As much as I can tell, the problem (lp:1238604) is that pending delete
fails because by the time the delete actually occurs, Glance API does
not have proper permissions to talk to Glance Registry.

So far so good, but the solution that we accepted is to forward
the user credentials to Registry... but only if configured to do so.
Does it make any sense to anyone? Why configure something that must
always work? How can sysadmin select the correct value?

-- Pete




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Glance] FFE Request: Allow enabling both v1 and v2 registries

[stuart . mclaren]

Hi,

Right now you can select running either the v1 or v2 registries but not
both at the same time. I'd like to ask for an FFE for this functionality
(Erno's patch in progress here: https://review.openstack.org/#/c/79957/)

With v1 on the road to deprecation I think this may help migrating from
a v1 to a v2 registry.

Thanks,

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] proxying SSL traffic for API requests

[stuart . mclaren]

All,

I know there's a preference for using a proxy to terminate
SSL connections rather than using the native python code.

There's a good write up of configuring the various proxies here:

http://docs.openstack.org/security-guide/content/ch020_ssl-everywhere.html

If we're not using native python SSL termination in TripleO we'll
need to pick which one of these would be a reasonable choice for
initial https support.

Pound may be a good choice -- its lightweight (6,000 lines of C),
easy to configure and gives good control over the SSL connections (ciphers etc).
Plus, we've experience with pushing large (GB) requests through it.

I'm interested if others have a strong preference for one of the other
options (stud, nginx, apache) and if so, what are the reasons you feel it
would make a better choice for a first implementation.

Thanks,

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] proxying SSL traffic for API requests

[stuart . mclaren]

Just spotted the openstack-ssl element which uses 'stunnel'...


On Wed, 26 Mar 2014, stuart.mcla...@hp.com wrote:


All,

I know there's a preference for using a proxy to terminate
SSL connections rather than using the native python code.

There's a good write up of configuring the various proxies here:

http://docs.openstack.org/security-guide/content/ch020_ssl-everywhere.html

If we're not using native python SSL termination in TripleO we'll
need to pick which one of these would be a reasonable choice for
initial https support.

Pound may be a good choice -- its lightweight (6,000 lines of C),
easy to configure and gives good control over the SSL connections (ciphers 
etc).

Plus, we've experience with pushing large (GB) requests through it.

I'm interested if others have a strong preference for one of the other
options (stud, nginx, apache) and if so, what are the reasons you feel it
would make a better choice for a first implementation.

Thanks,

-Stuart



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] proxying SSL traffic for API requests

[stuart . mclaren]

Thanks Chris.

Sounds like you're saying building out the apache element may be a sensible
next step?

-Stuart


Hi

We don't have a strong attachment to stunnel though, I quickly dropped it in 
front of our CI/CD undercloud and Rob wrote the element so we could repeat the 
deployment.

In the fullness of time I would expect there to exist elements for several SSL 
terminators, but we shouldn't necessarily stick with stunnel because it 
happened to be the one I was most familiar with :)

I would think that an httpd would be a good option to go with as the default, 
because I tend to think that we'll need an httpd running/managing the python 
code by default.

Cheers,
--
Chris Jones


On 26 Mar 2014, at 13:49, stuart.mclaren at hp.com wrote:

Just spotted the openstack-ssl element which uses 'stunnel'...



On Wed, 26 Mar 2014, stuart.mclaren at hp.com wrote:

All,

I know there's a preference for using a proxy to terminate
SSL connections rather than using the native python code.

There's a good write up of configuring the various proxies here:

http://docs.openstack.org/security-guide/content/ch020_ssl-everywhere.html

If we're not using native python SSL termination in TripleO we'll
need to pick which one of these would be a reasonable choice for
initial https support.

Pound may be a good choice -- its lightweight (6,000 lines of C),
easy to configure and gives good control over the SSL connections (ciphers etc).
Plus, we've experience with pushing large (GB) requests through it.

I'm interested if others have a strong preference for one of the other
options (stud, nginx, apache) and if so, what are the reasons you feel it
would make a better choice for a first implementation.

Thanks,

-Stuart


___
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] 'ordereddict' requirement

[stuart . mclaren]

All,

There was a plan to use pypi's 'ordereddict' in Icehouse, to
replace how we're currently providing that functionality.

However, there are no ordereddict packages for Debian/Ubuntu
and there are no plans to provide them. (See Thomas Goirand's comment
here: https://review.openstack.org/#/c/48475/3/requirements.txt)

I think this means that it makes sense to stick with our current solution until
python 2.6 support is dropped. On that basis I've uploaded a change
to requirements to drop ordereddict: https://review.openstack.org/#/c/52053/

Please jump in with '-1's/'+1's as appropriate.

Thanks,

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Glance] Support of v1 and v2 glance APIs in Nova

[stuart . mclaren]

I'd just like to echo Tim Bell's comments.


From a rolling upgrade perspective where you have Glance nodes behind

a load balancer we'd probably need a way to manually 'hold back' on v1
until all nodes are upgraded to support v2 aswell.
(Otherwise your auto-discovery may hit an upgraded node but your
subsequent API request could hit a node which hasn't been upgraded yet.)


On 10/17/2013 03:12 PM, Eddie Sheffield wrote:

I don't oppose having the version autodiscovered. But I do feel the option 
should be available to override if desired. I've had many problems with

over the years with autoconfiguring systems not doing what I need to be willing 
to depend on them 100% without a manual override available if at all
possible. I'm thinking particularly for testing or upgrade evaluation 
scenarios. For example, you want to turn on v2 in glance and evaluate it for a 
bit
before committing to using it for all of your nova nodes.


The current patch also had a couple of comments from Dan Prince and Chris 
Behrens early on when this was brought up there supporting the use of a

config value.


From the implementation side of things, to do this properly would require some 
changes to the glanceclient to allow querying of available versions.

Lacking that ability currently is one reason this is currently a config value. 
Once the client supports this, the nova side change would likely be quite
small.


Might I propose a compromise?

1) For the VERY short term, keep the config value and get the change otherwise 
reviewed and hopefully accepted.

2) Immediately file two blueprints:
   - python-glanceclient - expose a way to discover available versions
   - nova - depends on the glanceclient bp and allowing autodiscovery of glance 
version
and making the config value optional (tho not deprecated / removed)


Supporting both seems reasonable.  At least then *most* people don't
need to worry about it and it just works, but the override is there if
necessary, since multiple people seem to be expressing a desire to have
it available.

Can we just do this all at once?  Adding this to glanceclient doesn't
seem like a huge task.

--
Russell Bryant


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance]: V2 api support for download_image policy?

[stuart . mclaren]

Hi Abishek,

download_image *should* apply to v2 in the same way as v1 I think.

If that's not what you're seeing can you enter a bug (with details) in 
launchpad?

https://bugs.launchpad.net/glance

Typically you'd mark it with 'private security' but 'public security'
makes sense now.

Thanks!

-Stuart


Hi All,

Can anyone let me know whether is download_image policy applicable only for V1 
api and not for V2 api?

Thanks  Regards,

Abhishek

__
Disclaimer:This email and any attachments are sent in strictest confidence for 
the sole use of the addressee and may contain legally privileged, confidential, 
and proprietary data.
If you are not the intended recipient, please advise the sender by replying 
promptly to this email and then delete and destroy this email and any 
attachments without any further use,
copying or forwarding
-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.openstack.org/pipermail/openstack-dev/attachments/20140603/b4236c77/attachment-0001.html


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] multi-tenant store in Juno

[stuart . mclaren]

All,

On my devstack setup neither image upload or image download work
for the Swift multi-tenant store in Juno (I'm getting E500s).

I'd be interested if someone can confirm.

With these changes upload/download worked for me:

glance upload
https://review.openstack.org/#/c/130757/

glance store upload
https://review.openstack.org/#/c/130755/

glance store download
https://review.openstack.org/#/c/130743/

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] security and swift multi-tenant fixes on stable branch

[stuart . mclaren]

All,

The 0.1.9 version of glance_store, and glance's master branch both
contain some fixes for the Swift multi-tenant store.

This security related change hasn't merged to glance_store yet:
https://review.openstack.org/130200

I'd like to suggest that we try to merge this security fix and release
it as as glance_store '0.1.10'. Then make glance's juno/stable branch
rely on glance_store '0.1.10' so that it picks up both the multi-tenant store
and security fixes.

The set of related glance stable branch patches would be:
https://review.openstack.org/134257
https://review.openstack.org/134286
https://review.openstack.org/134289/ (0.1.10 dependency -- also requires a 
global requirements change)

Does this seem ok?

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] security and swift multi-tenant fixes on stable branch

[stuart . mclaren]

On 2014-11-13 18:28:14 +0100 (+0100), Ihar Hrachyshka wrote:
[...]
I think those who maintain glance_store module in downstream
distributions will cherry-pick the security fix into their
packages, so there is nothing to do in terms of stable branches to
handle the security issue.
[...]



As a counterargument, some Oslo libs have grown stable branches for
security backports and cut corresponding point releases on an
as-needed basis so as to avoid introducing new features in stable
server deployments.
--
Jeremy Stanley


The current glance stable/juno requirement for glance_store is = 0.1.1.

If you run stable/juno against glance_store 0.1.1 and try
to create an image, you get (multi-tenant store):

$ glance image-create --name image1 --container-format bare --disk-format raw
html
 head
  title410 Gone/title
 /head
 body
  h1410 Gone/h1
  Error in store configuration. Adding images to store is disabled.br /br /
 /body
/html (HTTP N/A)

With the latest (0.1.9) glance_store, you get:

$ glance image-create --name image1 --container-format bare --disk-format raw
html
 head
  title500 Internal Server Error/title
 /head
 body
  h1500 Internal Server Error/h1
  Failed to upload image 702d5865-8925-4d0d-b52c-c93833dc5eaabr /br /
 /body
/html (HTTP 500)

Before glance_store was separated out it would have been straightforward
to backport the relevant fixes to Glance's tightly coupled in-tree store code.

I'm neutral on the mechanics, but I think we need to get to a point where
if someone is running stable/juno and has a version of glance_store which
satisfies what's specified in requirements.txt they should have secure,
working code.

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova/glance] performance related change

[stuart . mclaren]

All,

This change: https://review.openstack.org/#/c/32562/ seems to
have slipped off the radar, possibly because it has a '-1' -- but
I'm waiting on some feedback before progressing.

The patch can provide a good performance improvement (depending on your
configuration): I measure  3x for image download/snapshot between glance
and nova when using https.

If any nova folks could take a look that would be much appreciated.
(The equivalent cinder patch has already merged.)

Thanks!

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] FFE Request: Add swift_store_ssl_compression param

[stuart . mclaren]

This change:

https://review.openstack.org/#/c/32919/

is the final piece of a puzzle that allows a (potentially significant)
performance improvement for image uploads (snapshots)/downloads when
using ssl.

Its the last change required for the following paths:

(nova/cinder) - (glance client) - (glance) - (swift client) - (swift)

Without this patch swift will be a bottleneck running at ~17 MB/s while
the other parts can potentially reach ~100 MB/s.

Risk: Currently the patch sets compression to be disabled by default
(giving better performance), but the old behaviour can be reverted
by setting the relevant config parameter. (We could even potentially
consider defaulting to the old behaviour.)

The patch was originally uploaded on Jun 13.

-Stuart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Nominating Zhi Yan Liu for glance-core

[stuart . mclaren]

+1


Hey,

I would like to nominate Zhi Yan Liu(lzydev) for glance core. I think Zhi has 
been an active reviewer/contributor to the glance community [1] and has
always been on top of reviews.

Thanks for the good work Zhi!

Iccha

[1] http://russellbryant.net/openstack-stats/glance-reviewers-30.txt


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.openstack.org/pipermail/openstack-dev/attachments/20131003/7fbed70d/attachment-0001.html

--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Nominating Fei Long Wang for glance core

[stuart . mclaren]

+1

Hey,

I would like to nominate Fei Long Wang(flwang) for glance core. I think Fei has 
been an active reviewer/contributor to the glance community [1] and has
always been on top of reviews.

Thanks for the good work Fei!

Iccha

[1] http://russellbryant.net/openstack-stats/glance-reviewers-30.txt


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.openstack.org/pipermail/openstack-dev/attachments/20131003/63f0cece/attachment-0001.html

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] zero downtime conf/kilo

[stuart . mclaren]

All,

The main zero downtime config patch has merged (thanks to everyone, especially 
Abhishek, for their help).

There's a couple of small, follow on corner case patches, that, eg ensure a config 
change to logging (INFO-DEBUG)
can be picked up dynamically.

Are we planning/hoping to merge them also? (I think it makes a better story 
than the current partial support, and could see kilo bugs
of the 'I changed tcp_keepalive and it wasn't picked up' or 'I changed the log 
level and it wasn't picked up' coming in.)

Thanks,

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] reminder to check 'Author' of patch set when reviewing

[stuart . mclaren]

A quick reminder for Glance cores to check if the 'Author' changes
between patch sets.

When someone other than the original submitter uploads a new patch set
they should, if appropriate, add themselves as a co-author rather than
replace the original patch author. (It's the 'Author' rather than the
'Owner' that will get listed in git log etc after the patch merges.)

Thanks!

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [taskflow][glance][debian] Issue with ordereddict

[stuart . mclaren]

Hi,

Debian are hitting this issue https://bugs.launchpad.net/glance/+bug/1445827.

Currently glance and glance_store include ordereddict in their
requirements.txt.  Since Glance no longer gates on py26 I think it should
be ok to remove that requirement from glance and glance_store
 (both repos have stable branches) which should partially fix the issue.

taskflow also seems to have this dependency:

 $ cat ./.tox/py27/lib/python2.7/site-packages/taskflow-0.7.1.dist-info/  
METADATA|grep order
 Requires-Dist: ordereddict

but, unlike glance, it still gates on py26. As I understand it, removing
this from taskflow's requirements would cause its py26 gate job to
start failing.

I'm interested in suggestions from the taskflow folks on what the best
thing to do here is.

Thanks,

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [eventlet][requirements] Issue with eventlet = 0.17.2 and ssl

[stuart . mclaren]

All,

When using a version of python which has the new SSLContext (ie python 3, 2.7.9 
and
at least some 2.7.8 versions) this issue:

https://github.com/eventlet/eventlet/issues/226

(where the combination of a newer python with https/eventlet and requests 
doesn't handle
SSL certs propertly) seems to prevent running OpenStack with https using later 
2.x python
versions.

Has anyone else hit this?

There is a fix now on eventlet's master branch, but it hasn't been released yet.
Since versions of eventlet prior to 0.17.3 aren't python 3 compatible [1] I'm
guessing that we need to wait for a new eventlet to be released and can't
update global-requirements to pin to an older version as a workaround?

Thanks.

-Stuart

[1] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061091.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra][glanceclient][cinderclient] Problems with juno check jobs

[stuart . mclaren]

Hi Gorka,

Glance is seeing something very similar [3].

I've updated the two bugs ([1],[3]) with some extra info.
Both issues seem to have started around April 7th.

Would anyone from infra be able to take a quick look?

Thanks,

-Stuart

[3] https://bugs.launchpad.net/glance/+bug/1442682


Date: Mon, 13 Apr 2015 13:35:42 +0200
From: Gorka Eguileor gegui...@redhat.com
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Heat] [Cinderclient] All patches getting -1
on  Cinderclient
Message-ID: 20150413113541.ga23...@mail.corp.redhat.com
Content-Type: text/plain; charset=us-ascii

Hi all,

Currently all patches in Cinderclient are getting -1 from Jenkins
because gate-tempest-dsvm-neutron-src-python-cinderclient-juno is
failing.

I opened a LP bug [1] on this, but basically the issue comes from Heat's
requirements cap on Cinderclient [2] to an upper bound of 1.1.1 when
current version is reported as 1.1.1.post100.

So if you're getting -1 from that job, remember it's not you. ;-)


Cheers,
Gorka.

[1] https://bugs.launchpad.net/tempest/+bug/1442086
[2] https://github.com/openstack/heat/blob/stable/juno/requirements.txt#L25


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance][Keystone] Glance and trusts

[stuart . mclaren]

On Jun 5, 2015, at 19:25, Bhandaru, Malini K malini.k.bhand...@intel.com 
wrote:

Continuing with David?s example and the need to control access to a Swift 
object that Adam points out,

How about using the Glance token from glance-API service to glance-registry but 
carry along extra data in the call, namely user-id, domain, and public/private 
information, so the object can be access controlled.

Alternately and encapsulating token

Glance-token user-token -- keeping it simple, only two levels.  This 
protects from on the cusp expired user-tokens.
Could check user quota before attempting the storage.


We already went over this type of design and determined it was sub-optimal. 
Instead we allow for passing the X-SERVICE-TOKEN, which is being passed in 
addition to the auth token.
Right now I do not believe that X-SERVICE-TOKEN is being used anywhere.


Support was recently added to the Swift server. (Glance - Swift support should 
land in Liberty.)


In the near future we are looking to make all inter-service (e.g. Nova 
-glance) communication have the service token.

This design was specifically implemented for the case you're describing.

In theory it would be possible to allow the quota check / read with only a 
service token (and a proper policy.json) but would require the user's token to 
do writes.




Should user not have paid dues, Glance knows which objects to garbage collect!

Regards
Malini

From: Adam Young [mailto:ayo...@redhat.com]
Sent: Friday, June 05, 2015 4:11 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Glance][Keystone] Glance and trusts

On 06/05/2015 10:39 AM, Dolph Mathews wrote:

On Thu, Jun 4, 2015 at 1:54 AM, David Chadwick d.w.chadw...@kent.ac.uk wrote:
I did suggest another solution to Adam whilst we were in Vancouver, and
this mirrors what happens in the real world today when I order something
from a supplier and a whole supply chain is involved in creating the end
product that I ordered. This is not too dissimilar to a user requesting
a new VM. Essentially each element in the supply chain trusts the two
adjacent elements. It has contracts with both its customers and its
suppliers to define the obligations of each party. When something is
ordered from it, it trusts the purchaser, and on the strength of this,
it will order from its suppliers. Each element may or may not know who
the original customer is, but if it needs to know, it trusts the
purchaser to tell it. Furthermore the customer does not need to delegate
any of his/her permissions to his/her supplier. If we used such a system
of trust between Openstack services, then we would not need delegation
of authority and trusts as they are implemented today. It could
significantly simplify the interactions between OpenStack services.

+1! I feel like this is the model that we started with in OpenStack, and have 
grown additional complexity over time without much benefit.

We could roll Glance into Nova, too, and get the same benefit.  There is a 
reason we have separate services.  GLance shoud not Trust Nova for all 
operations, just some.

David's example elides the fact that there  are checks built in to the supply 
chain system to prevent cheating.







regards
David

On 03/06/2015 21:03, Adam Young wrote:

On 06/02/2015 12:57 PM, Mikhail Fedosin wrote:

Hello! I think it's a good time to discuss implementation of trusts in
Glance v2 and v3 api.

Currently we have two different situations during image creation where
our token may expire, which leads to unsuccessful operation.

First is connection between glance-api and glance-registry. In this
case we have a solution (https://review.openstack.org/#/c/29967/) -
use_user_token parameter in glance-api.conf, but it is True by default
. If it's changed to False then glance-api will use its own
credentials to authorize in glance-registry and it prevents many
possible issues with user token expiration. So, I'm interested if
there are some performance degradations if we change use_user_token to
False and what are the reasons against making it the default value.

Second one is linked with Swift. Current implementation uploads chunks
one by one and requires authorization each time. It may lead to
problems: for example we have to upload 100 chunks, after 99th one,
token expired and glance can't upload the last one, catches an
exception and tries to remove stale chunks from storage. Of course it
will fail, because token is not valid anymore, and that's why there
will be 99 garbage objects in the storage.
With Single-tenant mode glance uses its own credentials to upload
files, so it's possible to create new connection on each chunk upload
or catch Unauthorized exception and recreate connections only in that
cases. But with Multi-tenant mode there is no way to do it, because
user credentials are required. So it seems that trusts is the only one
solution here.

The problem with using trusts is that it would need to be created
per-user, and that is going to 

Re: [openstack-dev] [glance] V3 Authentication for swift store

[stuart . mclaren]

Hi Jamie,

Glance has another way of specifying the swift credentials for the single
tenant store which may (?) be useful here.

In glance-swift.conf you can specify something like:

[ref1]
user = tenant:user1
key = key1
auth_address = auth...@example.com

which means that in the database 'ref1' is stored instead of the
credentials: the location ends up looking something like:

swift+config://ref1@swift/container/object

The swift+config schema is used to indicate the real creds should be
fetched from the config file. (This avoids having to put them in the
database which isn't desirable and complicates password changing.)

We'd have to think about corner cases (I think --copy-from should be ok).

-Stuart


Hey everyone,

TL;DR: glance_store requires a way to do v3 authentication to the swift backend.

The keystone team is making a push to properly deprecate the v2 authentication 
APIs this cycle. As part of that we have a series of devstack reviews that 
moves devstack over to only using v3 APIs[1] and an experimental gate job that 
runs devstack with the keystone v2 api disabled.

The current blocker for this gate job is that in glance's single-tenant swift 
backend mode the config options only allow v2 authentication.

Looking at glance store the username and password are stored as part of the 
location parameter in the form:

swift://username:project_name:password@keystone/container

even though devstack is still using the (deprecated?)

swift_store_user = username:project_name
swift_store_key = password
swift_store_container = container

I don't know how this relates to swift_store_config_files.

There is support for v3 in swiftclient (though it's kind of ugly), to do v3 
authentication i have to do:

c = swiftclient.Connection(authurl='http://keystoneurl:5000/v3',
  user=username,
  key=password,
  auth_version='3',
  os_options={'project_name': project_name,
  'project_domain_id': 'default',
  'user_domain_id': 'default'})

However in future we are trying to open up authentication so it's not limited 
to only user/password authentication. Immediate goals for service to service 
communications are to enable SSL client certificates and kerberos 
authentication. This would be handled by keystoneclient sessions but they are 
not supported by swift and it would require a significant rewrite of 
swiftclient to do, and the swift team has indicated they do not which to invest 
more time into their client.

This leads me to my question: How do we support additional authentication 
parameters and in future different parameters?

We could undo the deprecation of the config file specified credentials and add 
the additional options there. This would get us the short term win of moving to 
v3 auth but would need to be addressed in future for newer authentication 
mechanissms.

I wrote a blog a while ago regarding how sessions supports loading different 
types of authentication from conf files[2], however as swiftclient doesn't 
support this the best it could do is fetch a url/token combo with which glance 
could make requests and it would have to handle reauthentication and retries 
somewhat manually. I actually think rewriting the required parts of the client 
wouldn't be too difficult, the problem then is whether this should live in 
glance or in swiftclient. This would also involve credentials in the config 
file rather than the location option.

I'm not overly familiar with glance_store so there may be other options or what 
i've suggested may not be possible but I'd love to hear some opinions from the 
glance team as this is quickly becoming a blocker for keystone.


Thanks,

Jamie



[1] https://review.openstack.org/#/c/186684/
[2] http://www.jamielennox.net/blog/2015/02/17/loading-authentication-plugins/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] [Cinder] [Glance] glance_store and glance

[stuart . mclaren]

I got zero responses on the mailing list raising a problem with Glance v2 [1].

I got zero responses on cross project meeting raising a problem with Glance v2
[2].

I'm very happy with my choice of words, because I think this hand slap on
Glance is the first time I got acknowledgement in my frustration with Glance.

I should not have to attend a Glance meeting to get someone to fix Glance v2
integration issues in Cinder.

Glance is trying to increase v2 integration with Nova besides show [3], but
I would recommend Nova to not accept further v2 integration until Glance has
figured out how to handle issues in projects that already have v2 integration.

To start, Glance should assign a cross project liaison [4] to actually respond
to integration issues in Cinder.

Having focuses on the following is not helping:

* Artifacts (I honestly don't know what this is and failed to find an
 explanation that's *not* some API doc).


Hi Mike,

There has definitely been debate around artifacts, both within and outside
the project.  Rather than beating us up, I'm genuinely interested to
know if you have any words of advice on how we could have handled this
differently (to avoid 'pissing off the community').

The original patch to extend Glance's mission to include artifacts is here:

https://review.openstack.org/#/c/98002

The set of approvers show that this was an OpenStack-wide effort rather
than a solo run by Glance.

At the summit in Vancouver we held a session to revisit this.  Around 40
people attended (including around 7 TC members) and debated artifacts
openly and with a constructive tone.

My memory is that opinions in the room were fairly equally split. One
TC member said it would be 'embarrasing' if OpenStack had two catalog
services. Another TC member adamently advocated that Glance should stick
to images.

We reached out to the community for feedback and acted as best we could
on the feedback we received.

It would have been ok (if unpopular) for us to have acted unilaterally.

How would Cinder have handled this type of situation? What could/should
we have done differently? What would you suggest we do now?


* Tagging


If you mean 'metadefs' I'd tend to agree here. But, post the big tent
model, we have -- at least in one case -- kept focus by promoting proposed
non-core functionality to its own project:

https://review.openstack.org/#/c/188014


* Role based properties [5] (who is asking for this, and why is Glance
enforcing roles?)


Protected properties are typically used for licensing, so there is a real
business/legal use case here. The public clouds I know of use them. Is the
implementation stellar? Possibly not.



This is a mess, and complete lack of focus on being what Glance was once good
at, a registry for images.


[1] - http://lists.openstack.org/pipermail/openstack-dev/2015-July/070714.html
[2] - 
http://eavesdrop.openstack.org/meetings/crossproject/2015/crossproject.2015-07-28-21.03.log.html#l-239
[3] - https://github.com/openstack/nova/blob/master/nova/image/glance.py#L305
[4] - 
https://wiki.openstack.org/wiki/CrossProjectLiaisons#Inter-project_Liaisons
[5] - https://review.openstack.org/#/c/211201/1

--
Mike Perez


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] V3 authentication to swift

[stuart . mclaren]

Thanks for the heads up Jamie, I'll try to take a look.

-Stuart


Glancers, 


A while ago I wrote an email outlining a couple of ways we could support 
keystone v3 authentication when using swift as a backend for glance_store. In 
the long term it would be best to transition swiftclient to use sessions as
this would allow us to do more extensive (kerberos, ssl etc) authentication in the 
future, however this is a longer term plan that requires a lot of effort and 
coordination. For right now glance-swift is the last piece of
devstack that only works with v2 authentication[1] and with a relatively small 
amount of code we can support v3 username/password which is by far the most 
common scenario.

I've been prompting for a while on IRC however can I please request some 
reviews of https://review.openstack.org/#/c/193422/ as it has now become a 
blocker for the v3 authentication effort.


Thanks for your help, 

Jamie 



[1] Note, that by this i mean that with v2 keystone disabled glance-swift 
communication is the last thing in the standard devstack gate path that can't be 
configured to v3. It does not mean that tempest will succeed or that all
of OpenStack is v3 compatible just that we can start running meaningful gate 
jobs.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] upcoming glanceclient release

[stuart . mclaren]

I've compiled a list of backwards incompatabilities where the new client
will impact (in some cases break) existing scripts:

https://wiki.openstack.org/wiki/Glance-v2-v1-client-compatability



On 27/08/15 15:32 -0400, Nikhil Komawar wrote:


As a part of our continued effort to make the v2 primary API and get
people to consume it without confusion we are planning to move ahead
with the client release (the release would set the default version of
API to 2). There haven't been any major/minor raised here.

An issue regarding the possible impact of this release due to major bump
was raised during the morning meeting however, the client release should
follow semver semantics and indicate the same. A corresponding review
for release notes exists that should merge before the release. This
medium of communication seems enough; it follows the prescription for
necessary communication. I don't seem to find a definition for necessary
and sufficient media for communicating this information so will take
what we usually follow.

There are a few bugs [1] that could be considered as part of this
release but do not seem to be blockers. In order to accommodate the
deadlines of the release milestones and impact of releases in the
upcoming week to other projects, we can continue to fix bugs and release
them as a part of 1.x.x releases sooner than later as time and resource
permit. Also, the high ones can be part of the stable/* backports if
needed but the description has only shell impact so there isn't a strong
enough reason.

So, we need to move ahead with this release for Liberty.


+1

We've been making small steps towards this for a couple of cycles and
I'm happy we're finally switching the default version on the client
library.

The above being said, I believe our client library needs a lot of more
work but this release should set us in a better position to do that.

For folks consuming glanceclient, here's what you need to know:

If you're using glanceclient from your software - that is, you're
using the library and not the CLI - there's nothing you need to do. If
you're using the library, I'm assuming you're creating a client
instance using[0] or by instantiating the specific versioned client
class. Both of these cases require you to specify an API version to
use.

However, if you're using the CLI and you want to stick using the V1,
then you'll need to update your scripts and make sure they use
`--os-image-api 1`. Respectively, if your scripts are using
`--os-image-api 2`, then you should feel free to ignore that argument.

As a good practice, for now, I'd recommend specifying the argument
regardless of the version.

For other changes, please review the rel notes[1] (as soon as they are
there, or you can read[2])

[0] 
https://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/client.py?h=stable/kilo#n21
[1] http://docs.openstack.org/developer/python-glanceclient/#release-notes
[2] https://review.openstack.org/#/c/217591/




[1]
https://bugs.launchpad.net/python-glanceclient/+bugs?field.tag=1.0.0-potential


Thanks Stuart for tagging these bugs and everyone for raising great
concerns pro/against the release[0]. And thanks Erno for pushing this
out.

[0] 
http://eavesdrop.openstack.org/meetings/glance/2015/glance.2015-08-27-14.00.log.html

Flavio


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] upcoming glanceclient release

[stuart . mclaren]

I've compiled a list of backwards incompatabilities where the new client
will impact (in some cases break) existing scripts:

https://wiki.openstack.org/wiki/Glance-v2-v1-client-compatability



Awesome!


To be honest there's a little more red there than I'd like.

Of the 72 commands I tried, the new client failed to even parse the input in 36 
cases.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] [nova] Verification of glance images before boot

[stuart . mclaren]

The glance client (running 'inside' the Nova server) will re-calculate
the checksum as it downloads the image and then compare it against the
expected value. If they don't match an error will be raised.


How can I know that the image that a new instance is spawned from - is
actually the image that was originally registered in glance - and has
not been maliciously tampered with in some way?

Is there some kind of verification that is performed against the md5sum
of the registered image in glance before a new instance is spawned?

Is that done by Nova?
Glance?
Both? Neither?

The reason I ask is some 'paranoid' security (that is their job I
suppose) people have raised these questions.

I know there is a glance BP already merged for L [1] - but I would like
to understand the actual flow in a bit more detail.

Thanks.

[1]
https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support

--
Best Regards,
Maish Saidel-Keesing



--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


End of OpenStack-dev Digest, Vol 41, Issue 22
*



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[stuart . mclaren]

I've been looking into the existing task-based-upload that Doug mentions:
can anyone clarify the following?

On a default devstack install you can do this 'task' call:

http://paste.openstack.org/show/462919


Yup. That's the one.


as an alternative to the traditional image upload (the bytes are streamed
from the URL).

It's not clear to me if this is just an interesting example of the kind
of operator specific thing you can configure tasks to do, or a real
attempt to define an alternative way to upload images.

The change which added it [1] calls it a 'sample'.

Is it just an example, or is it a second 'official' upload path?


It's how you have to upload images on Rackspace.


Ok, so Rackspace have a task called image_import. But it seems to take
different json input than the devstack version. (A Swift container/object
rather than a URL.)

That seems to suggest that tasks really are operator specific, that there
is no standard task based upload ... and it should be ok to try
again with a clean slate.


If you want to see the
full fun:

https://github.com/openstack-infra/shade/blob/master/shade/__init__.py#L1335-L1510

Which is "I want to upload an image to an OpenStack Cloud"

I've listed it on this slide in CLI format too:

http://inaugust.com/talks/product-management/index.html#/27

It should be noted that once you create the task, you need to poll the
task with task-show, and then the image id will be in the completed
task-show output.

Monty


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[stuart . mclaren]

After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.


Doug, thanks for reaching out here.

I've been looking into the existing task-based-upload that Doug mentions:
can anyone clarify the following?

On a default devstack install you can do this 'task' call:

http://paste.openstack.org/show/462919

as an alternative to the traditional image upload (the bytes are streamed
from the URL).

It's not clear to me if this is just an interesting example of the kind
of operator specific thing you can configure tasks to do, or a real
attempt to define an alternative way to upload images.

The change which added it [1] calls it a 'sample'.

Is it just an example, or is it a second 'official' upload path?

-Stuart

[1] https://review.openstack.org/#/c/44355

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Models and validation for v2

[stuart . mclaren]

We've been taking validation out as issues have been reported (it was
removed from image-list recently for example).

Removing across the board probably does make sense.



Agree with you. That's why I am asking about reasoning. Perhaps, we need to
realize how to get rid of this in glanceclient.

Best regards,
Kairat Kushaev

On Wed, Sep 30, 2015 at 7:04 PM, Jay Pipes  wrote:


On 09/30/2015 09:31 AM, Kairat Kushaev wrote:


Hi All,
In short terms, I am wondering why we are validating responses from
server when we are doing
image-show, image-list, member-list, metadef-namespace-show and other
read-only requests.

AFAIK, we are building warlock models when receiving responses from
server (see [0]). Each model requires schema to be fetched from glance
server. It means that each time we are doing image-show, image-list,
image-create, member-list and others we are requesting schema from the
server. AFAIU, we are using models to dynamically validate that object
is in accordance with schema but is it the case when glance receives
responses from the server?

Could somebody please explain me the reasoning of this implementation?
Am I missed some usage cases when validation is required for server
responses?

I also noticed that we already faced some issues with such
implementation that leads to "mocking" validation([1][2]).



The validation should not be done for responses, only ever requests (and
it's unclear that there is value in doing this on the client side at all,
IMHO).

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- next part --
An HTML attachment was scrubbed...
URL: 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Add Ian Cordasco back into glance-core

[stuart . mclaren]

+1

I'm delighted to hear this.

Welcome back Ian.



+1 from me!  Glad to have Ian back.

On 12/7/15, 11:36 AM, "Flavio Percoco"  wrote:

> Greetings,
> 
> Not long ago, Ian Cordasco, sent an email out stepping down from his

> core roles as he didn't have the time to dedicate to the project
> team's he was part of.
> 
> Ian has contacted me mentioning that he's gotten clearance, and

> therefore, time to dedicate to Glance and other activities around our
> community (I'll let him expand on this and answer questions if there
> are).
> 
> As it was mentioned in the "goodbye thread" - and because Ian knows

> Glance quite well already, including the processes we follow - I'd
> like to propose a fast-track addition for him to join the team again.
> 
> Please, just like for every other folk volunteering for this role, do

> provide your feedback on this. If no rejections are made, I'll proceed
> to adding Ian back to our core team in a week from now.
> 
> Cheers,

> Flavio
> 
> --

> @flaper87
> Flavio Percoco






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-glanceclient] Return request-id to caller

[stuart . mclaren]

Excerpts from Flavio Percoco's message of 2015-12-09 09:09:10 -0430:

On 09/12/15 11:33 +, Kekane, Abhishek wrote:

Hi Devs,



We are adding support for returning ?x-openstack-request-id?  to the caller as
per the design proposed in cross-project specs:

http://specs.openstack.org/openstack/openstack-specs/specs/
return-request-id.html



Problem Description:

Cannot add a new property of list type to the warlock.model object.



How is a model object created:

Let?s take an example of glanceclient.api.v2.images.get() call [1]:



Here after getting the response we call model() method. This model() does the
job of creating a warlock.model object(essentially a dict) based on the schema
given as argument (image schema retrieved from glance in this case). Inside
model() the raw() method simply return the image schema as JSON object. The
advantage of this warlock.model object over a simple dict is that it validates
any changes to object based on the rules specified in the reference schema.
The keys of this  model object are available as object properties to the
caller.



Underlying reason:

The schema for different sub APIs is returned a bit differently. For images,
metadef APIs glance.schema.Schema.raw() is used which returns a schema
containing ?additionalProperties?: {?type?: ?string?}. Whereas for members and
tasks APIs glance.schema.Schema.minimal() is used to return schema object which
does not contain ?additionalProperties?.



So we can add extra properties of any type to the model object returned from
members or tasks API but for images and metadef APIs we can only add properties
which can be of type string. Also for the latter case we depend on the glance
configuration to allow additional properties.



As per our analysis we have come up with two approaches for resolving this
issue:



Approach #1:  Inject request_ids property in the warlock model object in glance
client

Here we do the following:

1. Inject the ?request_ids? as additional property into the model object
(returned from model())

2. Return the model object which now contains request_ids property



Limitations:

1. Because the glance schemas for images and metadef only allows additional
properties of type string, so even though natural type of request_ids should be
list we have to make it as a comma separated ?string? of request ids as a
compromise.

2. Lot of extra code is needed to wrap objects returned from the client API so
that the caller can get request ids. For example we need to write wrapper
classes for dict, list, str, tuple, generator.

3. Not a good design as we are adding a property which should actually be a
base property but added as additional property as a compromise.

4. There is a dependency on glance whether to allow custom/additional
properties or not. [2]



Approach #2:  Add ?request_ids? property to all schema definitions in glance



Here we add  ?request_ids? property as follows to the various APIs (schema):



?request_ids?: {

 "type": "array",

 "items": {

   "type": "string"

 }

}



Doing this will make changes in glance client very simple as compared to
approach#1.

This also looks a better design as it will be consistent.

We simply need to modify the request_ids property in  various API calls for
example glanceclient.v2.images.get().



Hey Abhishek,

thanks for working on this.

To be honest, I'm a bit confused on why the request_id needs to be an
attribute of the image. Isn't it passed as a header? Does it have to
be an attribute so we can "print" it?


The requirement they're trying to meet is to make the request id
available to the user of the client library [1]. The user typically
doesn't have access to the headers, so the request id needs to be
part of the payload returned from each method. In other clients


Will this work if the payload is image data?


that work with simple data types, they've subclassed dict, list,
etc. to add the extra property. This adds the request id to the
return value without making a breaking change to the API of the
client library.

Abhishek, would it be possible to add the request id information
to the schema data in glance client, before giving it to warlock?
I don't know whether warlock asks for the schema or what form that
data takes (dictionary, JSON blob, etc.). If it's a dictionary
visible to the client code it would be straightforward to add data
to it.

Failing that, is it possible to change warlock to allow extra
properties with arbitrary types to be added to objects? Because
validating inputs to the constructor is all well and good, but
breaking the ability to add data to an object is a bit un-pythonic.

If we end up having to change the schema definitions in the Glance API,
that also means changing those API calls to add the request id to the
return value, right?

Doug

[1] 
http://specs.openstack.org/openstack/openstack-specs/specs/return-request-id.html



As it is presented in your email, I'd probably go with option #2 but
I'm curious to know 

Re: [openstack-dev] [glance] Seeking FFE for "Support single disk image OVA/OVF package upload"

[stuart . mclaren]

Hello Glance Team!
   Hope you had a wonderful vacation and wishing you health and 
happiness for 2016.

Would very much appreciate your considering https://review.openstack.org/194868 
for a feature freeze exception.

I believe the spec is pretty solid, and we can deliver on the implementation by 
M-2. But were unable to get enough core



Votes during the holiday season.



   Regards



   Malini




-- next part --
An HTML attachment was scrubbed...
URL: 



I downloaded the patch which implements the spec 
(https://review.openstack.org/#/c/214810):

I can make this REST API call to perform OVA import:

http://paste.openstack.org/show/483332/

(it exercises the new OVA extract code path).

There's a parallel effort in the project to provide 'official' (Defcore)
APIs for image upload/conversion. What will be the advantage of having
two different REST API calls (a 'tasks' based one and a DefCore one)
for importing OVAs?

It seems to be possible to successfully make the above API call whether
you're admin or not and whether the server has the patch applied or
not. Is that expected?

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Seeking FFE for "Support single disk image OVA/OVF package upload"

[stuart . mclaren]

Hello Glance Team!
  Hope you had a wonderful vacation and wishing you health 
and happiness for 2016.


Would very much appreciate your considering 
https://review.openstack.org/194868 for a feature freeze exception.


I believe the spec is pretty solid, and we can deliver on the 
implementation by M-2. But were unable to get enough core



Votes during the holiday season.



  Regards



  Malini




-- next part --
An HTML attachment was scrubbed...
URL: 



I downloaded the patch which implements the spec 
(https://review.openstack.org/#/c/214810):


I can make this REST API call to perform OVA import:

http://paste.openstack.org/show/483332/

(it exercises the new OVA extract code path).

There's a parallel effort in the project to provide 'official' (Defcore)
APIs for image upload/conversion. What will be the advantage of having
two different REST API calls (a 'tasks' based one and a DefCore one)
for importing OVAs?


As you mentioned above, the team is working on refactoring the image
import process for Glance. The solution has different requirements and
dependencies. One of those dependencies is making the existing task
API admin only to then be able to deprecate it in the future.

This has been discussed several times at the summit, in meetings and,
to make sure it's clear to everyone (apparently it isn't) it's also
been made of the spec of the refactor you mentioned[0] (see work
items).

[0] 
https://specs.openstack.org/openstack/glance-specs/specs/mitaka/approved/image-import/image-import-refactor.html#work-items



 I understand we're aiming to retire the tasks API and that this call is
 intended to be admin only.

 I guess what I'd like to get a handle on is this:

 In the future, if a user asks "Why did my OVA API call fail?", we'll
 need to ask "Well, which OVA API did you use?"

 At that level there is a duplication -- something we'd generally try to
 avoid. If we're going to have that duplication I'd just like to understand
 why. For example, is it because we want the functionality exposed
 sooner? Or is it that the ability to trigger the new functionality via
 a /tasks call is a side-effect of the task implementation (one we don't
 really want in this case) and it would be too much effort to change it.



It seems to be possible to successfully make the above API call whether
you're admin or not and whether the server has the patch applied or
not. Is that expected?


You'll be able to run that request until this[0] is done.


 The current implementation hard codes that the OVA functionality is admin
 only.  But the call still "succeeds" as a non-admin (creates an active
 image) because it re-uses the 'import' task type (ie we've an overloaded
 API). That means that if you're admin the API is ambiguous (knowing
 the request and response is not enough to know what actually happened).
 That may only be the current implementation though, and may not be what
 the spec intended.

 (Let's follow this bit up on the spec.)


In addition
to this work, there's also the requirement to make the task executable
only by admins.
This has been explicitly mentioned in the OVF spec and
we need to test/enforce that the code respects this.

Note that we're evaluating an exception for the *spec* and not the
code. Therefore, using the current version of the code as a reference
rather than what's in the spec is probably not ideal.

One final note that I'd like to make. The *task class/implementation*
has nothing to do with the API. It can be functionally tested without
API calls and it can be disabled. The fact that you can run it using
the old task API doesn't mean that you should or that it's what we're
recomending. The old task API is taking its first step down the
deprecation path and it'll take some time for that to happen. This, I
insist, does not mean the team is encouraging such API.

The OVF work was delayed in Kilo. We also blocked in Liberty because
we knew the upload path needed to be refectored. In Mitaka we blocked
it until the very end of the spec review process because we wanted to
make sure it wouldn't interfeer with the priorities of the cycle. Now
that we know that, I can't hardly think of a reason to not let this
through. One motivation is that I don't think it'll confuse folks as
we're clearly saying (in code, communications and whatnot) that the
old task API should go away.

Sure, some ppl don't listen and the world isn't perfect but there are
trade offs and those are the ones we're evaluating.

[0] https://bugs.launchpad.net/glance/+bug/1527716

Cheers,
Flavio



-Stuart



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

[openstack-dev] [glance] FFE for "Multi-tenant Swift store service token support"

[stuart . mclaren]

Hi,

I'd like to request an exception for this spec:

 https://review.openstack.org/#/c/170564/

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][all] glance_store drivers deprecation/stabilization: Volunteers needed

[stuart . mclaren]

On 11/01/16 15:52 -0500, Flavio Percoco wrote:

Greetings,

Gentle reminder that this is happening next week.

Cheers,
Flavio

- Original Message -

From: "Flavio Percoco" 
To: openstack-dev@lists.openstack.org
Sent: Thursday, December 10, 2015 9:16:09 AM
Subject: [glance][all] glance_store drivers deprecation/stabilization: 
Volunteers needed

Greetings,

As some of you know, there's a proposal (still a rough draft) for
refactoring the glance_store API. This library is the home for the
store drivers used by Glance to save or access the image data.

As other drivers in OpenStack, this library is facing the issue of
having unmaintained, untested and incomplete implementations of stores
that are, hypotetically, being used in production environments.

In order to guarantee some level of stability and, more important,
maintenance, the Glance team is looking for volunteers to sign up as
maintainers/keepers of the existing drivers.

Unfortunately, given the fact that our team is not as big as we would
like and that we don't have the knowledge to provide support for every
single driver, the Glance team will have to deprecate, and later
remove, the drivers that will remain without a maintainer.

Each driver will have to have a voting CI job running (maintained by
the driver maintainer) that will have to run Glance's functional tests
to ensure the API features are also supported by the driver.

There are 2 drivers I belive shouldn't fall into this category and
that should be maintained by the Glance community itself. These
drivers are:

- Filesystem
- Http

Please, find the full list of drivers here[0] and feel free to sign up
as volunteer in as many drivers as your time permits to maintain.
Please, provide all the information required as the lack of it will
result in the candidacy not being valid. As some sharp eyes will
notice, the Swift driver is not in the list above. The reason for that
is that, although it's a key piece of OpenStack, not everyone in the
Glance community knows the code of that driver well-enough and there
are enough folks that know it that could perhaps volunteer as
maintainers/reviewers for it. Furthermore, adding the swift driver
there would mean we should probably add the Cinder one as well as it's
part of OpenStack just like Swift. We can extend that list later. For
now, I'd like to focus on bringing some stability to the library.

The above information, as soon as it's complete or the due date is
reached, will be added to glance_store's docs so that folks know where
to find the drivers maintainers and who to talk to when things go
south.

Here's an attempt to schedule some of this work (please refer to
this tag[0.1] and this soon-to-be-approved review[0.2] to have more
info w.r.t the deprecation times and backwards compatibility
guarantees):

- By mitaka 2 (Jan 16-22), all drivers should have a maintainer.
  Drivers without one, will be marked as deprecated in Mitaka.


This has been done!

http://docs.openstack.org/developer/glance_store/drivers/index.html

Only 1 driver was left without maintainer and, as established, I've marked it as
deprecated:

https://review.openstack.org/#/c/266077/

Thanks to everyone who volunteered!


Thanks for leading this Flavio!


Flavio



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] v2 image upload from url

[stuart . mclaren]

Sorry, can you give an example of the exact command you are using, please?
On 5 Feb 2016 22:45, "Fox, Kevin M"  wrote:


We've been using the upload image from http url for a long time and when
we upgraded to liberty we noticed it broke because the client's defaulting
to v2 now. How do you do image upload via http with v2? Is there a
different command/method?


Is this really a Glance question? If so, the Glance v2 API doesn't currently 
provide
a way to upload an image by specifying a URL (ie there's no equivalent to v1's 
copy-from).



Thanks,
Kevin


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [API] [Glance] New v2 Image Import

[stuart . mclaren]

On 08/01/16 17:14 +, Ian Cordasco wrote:

Hey all,


Yo!

Thanks for taking the time to read the spec and writing this email.


Yes, the more eyes the better -- so thanks.

Flavio asked me to document the various alternatives. At very least
it's good to capture them in the spec.

https://review.openstack.org/#/c/271021

Any feedback on these would be great.

Thanks,

-Stuart





I'm well aware that this isn't exactly timely because we've partially agreed on 
the import spec in large strokes. [1]_ I haven't had a great deal of time 
before now to really dig into the spec, though, so I'm bringing my feedback. 
I've spoken to some API-WG members and other Glance team members and I thought 
it would be best to discuss some of the concerns here.

An outline (or tl;dr) is roughly

- We're exposing way too much complexity to the user


I agree with this to some extent and I think we could work out a way
to reduce some of these calls or make them optional as in they should
always be exposed but the user doesn't need to call them to complete
an upload.


- The user has to be very aware of discovering all of the data from the cloud
? - The user has to store a lot of that data to make subsequent requests
??- Exposing information to the user is not bad, making the user resubmit it to 
the cloud is bad


This is the part I'd like to simplify a bit in the current spec.


- We're introducing two new image states. Do we need two?


In some of the patch sets we discused ways to avoid adding new status
but the agreement seemed to have been to add these two to communicate
things properly to users and be ease the way we reason about the
process within Glance. I'll quote Stuart "When you need a new status,
you need a new status"


- Quotas - Are these even still a thing?


Yes they are and they were discussed in the spec as well.


So the problem description explains why uploading an image can be very complex 
and why some clouds need to not allow uploads directly to Glance. I understand 
that very well. I also recognize that I am the person who advocated so strongly 
for having information endpoints that help the user discover information about 
the cloud. I think this allows for us to make smarter clients that can help the 
user do the right thing (DTRT) instead of shooting themselves in the foot 
accidentally.

All of that said, I have some very severe reservations about the extra surface 
area we're adding to the images API and in particular how a user's view of the 
image upload flow would work. We've shipped v2 of Glance as the default version 
now in python-glanceclient (as of Liberty) so users are already going to be 
familiar with a two-step dance of creating an image:

1. Create the image record (image-create)
2. Upload the image data (image-upload)

In this new workflow, the dance is extended

1. Get information about how the cloud is configured and prefers images


Technically, this call is optional. It is a requirement for scripts
that want to work accross multiple clouds as you need this info for
#4.


2. Create the image record
3. Upload the image data somewhere (directly to Glance, or to some other 
service in that cloud perhaps [e.g., swift]) that is specified in the data 
retrieved from step 1
4. Tell Glance to import/process the image


This one I think we could simplify a bit by not making all these
fields required and allow the cloud to have their own defaults.



Here are my issues with this new upload workflow:

1. The user must pre-fetch the information to have any chance of their upload 
being successful


Not really (read above).


2. They have to use that information to find out where to upload the image 
bytes (data) to


Not really. The upload info is returned in the `image-create` call
which is a requirement in any of the evaluated workflow options. The
new header tells the user where the image data should go.


3. In step 4, they have to tell Glance if they're using the direct/indirect 
(currently named glance-direct and swift-local respectively) and then give it 
information (retrieved from step 1) about what the target formats are for that 
cloud



Again, I believe we could make target_* fields optional in the
*import* call as I (the user) most of the time don't care about the
final format of my image as long as it boots. Perhaps, the most
critical information in call #1 is the one related to the srouce_*
fields as that's something the user must respect. Not passing the
target_* fields in the import call will let the cloud use whatever it
has as default target format, which can be communicated through call
#1.

[snip just want to focus on the proposal]


So what if we modeled this a little

1. The user creates an image record (image-create)
2. The user uses the link in the response to upload the image data (we don't 
care whether it is direct or not because the user can follow it without 
thinking about it)
3. The user sends a request to process the image without having to specify what 

[openstack-dev] [glance] PTL candidacy

[stuart . mclaren]

Hi everyone,

I'd like to be considered for Glance PTL.

First: thanks (again) to Flavio for his leadership over the last cycle.

Glance is in pretty good shape right now: we have a coherent team which
is focussed on a well defined set of priorities. It's a relatively small
team, but we're fortunate to have a good range of strengths; every one
brings something valuable.

I've been contributing to Glance since Essex. I've probably touched
most areas of the code at some point since then, and I hope I've built
up a good understanding of the code base, the thinking behind most of it
... and the quirks too. I can't claim to know all the OpenStack code base
(who can?) but I've good relationships with folks from a bunch of other
projects I can reach out to, especially the ones Glance interacts with
(Keystone, Swift, Cinder etc).

For several years I was also an operator. Starting from Diablo (back when
we were still using integers, not uuids) I was responsible for running
Glance in HP's Public Cloud (24x7x365). That gave me a good understanding
of the issues operators face and, where I can, I've contributed back
based on my experience.

As a developer and operator my goal has been the same: to give the best
possible experience for users.

If elected I will be able to devote myself full time to the project,
be available on IRC, and hope to increase my own review rate.

Priorities for the Newton cycle:

* Image import refactor

We need to do everything we can to ensure DefCore can include an image
upload capability.

* v2 API adoption

We need to make it possible to deploy OpenStack without enabling Glance's
v1 API. There were some heroic efforts on the Nova side of this during
Mitaka (thanks Mike).

* Maintain focus.

Yes, we will work on other things during the cycle. But we shouldn't
commit to anything that could impact on our priorities. It's ok to say
"no" now and again. There will be other cycles.

Thanks for your consideration.

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Block subtractive schema changes

[stuart . mclaren]

I think this makes sense (helps us spot things which could impact upgrade).


Hi Glance Team,

I have registered a blueprint [1] for blocking subtractive schema changes.
Cinder and Nova are already supporting blocking of subtractive schema 
operations. Would like to add similar support here.

Please let me know your opinion on the same.

[1] https://blueprints.launchpad.net/glance/+spec/block-subtractive-operations


Thank you,

Abhishek Kekane


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] Hiatus

[stuart . mclaren]

Hi,

A change in focus means I'm going to be less involved upstream for the
next while.

I realise the timing isn't great for the DefCore image import stuff.
I'm hoping someone else may be able to pick up where I've left off [1].

Thanks,

-Stuart

[1] https://review.openstack.org/#/c/312972,
https://review.openstack.org/#/c/312653,
https://review.openstack.org/#/c/308386

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] Image import: operator plugins

[stuart . mclaren]

Hi,

As part of Glance's image import [1] work we need to define how operators
can run site specific code as part of the import process. For example
an operator may want to perform some site-specific validation on the
image bits.

Note that I'm not so much interested in what we use to do this (ie
we don't need to discuss the merits of taskflow/stevedore). Rather I'm
just interested in what level of abstraction we should provide.

Currently, the closest thing we have to a plugin [2] provides no
abstraction at all. You need to understand the Glance internals to use
it, and are not insulated from any future changes in Glance's internals.
You can do things like update the image state directly (possibly in a racy
or invalid way, leading to a poor experience for users) in your plugin.

To what degree are the following true?

1. An operator should not have to become intimitely familiar with
internal Glance code in order to write a plugin.
2. The plugin should be as insulated as possible from future changes in
Glance's internals.
3. The plugin should have very limited access to what it can touch. It
should not be possible for plugins to cause race conditions by
setting the image state for example. (This is related to 1 and 2).
4. The interface should just require instantiating a class.  We shouldn't
require plugin writers to know implementation details eg how glance logs
(instead, provide a log method), or whether or not we use taskflow.

To give an example of an interface of this kind, we could just provide
an image data iterator to the plugin class. Obviously this is a really
incomplete example, but gives an idea of a simple interface which would
allow a plugin to read the image data and perform validation without
knowing/being exposed to Glance's internals.

(Note: I'd like to keep the scope of this thread to what kind of
interface/contract a plugin should have, not what we use to implement
it, whether we should be doing this at all etc)

Thanks,

-Stuart

[1] https://review.openstack.org/#/c/232371
[2] https://review.openstack.org/#/c/214810/26/glance/async/flows/ovf_process.py

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance][Heat][Horizon] Glance v2 and custom locations

[stuart . mclaren]

Hi Mike,

Here's my $0.02 on setting locations directly:

I think there will always be some (probably public cloud) operators who
are wary of allowing regular users to do this. For this reason,
and also because the API calls are backend specific (setting location
isn't supported for a filesystem backend [1]), my guess is it's unlikely that
API calls for setting locations directly will become part of DefCore.

In general I'm not a huge fan of setting the image locations directly 
because:


* Any time you allow setting the location directly you're risking the
database and the image data getting out of sync. Eg an 'active' image's
data can be deleted from under it.
* Some ways of setting the locations directly give you images without
a known checksum.

In my mind, the feature is probably best restricted (by default at least)
to power users (admins) and other services -- eg Cinder can do this
to reduce copying data around in some cases. Existing policies can be
used to restrict to admin; something like service tokens [1] could be
used to allow other services (eg Cinder) to do this on behalf of users,
while preventing users from doing it directly themselves.

To keep things as inter-operable as possible it might be good to ensure
that services are able to use Glance even if setting locations directly
is completely disabled.

-Stuart

[1] https://review.openstack.org/#/c/141706
[2] 
https://specs.openstack.org/openstack/keystone-specs/specs/keystonemiddleware/implemented/service-tokens.html


Hello!

As you may know glance v1 is going to be deprecated in Newton cycle. Almost
all projects support glance v2 at this moment, Nova uses it by default.
Only one thing that blocks us from complete adoption is a possibility to
set custom locations to images. In v1 any user can set a location to his
image, but in v2 this functionality is not allowed by default, which
prevents v2 adoption in services like Horizon or Heat.

It all happens because of differences between v1 and v2 locations. In v1 it
is pretty easy - user specifies an url and send a request, glance adds this
url to the image and activates it.
In v2 things are more complicated: v2 supports multiple locations per
image, which means that when user wants to download image file glance will
choose the best one from the list of locations. It leads to some
inconsistencies: user can add or delete locations from his image even if it
is active.

To enable adding custom locations operator has to set True to config option
'show_multiple_locations'. After that any user will be able to add or
remove his image locations, update locations metadata, and finally see
locations of all images even if they were uploaded to local storage. All
this things are not desired if glance v2 has public interface, because it
exposes inner cloud architecture. It leads to the fact that Heat and
Horizon and Nova in some cases and other services that used to set custom
locations in glance v1 won't be able to adopt glance v2. Unfortunately,
removing this behavior in v2 isn't easy, because it requires serious
architecture changes and breaks API. Moreover, many vendors use these
features in their clouds for private glance deployments and they really
won't like if we break anything.

So, I want to hear opinions from Glance community and other involved 
people.


Best regards,
Mikhail Fedosin
-- next part --
An HTML attachment was scrubbed...
URL: 



--



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance][Glare] External locations design

[stuart . mclaren]

Hi Kairat,

I think it's great to try and tease through the various issues here.
I added some comments to the etherpad.

-Stuart


Hello all,

I would like to start to describe some design decisions we made in Glare
code (https://review.openstack.org/#/q/topic:bp/glare-api+status:open).  If
you are not familiar with Glare I suggest you to read the following spec:

https://github.com/openstack/glance-specs/blob/master/specs/newton/approved/glance/glare-api.rst

I hope it will help other folks to understand Glare approach and provide
some constructive feedback for Glare. I think that we can also use Glare
solution for Glance in near future to address some drawbacks we have in
Glance.

Glare locations

Glance and Glare have possibility to set some external url as
image(artifact) location. This feature is quite useful for users who would
like to refer to some external image or artifact (for example, Fedora image
on official Fedora site) and not to store this image or artifact in the
cloud.

External locations in Glance have several specialities:

  1.

  It is possible to setup multiple locations for an image. Glance uses
  special location strategy to define which location to use. This strategy
  defined in glance codebase and can be configured in glance conf.
  2.

  Glance doesn?t differ image locations specified by url and image
  locations uploaded to Glance backend. Glance has some restrictions about
  which urls to use for locations (see Glance docs for more info).


Glare external locations designed in different way to address some
drawbacks we have in Glance. So the approach is the following:

  1.

  Glare doesn?t support multiple locations, you can specify dict of blobs
  in artifact type and add url for each blob in dict. User must define a
  name(f.e. region name or priority) for blob in dict and this name can be
  used to retrieve this blob from artifact. So decision about which location
  to use will be outside of Glare.
  2.

  Glare adds a special flag to database for external locations. So they
  will be treated differently in Glare when delete artifact. If blob value is
  external url then we don?t need to pass this url to backend and just delete
  the record in DB. For now, Glare allows only http(s) locations set but it
  may be extended in future but the idea still the same: external location
  are just records in DB.
  3.

  Glare saves blob size and checksum when specifying external url. When
  user specified url Glare downloads the blob by url, calculates its size and
  checksum. Of course, it leads to some performance degradation but we can
  ensure that the external blob is immutable. We made this because security
  seems more important for Glare than performance. Also there are plans to
  extend this approach to support subscriptions for external locations so we
  can increase secureness of that operation.


I think that some of the features above can be implemented in Glance. For
example, we can treat our locations as only read-only links if external
flag will be implemented.  It will allow us to ensure that only blobs
uploaded through the Glance will be managed.

Additionally, if we will calculate checksum and size for external urls, we
can ensure that  all multiple locations refers to the same blob. So
management of multiple locations(deletion/creation) can be more secure.
Also we can ensure that the external url blob was not changed.

I understand that we need a spec for that but I would like to discuss this
at high level first. Here is etherpad for discussion:
https://etherpad.openstack.org/p/glare-locations


Best regards,
Kairat Kushaev
-- next part --
An HTML attachment was scrubbed...
URL: 


--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


End of OpenStack-dev Digest, Vol 52, Issue 2




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance][TC][Heat][App-Catalog][Murano][Tacker] Glare as a new Project

[stuart . mclaren]

I think this makes sense.

"Should Artifacts be part of Glance?" is something folks have been
debating for several years now, with seemingly equal numbers on
each side.

Even though it was far from unanimous, several summits ago it was decided
to aim for a a v3 Glance API which would support all artifact types,
including images. That turned out to be too ambitious for a project that
really was starting at times to operate close to maintenance mode. (
eg I haven't contributed at all this cycle). The realisation that
V3 wasn't going to happen in our lifetime, difficulties mapping between
V3 and V2 (and V1!) calls, and (IIRC) DefCore feedback, meant V3 was
abandoned: we'd have two separate APIs. That's actually a decision that
was taken a while ago, and isn't new.

As a previously stressed out operator (is there any other kind?) I
understand that anything that means you have more work to do is painful,
and should be considered a negative. But it shouldn't be the only
consideration. For me, moving a distinct API to its own project makes
the most longterm sense.

Thanks to Mike, Alex, Kairat et al for their (hopefully continuing)
Glance contributions and patience.

-Stuart


Hi all,
after 6 months of Glare v1 API development we have decided to continue our
work in a separate project in the "openstack" namespace with its own core
team (me, Kairat Kushaev, Darja Shkhray and the original creator -
Alexander Tivelkov). We want to thank Glance community for their support
during the incubation period, valuable advice and suggestions - this time
was really productive for us. I believe that this step will allow the Glare
project to concentrate on feature development and move forward faster.
Having the independent service also removes inconsistencies in
understanding what Glance project is: it seems that a single project cannot
own two different APIs with partially overlapping functionality. So with
the separation of Glare into a new project, Glance may continue its work on
the OpenStack Images API, while Glare will become the reference
implementation of the new OpenStack Artifacts API.

Nevertheless, Glare team would like to continue to collaborate with the
Glance team in a new - cross-project - format. We still have lots in
common, both in code and usage scenarios, so we are looking forward for
fruitful work with the rest of the Glance team. Those of you guys who are
interested in Glare and the future of Artifacts API are also welcome to
join the Glare team: we have a lot of really exciting tasks and will always
welcome new members.
Meanwhile, despite the fact that my focus will be on the new project, I
will continue to be part of the Glance team and for sure I'm going to
contribute in Glance, because I am interested in this project and want to
help it be successful.

We'll have the formal patches pushed to project-config earlier next week,
appropriate repositories, wiki and launchpad space will be created soon as
well.  Our regular weekly IRC meeting remains intact: it is 17:30 UTC
Mondays in #openstack-meeting-alt, it will just become a Glare project
meeting instead of a Glare sub-team meeting. Please feel free to join!

Best regards,
Mikhail Fedosin

P.S. For those of you who may be curious on the project name. We'll still
be called "Glare", but since we are on our own now this acronym becomes
recursive: GLARE now stands for "GLare Artifact REpository" :)
-- next part --
An HTML attachment was scrubbed...
URL: 


--


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev