Re: [openstack-dev] Where should Schema files live?

2014-12-08 Thread Sandy Walsh

From: Sandy Walsh [sandy.wa...@rackspace.com] Monday, December 01, 2014 9:29 AM

From: Duncan Thomas [duncan.tho...@gmail.com]
Sent: Sunday, November 30, 2014 5:40 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Where should Schema files live?

Duncan Thomas
On Nov 27, 2014 10:32 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:

 We were thinking each service API would expose their schema via a new 
 /schema resource (or something). Nova would expose its schema. Glance its 
 own. etc. This would also work well for installations still using older 
 deployments.
This feels like externally exposing info that need not be external (since the 
notifications are not external to the deploy) and it sounds like it will 
potentially leak fine detailed version and maybe deployment config details 
that you don't want to make public - either for commercial reasons or to make 
targeted attacks harder


Yep, good point. Makes a good case for standing up our own service or just 
relying on the tarballs being in a well know place.

Hmm, I wonder if it makes sense to limit the /schema resource to service 
accounts. Expose it by role.

There's something in the back of my head that doesn't like calling out to the 
public API though. Perhaps unfounded.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-12-08 Thread Eoghan Glynn


 From: Sandy Walsh [sandy.wa...@rackspace.com] Monday, December 01, 2014 9:29
 AM
  
 From: Duncan Thomas [duncan.tho...@gmail.com]
 Sent: Sunday, November 30, 2014 5:40 AM
 To: OpenStack Development Mailing List
 Subject: Re: [openstack-dev] Where should Schema files live?
  
 Duncan Thomas
 On Nov 27, 2014 10:32 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:
  
  We were thinking each service API would expose their schema via a new
  /schema resource (or something). Nova would expose its schema. Glance
  its own. etc. This would also work well for installations still using
  older deployments.
 This feels like externally exposing info that need not be external (since
 the notifications are not external to the deploy) and it sounds like it
 will potentially leak fine detailed version and maybe deployment config
 details that you don't want to make public - either for commercial reasons
 or to make targeted attacks harder
  
  
 Yep, good point. Makes a good case for standing up our own service or just
 relying on the tarballs being in a well know place.
 
 Hmm, I wonder if it makes sense to limit the /schema resource to service
 accounts. Expose it by role.
 
 There's something in the back of my head that doesn't like calling out to the
 public API though. Perhaps unfounded.

I'm wondering here how this relates to the other URLs in the
service catalog that aren't intended for external consumption,
e.g. the internalURL and adminURL.

I had assumed that these URLs would be visible to external clients,
but protected by firewall rules such that clients would be unable
to do anything in anger with those raw addresses from the outside.

So would including a schemaURL in the service catalog actually
expose an attack surface, assuming this was in general safely
firewalled off in any realistic deployment?

Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-12-08 Thread Adam Young
Isn't this what the API repos are for?  Should EG the Keystone schemes 
be served from


https://github.com/openstack/identity-api/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-12-08 Thread Lance Bragstad
Keystone also has API documentation in the keystone-spec repo [1], which
went in with [2] and [3].

[1] https://github.com/openstack/keystone-specs/tree/master/api
[2] https://review.openstack.org/#/c/128712/
[3] https://review.openstack.org/#/c/130577/

On Mon, Dec 8, 2014 at 1:06 PM, Adam Young ayo...@redhat.com wrote:

 Isn't this what the API repos are for?  Should EG the Keystone schemes be
 served from

 https://github.com/openstack/identity-api/


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-12-01 Thread Sandy Walsh

From: Duncan Thomas [duncan.tho...@gmail.com]
Sent: Sunday, November 30, 2014 5:40 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Where should Schema files live?

Duncan Thomas
On Nov 27, 2014 10:32 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:

 We were thinking each service API would expose their schema via a new 
 /schema resource (or something). Nova would expose its schema. Glance its 
 own. etc. This would also work well for installations still using older 
 deployments.
This feels like externally exposing info that need not be external (since the 
notifications are not external to the deploy) and it sounds like it will 
potentially leak fine detailed version and maybe deployment config details 
that you don't want to make public - either for commercial reasons or to make 
targeted attacks harder


Yep, good point. Makes a good case for standing up our own service or just 
relying on the tarballs being in a well know place.

Thanks for the feedback.


-S

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-30 Thread Duncan Thomas
Duncan Thomas
On Nov 27, 2014 10:32 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:


 We were thinking each service API would expose their schema via a new
/schema resource (or something). Nova would expose its schema. Glance its
own. etc. This would also work well for installations still using older
deployments.

This feels like externally exposing info that need not be external (since
the notifications are not external to the deploy) and it sounds like it
will potentially leak fine detailed version and maybe deployment config
details that you don't want to make public - either for commercial reasons
or to make targeted attacks harder
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-26 Thread Jay Pipes

On 11/20/2014 08:12 AM, Sandy Walsh wrote:

Hey y'all,

To avoid cross-posting, please inform your -infra / -operations buddies about 
this post.

We've just started thinking about where notification schema files should live 
and how they should be deployed. Kind of a tricky problem.  We could really use 
your input on this problem ...

The assumptions:
1. Schema files will be text files. They'll live in their own git repo 
(stackforge for now, ideally oslo eventually).
2. Unit tests will need access to these files for local dev
3. Gating tests will need access to these files for integration tests
4. Many different services are going to want to access these files during 
staging and production.
5. There are going to be many different versions of these files. There are 
going to be a lot of schema updates.

Some problems / options:
a. Unlike Python, there is no simple pip install for text files. No version 
control per se. Basically whatever we pull from the repo. The problem with a 
git clone is we need to tweak config files to point to a directory and that's a 
pain for gating tests and CD. Could we assume a symlink to some well-known 
location?
 a': I suppose we could make a python installer for them, but that's a pain 
for other language consumers.
b. In production, each openstack service could expose the schema files via 
their REST API, but that doesn't help gating tests or unit tests. Also, this 
means every service will need to support exposing schema files. Big 
coordination problem.
c. In production, We could add an endpoint to the Keystone Service Catalog to 
each schema file. This could come from a separate metadata-like service. Again, 
yet-another-service to deploy and make highly available.
d. Should we make separate distro packages? Install to a well known location 
all the time? This would work for local dev and integration testing and we 
could fall back on B and C for production distribution. Of course, this will 
likely require people to add a new distro repo. Is that a concern?

Personally, I'm leaning towards option D but I'm not sure what the implications 
are.

We're early in thinking about these problems, but would like to start the 
conversation now to get your opinions.

Look forward to your feedback.


OK, so the goal of this effort should be to have a single OpenStack 
standard for what the payload and structure of notification messages 
will look like. That means, to me at least, that these schema files should:


 a) Live in a single repo in the openstack/ code namespace 
(openstack/notification-schemas?)


 b) Be published to an openstack.org subdomain, served by some static 
web server for all the world to read and/or mirror


Let clients and servers that need to read and write these messages 
download the schemas as-needed.


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-25 Thread Eoghan Glynn


 I think Doug's suggestion of keeping the schema files in-tree and pushing
 them to a well-known tarball maker in a build step is best so far.
 
 It's still a little clunky, but not as clunky as having to sync two repos.

Yes, I tend to agree.

So just to confirm that my understanding lines up:

* the tarball would be used by the consumer-side for unit tests and
  limited functional tests (where the emitter service is not running)

* the tarball would be also be used by the consumer-side in DSVM-based
  CI and in a full production deployments (where the emitter service is
  running)

* the tarballs will be versioned, with old versions remaining accessible
  (as per the current practice with released source on tarballs.openstack.org)

* the consumer side will know which version of each schema it expects to
  support, and will download the appropriate tarball at runtime

* the emitter side will signal the schema version that's it actually using,
  via say a well-known field in the notification body

* the consumer will reject notification payloads with a mismatched major
  version to what it's expecting to support
 
 [snip]
   d. Should we make separate distro packages? Install to a well known
   location all the time? This would work for local dev and integration
   testing and we could fall back on B and C for production distribution.
   Of
   course, this will likely require people to add a new distro repo. Is
   that
   a concern?
 
  Quick clarification ... when you say distro packages, do you mean
  Linux-distro-specific package formats such as .rpm or .deb?
 
  Yep.
 
 So that would indeed work, but just to sound a small note of caution
 that keeping an oft-changing package (assumption #5) up-to-date for
 fedora20/21  epel6/7, or precise/trusty, would involve some work.
 
 I don't know much about the Debian/Ubuntu packaging pipeline, in
 particular how it could be automated.
 
 But in my small experience of Fedora/EL packaging, the process is
 somewhat resistant to many fine-grained updates.
 
 Ah, good to know. So, if we go with the tarball approach, we should be able
 to avoid this. And it allows the service to easily service up the schema
 using their existing REST API.

I'm not clear on how servicing up the schema via an existing API would
avoid the co-ordination issue identified in the original option (b)?

Would that API just be a very simple proxying in front of the well-known
source of these tarballs? 

For production deployments, is it likely that some shops will not want
to require access to an external site such as tarballs.openstack.org?

So in that case, would we require that they mirror, or just assume that
downstream packagers will bundle the appropriate schema versions with
the packages for the emitter and consumer services?

Cheers,
Eoghan

 Should we proceed under the assumption we'll push to a tarball in a
 post-build step? It could change if we find it's too messy.
 
 -S
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-24 Thread Sandy Walsh
From: Eoghan Glynn [egl...@redhat.com] Friday, November 21, 2014 11:03 AM
  Some problems / options:
  a. Unlike Python, there is no simple pip install for text files. No
  version control per se. Basically whatever we pull from the repo. The
  problem with a git clone is we need to tweak config files to point to a
  directory and that's a pain for gating tests and CD. Could we assume a
  symlink to some well-known location?
  a': I suppose we could make a python installer for them, but that's a
  pain for other language consumers.

 Would it be unfair to push that burden onto the writers of clients
 in other languages?
 
 i.e. OpenStack, being largely python-centric, would take responsibility
 for both:
 
   1. Maintaining the text versions of the schema in-tree (e.g. as json)
 
 and:
 
   2. Producing a python-specific installer based on #1
 
 whereas, the first Java-based consumer of these schema would take
 #1 and package it up in their native format, i.e. as a jar or
 OSGi bundle.

I think Doug's suggestion of keeping the schema files in-tree and pushing them 
to a well-known tarball maker in a build step is best so far. 

It's still a little clunky, but not as clunky as having to sync two repos. 

[snip]
  d. Should we make separate distro packages? Install to a well known
  location all the time? This would work for local dev and integration
  testing and we could fall back on B and C for production distribution. Of
  course, this will likely require people to add a new distro repo. Is that
  a concern?

 Quick clarification ... when you say distro packages, do you mean
 Linux-distro-specific package formats such as .rpm or .deb?

 Yep.

So that would indeed work, but just to sound a small note of caution
that keeping an oft-changing package (assumption #5) up-to-date for
fedora20/21  epel6/7, or precise/trusty, would involve some work.

I don't know much about the Debian/Ubuntu packaging pipeline, in
particular how it could be automated.

But in my small experience of Fedora/EL packaging, the process is
somewhat resistant to many fine-grained updates.

Ah, good to know. So, if we go with the tarball approach, we should be able to 
avoid this. And it allows the service to easily service up the schema using 
their existing REST API. 

Should we proceed under the assumption we'll push to a tarball in a post-build 
step? It could change if we find it's too messy. 

-S

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Sandy Walsh

From: Eoghan Glynn [egl...@redhat.com] Thursday, November 20, 2014 5:34 PM

Some questions/observations inline.

 Hey y'all,

 To avoid cross-posting, please inform your -infra / -operations buddies 
 about this post.

 We've just started thinking about where notification schema files should 
 live and how they should be deployed. Kind of a tricky problem.  We could 
 really use your input on this problem ...

 The assumptions:
 1. Schema files will be text files. They'll live in their own git repo 
 (stackforge for now, ideally oslo eventually).
 2. Unit tests will need access to these files for local dev
 3. Gating tests will need access to these files for integration tests
 4. Many different services are going to want to access these files during 
 staging and production.
 5. There are going to be many different versions of these files. There are 
 going to be a lot of schema updates.

 Some problems / options:
 a. Unlike Python, there is no simple pip install for text files. No version 
 control per se. Basically whatever we pull from the repo. The problem with a 
 git clone is we need to tweak config files to point to a directory and 
 that's a pain for gating tests and CD. Could we assume a symlink to some 
 well-known location?
 a': I suppose we could make a python installer for them, but that's a 
 pain for other language consumers.

Would it be unfair to push that burden onto the writers of clients
in other languages?

i.e. OpenStack, being largely python-centric, would take responsibility
for both:

  1. Maintaining the text versions of the schema in-tree (e.g. as json)

and:

  2. Producing a python-specific installer based on #1

whereas, the first Java-based consumer of these schema would take
#1 and package it up in their native format, i.e. as a jar or
OSGi bundle.

Certainly an option. My gut says it will lead to abandoned/fragmented efforts.
If I was a ruby developer, would I want to take on the burden of maintaining 
yet another package? 
I think we need to treat this data as a form of API and there it's our 
responsibility to make easily consumable. 

(I'm not hard-line on this, again, just my gut feeling)

 b. In production, each openstack service could expose the schema files via 
 their REST API, but that doesn't help gating tests or unit tests. Also, this 
 means every service will need to support exposing schema files. Big 
 coordination problem.

I kind of liked this schemaURL endpoint idea when it was first
mooted at summit.

The attraction for me was that it would allow the consumer of the
notifications always have access to the actual version of schema
currently used on the emitter side, independent of the (possibly
out-of-date) version of the schema that the consumer has itself
installed locally via a static dependency.

However IIRC there were also concerns expressed about the churn
during some future rolling upgrades - i.e. if some instances of
the nova-api schemaURL endpoint are still serving out the old
schema, after others in the same deployment have already been
updated to emit the new notification version.

Yeah, I like this idea too. In the production / staging phase this seems like 
the best route. The local dev / testing situation seems to be the real tough 
nut to crack. 

WRT rolling upgrades we have to ensure we update the service catalog first, the 
rest should be fine. 

 c. In production, We could add an endpoint to the Keystone Service Catalog 
 to each schema file. This could come from a separate metadata-like service. 
 Again, yet-another-service to deploy and make highly available.

Also to {puppetize|chef|ansible|...}-ize.

Yeah, agreed, we probably don't want to do down that road.

Which is kinda unfortunate since it's the lowest impact on other projects. 

 d. Should we make separate distro packages? Install to a well known location 
 all the time? This would work for local dev and integration testing and we 
 could fall back on B and C for production distribution. Of course, this will 
 likely require people to add a new distro repo. Is that a concern?

Quick clarification ... when you say distro packages, do you mean
Linux-distro-specific package formats such as .rpm or .deb?

Yep. 

Cheers,
Eoghan

Thanks for the feedback!
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Sandy Walsh
From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 5:09 PM
On Nov 20, 2014, at 3:40 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:
 From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 3:51 
 PM
 On Nov 20, 2014, at 8:12 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:
 The assumptions:
 1. Schema files will be text files. They'll live in their own git repo 
 (stackforge for now, ideally oslo eventually).
 Why wouldn’t they live in the repo of the application that generates the 
 notification, like we do with the database schema and APIs defined by those 
 apps?

 That would mean downstream consumers (potentially in different languages) 
 would need to pull all repos and extract just the schema parts. A separate 
 repo would make it more accessible.

OK, fair. Could we address that by publishing the schemas for an app in a tar 
ball using a post merge job?

That's something to consider. At first blush it feels a little clunky to pull 
all projects to extract schemas whenever any of the projects change. 

But there is something to be said about having the schema files next to the 
code that going to generate the data. 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Doug Hellmann

On Nov 21, 2014, at 9:11 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:

 From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 5:09 
 PM
 On Nov 20, 2014, at 3:40 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:
 From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 
 3:51 PM
 On Nov 20, 2014, at 8:12 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:
 The assumptions:
 1. Schema files will be text files. They'll live in their own git repo 
 (stackforge for now, ideally oslo eventually).
 Why wouldn’t they live in the repo of the application that generates the 
 notification, like we do with the database schema and APIs defined by 
 those apps?
 
 That would mean downstream consumers (potentially in different languages) 
 would need to pull all repos and extract just the schema parts. A separate 
 repo would make it more accessible.
 
 OK, fair. Could we address that by publishing the schemas for an app in a 
 tar ball using a post merge job?
 
 That's something to consider. At first blush it feels a little clunky to pull 
 all projects to extract schemas whenever any of the projects change. 

Oh, that’s not what I meant. I meant that there would be a separate package for 
each project’s schema(s). We could have a meta package that we build, too, but 
I was thinking of bundling them separately.

 
 But there is something to be said about having the schema files next to the 
 code that going to generate the data. 

It feels like it would be easier to keep them in sync that way. It would let us 
gate on that being the case, for example, without having to stage commits in 
multiple repositories.

 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Eoghan Glynn


  Why wouldn’t they live in the repo of the application that generates the
  notification, like we do with the database schema and APIs defined by
  those apps?
 
  That would mean downstream consumers (potentially in different languages)
  would need to pull all repos and extract just the schema parts. A
  separate repo would make it more accessible.
 
 OK, fair. Could we address that by publishing the schemas for an app in a
 tar ball using a post merge job?
 
 That's something to consider. At first blush it feels a little clunky to pull
 all projects to extract schemas whenever any of the projects change.
 
 But there is something to be said about having the schema files next to the
 code that going to generate the data.

My initial read of Doug's proposal was for a tarball of the project's schemas
to be published somewhere out-of-tree, e.g. to tarballs.openstack.org, via a
post-merge git hook or some-such.

Not 100% sure that's a correct interpretation of the proposal, but it would
avoid the need for the consumer projects to pull the repos of the emitter
projects. 

Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Eoghan Glynn


  Some problems / options:
  a. Unlike Python, there is no simple pip install for text files. No
  version control per se. Basically whatever we pull from the repo. The
  problem with a git clone is we need to tweak config files to point to a
  directory and that's a pain for gating tests and CD. Could we assume a
  symlink to some well-known location?
  a': I suppose we could make a python installer for them, but that's a
  pain for other language consumers.
 
 Would it be unfair to push that burden onto the writers of clients
 in other languages?
 
 i.e. OpenStack, being largely python-centric, would take responsibility
 for both:
 
   1. Maintaining the text versions of the schema in-tree (e.g. as json)
 
 and:
 
   2. Producing a python-specific installer based on #1
 
 whereas, the first Java-based consumer of these schema would take
 #1 and package it up in their native format, i.e. as a jar or
 OSGi bundle.
 
 Certainly an option. My gut says it will lead to abandoned/fragmented
 efforts.
 If I was a ruby developer, would I want to take on the burden of maintaining
 yet another package?
 I think we need to treat this data as a form of API and there it's our
 responsibility to make easily consumable.
 
 (I'm not hard-line on this, again, just my gut feeling)

OK, that's fair.

[snip]
  d. Should we make separate distro packages? Install to a well known
  location all the time? This would work for local dev and integration
  testing and we could fall back on B and C for production distribution. Of
  course, this will likely require people to add a new distro repo. Is that
  a concern?
 
 Quick clarification ... when you say distro packages, do you mean
 Linux-distro-specific package formats such as .rpm or .deb?
 
 Yep.

So that would indeed work, but just to sound a small note of caution
that keeping an oft-changing package (assumption #5) up-to-date for
fedora20/21  epel6/7, or precise/trusty, would involve some work.

I don't know much about the Debian/Ubuntu packaging pipeline, in
particular how it could be automated.

But in my small experience of Fedora/EL packaging, the process is
somewhat resistant to many fine-grained updates.

Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-20 Thread Doug Hellmann

On Nov 20, 2014, at 8:12 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:

 Hey y'all,
 
 To avoid cross-posting, please inform your -infra / -operations buddies about 
 this post. 
 
 We've just started thinking about where notification schema files should live 
 and how they should be deployed. Kind of a tricky problem.  We could really 
 use your input on this problem ...
 
 The assumptions:
 1. Schema files will be text files. They'll live in their own git repo 
 (stackforge for now, ideally oslo eventually). 

Why wouldn’t they live in the repo of the application that generates the 
notification, like we do with the database schema and APIs defined by those 
apps?

 2. Unit tests will need access to these files for local dev
 3. Gating tests will need access to these files for integration tests
 4. Many different services are going to want to access these files during 
 staging and production. 
 5. There are going to be many different versions of these files. There are 
 going to be a lot of schema updates. 
 
 Some problems / options:
 a. Unlike Python, there is no simple pip install for text files. No version 
 control per se. Basically whatever we pull from the repo. The problem with a 
 git clone is we need to tweak config files to point to a directory and that's 
 a pain for gating tests and CD. Could we assume a symlink to some well-known 
 location?
a': I suppose we could make a python installer for them, but that's a pain 
 for other language consumers.
 b. In production, each openstack service could expose the schema files via 
 their REST API, but that doesn't help gating tests or unit tests. Also, this 
 means every service will need to support exposing schema files. Big 
 coordination problem.
 c. In production, We could add an endpoint to the Keystone Service Catalog to 
 each schema file. This could come from a separate metadata-like service. 
 Again, yet-another-service to deploy and make highly available. 
 d. Should we make separate distro packages? Install to a well known location 
 all the time? This would work for local dev and integration testing and we 
 could fall back on B and C for production distribution. Of course, this will 
 likely require people to add a new distro repo. Is that a concern?
 
 Personally, I'm leaning towards option D but I'm not sure what the 
 implications are. 
 
 We're early in thinking about these problems, but would like to start the 
 conversation now to get your opinions. 
 
 Look forward to your feedback.
 
 Thanks
 -Sandy
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-20 Thread Sandy Walsh
From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 3:51 PM
   On Nov 20, 2014, at 8:12 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:
Hey y'all,
   
To avoid cross-posting, please inform your -infra / -operations buddies 
about this post.
   
We've just started thinking about where notification schema files should 
live and how they should be deployed. Kind of a tricky problem.  We could 
really use your input on this problem ...
   
The assumptions:
1. Schema files will be text files. They'll live in their own git repo 
(stackforge for now, ideally oslo eventually).
   Why wouldn’t they live in the repo of the application that generates the 
notification, like we do with the database schema and APIs defined by those 
apps?

That would mean downstream consumers (potentially in different languages) would 
need to pull all repos and extract just the schema parts. A separate repo would 
make it more accessible. 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-20 Thread Doug Hellmann

On Nov 20, 2014, at 3:40 PM, Sandy Walsh sandy.wa...@rackspace.com wrote:

 From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 3:51 
 PM
 On Nov 20, 2014, at 8:12 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:
 Hey y'all,
 
 To avoid cross-posting, please inform your -infra / -operations buddies 
 about this post.
 
 We've just started thinking about where notification schema files should 
 live and how they should be deployed. Kind of a tricky problem.  We could 
 really use your input on this problem ...
 
 The assumptions:
 1. Schema files will be text files. They'll live in their own git repo 
 (stackforge for now, ideally oslo eventually).
 Why wouldn’t they live in the repo of the application that generates the 
 notification, like we do with the database schema and APIs defined by those 
 apps?
 
 That would mean downstream consumers (potentially in different languages) 
 would need to pull all repos and extract just the schema parts. A separate 
 repo would make it more accessible. 

OK, fair. Could we address that by publishing the schemas for an app in a tar 
ball using a post merge job?

Doug

 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should Schema files live?

2014-11-20 Thread Eoghan Glynn

Thanks for raising this Sandy,

Some questions/observations inline.

 Hey y'all,
 
 To avoid cross-posting, please inform your -infra / -operations buddies about 
 this post. 
 
 We've just started thinking about where notification schema files should live 
 and how they should be deployed. Kind of a tricky problem.  We could really 
 use your input on this problem ...
 
 The assumptions:
 1. Schema files will be text files. They'll live in their own git repo 
 (stackforge for now, ideally oslo eventually). 
 2. Unit tests will need access to these files for local dev
 3. Gating tests will need access to these files for integration tests
 4. Many different services are going to want to access these files during 
 staging and production. 
 5. There are going to be many different versions of these files. There are 
 going to be a lot of schema updates. 
 
 Some problems / options:
 a. Unlike Python, there is no simple pip install for text files. No version 
 control per se. Basically whatever we pull from the repo. The problem with a 
 git clone is we need to tweak config files to point to a directory and that's 
 a pain for gating tests and CD. Could we assume a symlink to some well-known 
 location?
 a': I suppose we could make a python installer for them, but that's a 
 pain for other language consumers.

Would it be unfair to push that burden onto the writers of clients
in other languages?

i.e. OpenStack, being largely python-centric, would take responsibility
for both:

  1. Maintaining the text versions of the schema in-tree (e.g. as json)

and:

  2. Producing a python-specific installer based on #1

whereas, the first Java-based consumer of these schema would take
#1 and package it up in their native format, i.e. as a jar or
OSGi bundle.

 b. In production, each openstack service could expose the schema files via 
 their REST API, but that doesn't help gating tests or unit tests. Also, this 
 means every service will need to support exposing schema files. Big 
 coordination problem.

I kind of liked this schemaURL endpoint idea when it was first
mooted at summit.

The attraction for me was that it would allow the consumer of the
notifications always have access to the actual version of schema
currently used on the emitter side, independent of the (possibly
out-of-date) version of the schema that the consumer has itself
installed locally via a static dependency.

However IIRC there were also concerns expressed about the churn
during some future rolling upgrades - i.e. if some instances of
the nova-api schemaURL endpoint are still serving out the old
schema, after others in the same deployment have already been
updated to emit the new notification version.

 c. In production, We could add an endpoint to the Keystone Service Catalog to 
 each schema file. This could come from a separate metadata-like service. 
 Again, yet-another-service to deploy and make highly available. 

Also to {puppetize|chef|ansible|...}-ize.

Yeah, agreed, we probably don't want to do down that road.

 d. Should we make separate distro packages? Install to a well known location 
 all the time? This would work for local dev and integration testing and we 
 could fall back on B and C for production distribution. Of course, this will 
 likely require people to add a new distro repo. Is that a concern?

Quick clarification ... when you say distro packages, do you mean 
Linux-distro-specific package formats such as .rpm or .deb?

Cheers,
Eoghan
 
 Personally, I'm leaning towards option D but I'm not sure what the 
 implications are. 
 
 We're early in thinking about these problems, but would like to start the 
 conversation now to get your opinions. 
 
 Look forward to your feedback.
 
 Thanks
 -Sandy
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev