date:20111027

Just to be clear we are talking about APIs fit for customer consumption 
here, not internal integrations where both ends are under our control.


On 10/27/2011 11:38 AM, George Reese wrote:

I disagree. The web was designed specifically to solve the distributed scaling 
problem and it's based on HTTP polling. It scales pretty well. The argument 
against polling not scaling inevitably neglects using caching properly.

The web was not designed to deal with a bunch of clients needing to
know about infrastructure changes the instant they happen.
Neither physics nor math were designed for that either. The CAP theorem 
simply doesn't allow a distributed system with an uptime guarantee to 
communicate changes "the instant they happen". Once you realize the best 
your clients can hope for is eventual consistency, the sooner you'll 
realize that polling is just fine.


BTW, here's Roy Fielding's article on this subject of poll vs push.
http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons

And API data should not be cached. The Rackspace API used to do that,
and it created a mess.
I'm not sure what you are referring to, but this is a classic strawman. 
Somebody implemented a "mess" using caching, so caching is bad!? You 
didn't say what the mess was, so there's no way to even evaluate your 
statement.

Push doesn't scaled because it requires the server to know about every client 
and track conversational state with them.

No, it doesn't. You push changes as they occur to a message queue. A
separate system tracks subscribers and sends them out. There is no
conversational state if done right.


A "separate" system? That's why you think it's simple -- you push the 
hard part outside of your box and claim victory. It's not a separate 
system, it's all one big cloud. If there are N interested clients the 
process you described requires O(N) resources. Moving it to another tier 
means it's somebody else's O(N) resources. You are illustrating 
Fielding's point in the article above: "People need to understand that 
general-purpose PubSub is not a solution to scalability problems — it 
simply moves the problem somewhere else, and usually to a place that is 
inversely supported by the economics. "


How exactly does this separate system know where to "send them out" to? 
Each client has to tell it and you have to store it and look it up on a 
per outbound message basis. And keep it accurate. Customers just love 
keeping you informed of where they want to send their messages. Do you 
know what happens when they forget to tell you they moved and they don't 
get the message? They blame you and ask for a credit memo. And do you 
know what happens when you tell them no. They go to your competitor. If 
there is no conversational state, then you aren't waiting for an 
acknowledgement from the other side for each message and you can't prove 
that it was delivered or even try again.



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Compute API schema docs now available

2011-10-27 Thread Anne Gentle

All,

You can now browse through the Compute API 1.1 schemas here:

http://docs.openstack.org/api/openstack-compute/1.1/xsd/

If other APIs would like to provide schemas, we can use the xslt transforms
to create these pages for browsing for other APIs. The source files are
located in http://github.com/openstack/compute-api/ repository.

I haven't figured out yet how to link to this from the
docs.openstack.org/api landing page but suggestions are welcome.

Thanks to David Cramer for debugging these to get them working.
Anne
*Anne Gentle*
a...@openstack.org
 my blog  | my
book|
LinkedIn  |
Delicious|
Twitter 
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone versioning and tarballs

2011-10-27 Thread Yun Mao

I think I'm close to figuring this out. You can take a look at the
devstack scripts. In particular,
https://github.com/cloudbuilders/devstack/blob/master/files/keystone_data.sh

Then you can source openrc to get the EC2_* environment variables.

However, it only works for euca-describe-instances,
euca-describe-images, at least for me.

When I tried euca-run-instances, the error is:
$ euca-run-instances ami-0004
Warning: failed to parse error message from AWS: :1:0: syntax error
None: None

The log on the nova-api daemon looks like this:
2011-10-27 18:29:22,288 DEBUG nova [-] HTTP PERF: 0.01362 seconds to
GET 127.0.0.1:35357 /v2.0/tokens/bd9c6abd-eeb4-4ba9-b49e-7aafe790ef9c)
from (pid=2774) getresponse
/opt/stack/keystone/keystone/common/bufferedhttp.py:99
2011-10-27 18:29:22,301 DEBUG nova [-] HTTP PERF: 0.01282 seconds to
GET 127.0.0.1:35357 /v2.0/tokens/bd9c6abd-eeb4-4ba9-b49e-7aafe790ef9c)
from (pid=2774) getresponse
/opt/stack/keystone/keystone/common/bufferedhttp.py:99
2011-10-27 18:29:22,302 DEBUG nova.api [-] action: RunInstances from
(pid=2774) __call__ /opt/stack/nova/nova/api/ec2/__init__.py:240
2011-10-27 18:29:22,302 DEBUG nova.api [-] arg: ImageId val:
ami-000 from (pid=2774) __call__
/opt/stack/nova/nova/api/ec2/__init__.py:242
2011-10-27 18:29:22,303 DEBUG nova.api [-] arg: MaxCount
 val: 1 from (pid=2774) __call__
/opt/stack/nova/nova/api/ec2/__init__.py:242
2011-10-27 18:29:22,303 DEBUG nova.api [-] arg: MinCount
 val: 1 from (pid=2774) __call__
/opt/stack/nova/nova/api/ec2/__init__.py:242
2011-10-27 18:29:22,303 DEBUG nova.api [-] arg: InstanceType
 val: m1.small from (pid=2774) __call__
/opt/stack/nova/nova/api/ec2/__init__.py:242
2011-10-27 18:29:22,303 AUDIT nova.api
[4f056dc4-6515-4bd0-bd09-0c1584b9fc39 demo 2] Unauthorized request for
controller=CloudController and action=RunInstances
2011-10-27 18:29:22,304 INFO nova.api
[4f056dc4-6515-4bd0-bd09-0c1584b9fc39 demo 2] 0.60822s 127.0.0.1 POST
/services/Cloud/ CloudController:RunInstances 401 [Boto/2.0 (linux2)]
application/x-www-form-urlencoded text/plain

Does anyone know what's going on? Thanks,

Yun

On Tue, Oct 25, 2011 at 8:51 AM, David Kranz  wrote:
> Along the same lines,  how do you export the shell variables for euca-tools
> with keystone since nova-manage to create the zipfile does not work?
>
>  -David
>
> On 10/24/2011 8:29 PM, Vishvananda Ishaya wrote:
>
> Speaking of keystone diablo tag, it is currently missing the following
> commit:
> https://github.com/openstack/keystone/commit/2bb474331d73e7c6d2a507cb097c50cfe65ad6b6
> This commit is required for the ec2 api to work with keystone.  Seems like
> we need to move the tag or create a keystone/stable branch and pull this in.
> Vish
> On Oct 24, 2011, at 12:03 AM, Mark McLoughlin wrote:
>
> Hey,
>
> I just noticed a few things when reviewing the Fedora packaging of
> keystone:
>
>  - There's no diablo release tarball on https://launchpad.net/keystone
>    like other projects
>
>  - The 2011.3 tag in git has version=1.0 in setup.py. Which versioning
>    scheme is keystone going to follow?
>
>  - The version in master is non-numeric 'essex' rather than e.g.
>    2011.3 or 1.1
>
> Thanks,
> Mark.
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] virtio for VM NICs

2011-10-27 Thread Yun Mao

Is there a reason that libvirt_use_virtio_for_bridges is not set to
True by default? Without virtio the network performance in kvm is
ridiculously slow.. Thanks,

Yun

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] nova-compute often stop since of libvirt-bin

2011-10-27 Thread DeadSun

hi,it's my libvirt-bin version:

dpkg -l | grep libvirt
ii libvirt-bin 0.9.2-4ubuntu15 the programs for the libvirt library
ii libvirt0 0.9.2-4ubuntu15 library for interfacing with different
virtualization systems
ii python-libvirt 0.9.2-4ubuntu15 libvirt Python bindings

2011/10/28 Jesse Andrews 

> What version of libvirt bin/d/python are you running?
>
> on debian/ubuntu do:
>
>   dpkg -l | grep libvirt
>
> On Thu, Oct 27, 2011 at 8:49 AM, Vishvananda Ishaya
>  wrote:
> > I haven't seen this happen.  Sounds like it may be a libvirt bug.
> > On Oct 27, 2011, at 7:48 AM, mao weijie wrote:
> >
> > I have found nova-compute often stopped that no update info in log and
> > service status is XXX. Then I restart nova-compute, but it is still stop
> in
> > "DEBUG nova.virt.libvirt_conn [-] Connecting to libvirt: qemu:///system
> from
> > (pid=31811) _get_connection
> /data/nova/nova/virt/libvirt/connection.py:205"
> > Everytime I need to stop libvirt-bin and start libvirt-bin, then I can
> start
> > nova-compute successfully.
> > Does anyone meet this case?
> >
> > 非淡泊�o以明志，非���o�o以致�h
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
> >
>



-- 
非淡薄无以明志，非宁静无以致远
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

I ran it against Nova's; needs some tweaking, but should be doable.

If you think you'll use it, I'll page it back in (like I said, it's been a long 
time) and fix it up. Of course, at some point it's going to be easier to 
re-implement (e.g., in Python), but if it looks like it's 80-90% of the way 
there, I don't mind putting some effort into it.

What do you think about an option to split output into multiple files? It would 
only work with libxslt/xsltproc, but given how long some of the documents would 
be, it might be worth it.

Cheers,

On 28/10/2011, at 11:25 AM, Nati Ueno wrote:

> Hi Mark
> 
> This is cool!
> Could you apply this for OpenStack WADL?
> Could you generate parameter list from XSD with XSLT?
> 
> 2011/10/27 Mark Nottingham :
>> Example output at:
>>  http://mnot.github.com/wadl_stylesheets/
>> 
>> Cheers,
>> 
>> 
>> On 28/10/2011, at 9:55 AM, Mark Nottingham wrote:
>> 
>>> FWIW, a long time ago* I wrote an XSLT to generate HTML docs from WADL -- 
>>> see:
>>>   https://github.com/mnot/wadl_stylesheets
>>> 
>>> I haven't maintained them in some time; however, if there's enough 
>>> interest, I can dust them off / update / etc.
>>> 
>>> Cheers,
>>> 
>>> * They went into github in 2009, but were written in about 2005.
>>> 
>>> 
>>> 
>>> On 28/10/2011, at 5:20 AM, Joseph Heck wrote:
>>> 
 Jorge -

 It's way back the beginning of this thread - A consolidated single website 
 with API docs as HTML pages that is easy for developers to consume. I'm 
 looking forward to seeing the WADL parser, already on that thread with 
 David Cramer directly. I can wait until he's got it in github, which he 
 said would likely be next week.

 The docs generated on doc.openstack.org are all in docbook format - neat, 
 but not what I'm after. As I mentioned some 40 msgs back (now quite lost, 
 I'm sure), what I'm looking to create is something like these sites 
 provide:

  https://dev.twitter.com/docs/api
  http://developer.netflix.com/docs/REST_API_Reference
  http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
  http://upcoming.yahoo.com/services/api/

 That we can generate (ideally dynamically, but I'm not wedded to that) 
 from the API's of all of Nova, Glance, Keystone and Quantum - both what 
 we've labelled as core and extensions.

 My goal isn't to make, parse, or manually read WADL's - it's to make this 
 set of web pages. If WADL helps me get there expediently, I'm all over it.

 -joe

 On Oct 27, 2011, at 11:03 AM, Jorge Williams wrote:
> As I stated in previous emails, we are pulling data from the WADL to grab 
> human-consumable REST API docs that live at docs.openstack.org today.  We 
> can certainly expand that capability to create a unified API 
> documentation set rather than individual guides.  A lot of the hard work 
> for parsing is already done, and we'll be releasing a WADL normalizer 
> that puts the WADL in an easer to process form.
> 
> Joe, I'd love to hear more about what you're trying to accomplish.  Maybe 
> we can help you leverage the tools we have to accomplish them.
> 
> -jOrGe W.
> 
> 
> On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:
> 
>> Yeah, that's what I've been poking at and the original start of this 
>> rather lengthy thread. Unfortunately, WADL, while it appears complete, 
>> is rather obnoxious for pulling out data. Or more accurately, I haven't 
>> fully understood the WADL specification in order to write a WADL parser 
>> to allow me to do just that. I'm poking at it now, but my original goal 
>> wasn't to write an XML parser but to just create a unified API 
>> documentation set on a web site to make it easier to consume OpenStack 
>> services.
>> 
>> -joe
>> 
>> On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
>>> It would be great if we could do some kind of transform of the IDL to 
>>> generate (some of) the human-consumable REST API documentation that 
>>> lives at docs.openstack.org. That would simplify the task of keeping 
>>> those docs up to date.
>>> 
>>> Lorin
>>> --
>>> Lorin Hochstein, Computer Scientist
>>> USC Information Sciences Institute
>>> 703.812.3710
>>> http://www.east.isi.edu/~lorin
>>> 
>>> 
>>> On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
 Sounds awesome!

 I've done an application like this in the past where an entire web UI 
 was data driven using a custom IDL. It had to have presentation hints 
 associated with it (acceptable values, display widget, etc). Not 
 something WADL supports inherently I'm sure. But, I know from 
 experience this can work.

 I don't really care what the IDL is, so long as we don't have to write 
 a parser for i

Re: [Openstack] +1, All services should have WADLs

2011-10-27 Thread Nati Ueno

Hi Mark

This is cool!
Could you apply this for OpenStack WADL?
Could you generate parameter list from XSD with XSLT?

2011/10/27 Mark Nottingham :
> Example output at:
>  http://mnot.github.com/wadl_stylesheets/
>
> Cheers,
>
>
> On 28/10/2011, at 9:55 AM, Mark Nottingham wrote:
>
>> FWIW, a long time ago* I wrote an XSLT to generate HTML docs from WADL -- 
>> see:
>>   https://github.com/mnot/wadl_stylesheets
>>
>> I haven't maintained them in some time; however, if there's enough interest, 
>> I can dust them off / update / etc.
>>
>> Cheers,
>>
>> * They went into github in 2009, but were written in about 2005.
>>
>>
>>
>> On 28/10/2011, at 5:20 AM, Joseph Heck wrote:
>>
>>> Jorge -
>>>
>>> It's way back the beginning of this thread - A consolidated single website 
>>> with API docs as HTML pages that is easy for developers to consume. I'm 
>>> looking forward to seeing the WADL parser, already on that thread with 
>>> David Cramer directly. I can wait until he's got it in github, which he 
>>> said would likely be next week.
>>>
>>> The docs generated on doc.openstack.org are all in docbook format - neat, 
>>> but not what I'm after. As I mentioned some 40 msgs back (now quite lost, 
>>> I'm sure), what I'm looking to create is something like these sites provide:
>>>
>>>      https://dev.twitter.com/docs/api
>>>      http://developer.netflix.com/docs/REST_API_Reference
>>>      http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
>>>      http://upcoming.yahoo.com/services/api/
>>>
>>> That we can generate (ideally dynamically, but I'm not wedded to that) from 
>>> the API's of all of Nova, Glance, Keystone and Quantum - both what we've 
>>> labelled as core and extensions.
>>>
>>> My goal isn't to make, parse, or manually read WADL's - it's to make this 
>>> set of web pages. If WADL helps me get there expediently, I'm all over it.
>>>
>>> -joe
>>>
>>> On Oct 27, 2011, at 11:03 AM, Jorge Williams wrote:
 As I stated in previous emails, we are pulling data from the WADL to grab 
 human-consumable REST API docs that live at docs.openstack.org today.  We 
 can certainly expand that capability to create a unified API documentation 
 set rather than individual guides.  A lot of the hard work for parsing is 
 already done, and we'll be releasing a WADL normalizer that puts the WADL 
 in an easer to process form.

 Joe, I'd love to hear more about what you're trying to accomplish.  Maybe 
 we can help you leverage the tools we have to accomplish them.

 -jOrGe W.


 On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:

> Yeah, that's what I've been poking at and the original start of this 
> rather lengthy thread. Unfortunately, WADL, while it appears complete, is 
> rather obnoxious for pulling out data. Or more accurately, I haven't 
> fully understood the WADL specification in order to write a WADL parser 
> to allow me to do just that. I'm poking at it now, but my original goal 
> wasn't to write an XML parser but to just create a unified API 
> documentation set on a web site to make it easier to consume OpenStack 
> services.
>
> -joe
>
> On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
>> It would be great if we could do some kind of transform of the IDL to 
>> generate (some of) the human-consumable REST API documentation that 
>> lives at docs.openstack.org. That would simplify the task of keeping 
>> those docs up to date.
>>
>> Lorin
>> --
>> Lorin Hochstein, Computer Scientist
>> USC Information Sciences Institute
>> 703.812.3710
>> http://www.east.isi.edu/~lorin
>>
>>
>> On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
>>> Sounds awesome!
>>>
>>> I've done an application like this in the past where an entire web UI 
>>> was data driven using a custom IDL. It had to have presentation hints 
>>> associated with it (acceptable values, display widget, etc). Not 
>>> something WADL supports inherently I'm sure. But, I know from 
>>> experience this can work.
>>>
>>> I don't really care what the IDL is, so long as we don't have to write 
>>> a parser for it in 10 different languages ... which is why XML/JSON 
>>> hold such appeal (although JSON in C keeps me awake at night).
>>>
>>> -S
>>>
>>> 
>>> From: Mark Nottingham [m...@mnot.net]
>>> Sent: Thursday, October 27, 2011 10:38 AM
>>> To: Sandy Walsh
>>> Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
>>> Subject: Re: [Openstack] +1,  All services should have WADLs
>>>
>>> I'm totally on board with having the interface being machine-consumable 
>>> at runtime -- see the previous discussion on versioning and 
>>> extensibility -- but WADL isn't really designed for this. I'm sketching 
>>> up something more appropriate, and wi

Re: [Openstack] API Versioning and Extensibility

On 28/10/2011, at 2:36 AM, George Reese wrote:

> The complete lack of evolution of the OSAPI combined with the irrational 
> resistance to the EC2 API has struck a nerve with me.
> 
> #1 Feature coverage in the OSAPI is atrocious. And I don't get the feeling 
> there's anyone seriously doing anything about it. Of course, you can always 
> say, "George, it's an Open Source project. If you don't like it, feel free to 
> fix it." Of course, I'm not worrying about all kinds of bizarre OpenStack 
> projects that have nothing to do with building a basic, functional cloud 
> platform either.

Great - do you have a list?

> #3 Push scales a hell of a lot better than having tools polling a cloud 
> constantly. It doesn't matter whether it is polling the API, polling a feed, 
> or polling a message queue. Polling is one of the most unscalable things you 
> can do in any distributed systems scenario.

That's a very absolute assertion to make, and doesn't reflect the experiences 
I've had.

I've helped build polling-based APIs and pull-based systems inside of Yahoo! 
and Akamai that scale quite well.

I've built polling-based systems where push-based message busses failed, 
miserably.

In my experience, push-based systems bring a LOT of complexity and 
implementation overhead, and are especially problematic to run across 
administrative domains; there are certainly cases that require them, but 
they're not a silver bullet. 

> Calling it a feed doesn't magically solve the problem. 

Absolutely not. They're a tool to be used, just like any other. 

> Push notifications are the only mechanism for solving the scaling issue. You 
> push any changes to a message queue. Agents pick up the changes and send them 
> on to subscriber endpoints. Not that hard.

Again, that's a big assertion to make. I'd rather evaluate the tools based upon 
the use case, rather than arguing by assertion.

Thanks,

--
Mark Nottingham   http://www.mnot.net/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

Example output at:
  http://mnot.github.com/wadl_stylesheets/

Cheers,


On 28/10/2011, at 9:55 AM, Mark Nottingham wrote:

> FWIW, a long time ago* I wrote an XSLT to generate HTML docs from WADL -- see:
>   https://github.com/mnot/wadl_stylesheets
> 
> I haven't maintained them in some time; however, if there's enough interest, 
> I can dust them off / update / etc.
> 
> Cheers,
> 
> * They went into github in 2009, but were written in about 2005.
> 
> 
> 
> On 28/10/2011, at 5:20 AM, Joseph Heck wrote:
> 
>> Jorge - 
>> 
>> It's way back the beginning of this thread - A consolidated single website 
>> with API docs as HTML pages that is easy for developers to consume. I'm 
>> looking forward to seeing the WADL parser, already on that thread with David 
>> Cramer directly. I can wait until he's got it in github, which he said would 
>> likely be next week.
>> 
>> The docs generated on doc.openstack.org are all in docbook format - neat, 
>> but not what I'm after. As I mentioned some 40 msgs back (now quite lost, 
>> I'm sure), what I'm looking to create is something like these sites provide:
>> 
>>  https://dev.twitter.com/docs/api
>>  http://developer.netflix.com/docs/REST_API_Reference
>>  http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
>>  http://upcoming.yahoo.com/services/api/
>> 
>> That we can generate (ideally dynamically, but I'm not wedded to that) from 
>> the API's of all of Nova, Glance, Keystone and Quantum - both what we've 
>> labelled as core and extensions.
>> 
>> My goal isn't to make, parse, or manually read WADL's - it's to make this 
>> set of web pages. If WADL helps me get there expediently, I'm all over it.
>> 
>> -joe
>> 
>> On Oct 27, 2011, at 11:03 AM, Jorge Williams wrote:
>>> As I stated in previous emails, we are pulling data from the WADL to grab 
>>> human-consumable REST API docs that live at docs.openstack.org today.  We 
>>> can certainly expand that capability to create a unified API documentation 
>>> set rather than individual guides.  A lot of the hard work for parsing is 
>>> already done, and we'll be releasing a WADL normalizer that puts the WADL 
>>> in an easer to process form.
>>> 
>>> Joe, I'd love to hear more about what you're trying to accomplish.  Maybe 
>>> we can help you leverage the tools we have to accomplish them.
>>> 
>>> -jOrGe W.
>>> 
>>> 
>>> On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:
>>> 
 Yeah, that's what I've been poking at and the original start of this 
 rather lengthy thread. Unfortunately, WADL, while it appears complete, is 
 rather obnoxious for pulling out data. Or more accurately, I haven't fully 
 understood the WADL specification in order to write a WADL parser to allow 
 me to do just that. I'm poking at it now, but my original goal wasn't to 
 write an XML parser but to just create a unified API documentation set on 
 a web site to make it easier to consume OpenStack services.
 
 -joe
 
 On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
> It would be great if we could do some kind of transform of the IDL to 
> generate (some of) the human-consumable REST API documentation that lives 
> at docs.openstack.org. That would simplify the task of keeping those docs 
> up to date.
> 
> Lorin
> --
> Lorin Hochstein, Computer Scientist
> USC Information Sciences Institute
> 703.812.3710
> http://www.east.isi.edu/~lorin
> 
> 
> On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
>> Sounds awesome!
>> 
>> I've done an application like this in the past where an entire web UI 
>> was data driven using a custom IDL. It had to have presentation hints 
>> associated with it (acceptable values, display widget, etc). Not 
>> something WADL supports inherently I'm sure. But, I know from experience 
>> this can work.
>> 
>> I don't really care what the IDL is, so long as we don't have to write a 
>> parser for it in 10 different languages ... which is why XML/JSON hold 
>> such appeal (although JSON in C keeps me awake at night).
>> 
>> -S
>> 
>> 
>> From: Mark Nottingham [m...@mnot.net]
>> Sent: Thursday, October 27, 2011 10:38 AM
>> To: Sandy Walsh
>> Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
>> Subject: Re: [Openstack] +1,  All services should have WADLs
>> 
>> I'm totally on board with having the interface being machine-consumable 
>> at runtime -- see the previous discussion on versioning and 
>> extensibility -- but WADL isn't really designed for this. I'm sketching 
>> up something more appropriate, and will be able to talk about it soon 
>> (hopefully).
>> 
>> 
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscri

[Openstack] Request for review - Keystone RBAC calls

2011-10-27 Thread Joe Savak

Hi Stackers,
Keystone has an RBAC prototype on its roadmap for e-1 and we'd 
like to get your feedback on the blueprint for it 
(https://blueprints.launchpad.net/keystone/+spec/rbac-keystone). OS-RBAC, a 
keystone extension, will allow Nova, Swift, Glance, and Dashboard to manage 
capabilities and assign capabilities to roles.  The prototype will not be the 
full implementation of it, but will include WADL, XSD, and some of the major 
calls.

Take a look and let us know your feedback.

Thanks,
Joe

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility


On 28/10/2011, at 2:39 AM, Bryan Taylor wrote:

> On 10/26/2011 11:19 PM, Mark Nottingham wrote:
>>> To be truly RESTful at the level of the Fielding article (which I
>>> actually think is the best description of HATEOAS there is) you
>>> shouldn't have these variants at all.  I worry about us trying to
>>> put lipstick on the pig -- all these variants are a crummy
>>> compromise to work around broken browsers that do allow changing
>>> the accepts header.
> 
>> It's perfectly legitimate to link to a resource that has only one
>> representation, such as having /foo.html to allow people to
>> specifically refer to the HTML version of the /foo resource. That's
>> effectively agent-driven negotiation, and it's just as valid
>> (RESTful, if you will) as server-driven negotiation. Nothing that Roy
>> has said contradicts that, because a core principle of REST (if we
>> really want to go there) is that important things have identity, and
>> that links are what drive things (agent-driven negotiation is just
>> linking, really).
> 
> It harms visibility, a key goal of REST, especially if we are doing this on 
> every resource. If I do a PUT to /foo.xml the intermediaries have no way to 
> know that I'm really affecting /foo .

Yes -- but by their nature, intermediaries are really limited beasts anyway. 
You can address this by surfacing relationships in Link headers, and writing 
generic specs to exploit those links (this is how we did cache invalidation 
with LCI - ).

It comes back to the central question of utility -- what do you want those 
intermediaries to do? If they need visibility for a particular purpose, that's 
great, and we should design to be able to exploit this in the future as much as 
possible, but we can never offer them full visibility.


> I've made the point before that I can define a resource to be anything I 
> want, including a particular representation of another resource. I'm not 
> opposed to using variants generally on the web, and Jorge's pdf example is 
> pretty much impossible to do without them. I'm just questioning whether it 
> makes sense to use them to solve our developer API introspection problem 
> within our web service APIs.

As a limited tool that's used well, I have no problem with it. I do agree that 
having them spread over the interface like a thick layer of peanut butter makes 
me uneasy.

Cheers,


--
Mark Nottingham   http://www.mnot.net/




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

FWIW, a long time ago* I wrote an XSLT to generate HTML docs from WADL -- see:
   https://github.com/mnot/wadl_stylesheets

I haven't maintained them in some time; however, if there's enough interest, I 
can dust them off / update / etc.

Cheers,

* They went into github in 2009, but were written in about 2005.



On 28/10/2011, at 5:20 AM, Joseph Heck wrote:

> Jorge - 
> 
> It's way back the beginning of this thread - A consolidated single website 
> with API docs as HTML pages that is easy for developers to consume. I'm 
> looking forward to seeing the WADL parser, already on that thread with David 
> Cramer directly. I can wait until he's got it in github, which he said would 
> likely be next week.
> 
> The docs generated on doc.openstack.org are all in docbook format - neat, but 
> not what I'm after. As I mentioned some 40 msgs back (now quite lost, I'm 
> sure), what I'm looking to create is something like these sites provide:
> 
>   https://dev.twitter.com/docs/api
>   http://developer.netflix.com/docs/REST_API_Reference
>   http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
>   http://upcoming.yahoo.com/services/api/
> 
> That we can generate (ideally dynamically, but I'm not wedded to that) from 
> the API's of all of Nova, Glance, Keystone and Quantum - both what we've 
> labelled as core and extensions.
> 
> My goal isn't to make, parse, or manually read WADL's - it's to make this set 
> of web pages. If WADL helps me get there expediently, I'm all over it.
> 
> -joe
> 
> On Oct 27, 2011, at 11:03 AM, Jorge Williams wrote:
>> As I stated in previous emails, we are pulling data from the WADL to grab 
>> human-consumable REST API docs that live at docs.openstack.org today.  We 
>> can certainly expand that capability to create a unified API documentation 
>> set rather than individual guides.  A lot of the hard work for parsing is 
>> already done, and we'll be releasing a WADL normalizer that puts the WADL in 
>> an easer to process form.
>> 
>> Joe, I'd love to hear more about what you're trying to accomplish.  Maybe we 
>> can help you leverage the tools we have to accomplish them.
>> 
>> -jOrGe W.
>> 
>> 
>> On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:
>> 
>>> Yeah, that's what I've been poking at and the original start of this rather 
>>> lengthy thread. Unfortunately, WADL, while it appears complete, is rather 
>>> obnoxious for pulling out data. Or more accurately, I haven't fully 
>>> understood the WADL specification in order to write a WADL parser to allow 
>>> me to do just that. I'm poking at it now, but my original goal wasn't to 
>>> write an XML parser but to just create a unified API documentation set on a 
>>> web site to make it easier to consume OpenStack services.
>>> 
>>> -joe
>>> 
>>> On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
 It would be great if we could do some kind of transform of the IDL to 
 generate (some of) the human-consumable REST API documentation that lives 
 at docs.openstack.org. That would simplify the task of keeping those docs 
 up to date.
 
 Lorin
 --
 Lorin Hochstein, Computer Scientist
 USC Information Sciences Institute
 703.812.3710
 http://www.east.isi.edu/~lorin
 
 
 On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
> Sounds awesome!
> 
> I've done an application like this in the past where an entire web UI was 
> data driven using a custom IDL. It had to have presentation hints 
> associated with it (acceptable values, display widget, etc). Not 
> something WADL supports inherently I'm sure. But, I know from experience 
> this can work.
> 
> I don't really care what the IDL is, so long as we don't have to write a 
> parser for it in 10 different languages ... which is why XML/JSON hold 
> such appeal (although JSON in C keeps me awake at night).
> 
> -S
> 
> 
> From: Mark Nottingham [m...@mnot.net]
> Sent: Thursday, October 27, 2011 10:38 AM
> To: Sandy Walsh
> Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
> Subject: Re: [Openstack] +1,  All services should have WADLs
> 
> I'm totally on board with having the interface being machine-consumable 
> at runtime -- see the previous discussion on versioning and extensibility 
> -- but WADL isn't really designed for this. I'm sketching up something 
> more appropriate, and will be able to talk about it soon (hopefully).
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net

Re: [Openstack] describing APIs for OpenStack consumers

Generating WADL (or anything else) from code is fine, as long as we have the 
processes / tools (e.g., CI) in place to assure that a trivial code change 
doesn't make a backwards-incompatible change in what we expose to clients.

Do we?

(really, we should have these in place regardless of how things are generated)

Cheers,


On 28/10/2011, at 4:50 AM, Nati Ueno wrote:

> Hi folks
> 
> I tried to generate WADL from nova code.
> I could get all resource URI and method from Routes object.
> However, I could not get input parameters from code.
> (The api method accesses body argument directly. This is also bad for
> input validation QA effort.)
> 
> But If we use some annotations, it may be solved.
> Also, It looks possible to generate Resource definitions from model class.
> 
> 2011/10/26 Ziad Sawalha :
>> So you would do a diff if the generated WADL against the expected WADL. That 
>> would mean we use both. I think that's a reasonable approach.
>> 
>> 
>> 
>> On Oct 26, 2011, at 12:31 PM, "Monsyne Dragon"  wrote:
>> 
>>> 
>>> On Oct 26, 2011, at 10:48 AM, Kevin L. Mitchell wrote:
>>> 
 On Tue, 2011-10-25 at 15:30 -0700, Joseph Heck wrote:
> It sounds like even though most of us hate WADL, it's what we're
> expending effort after to make a consolidated API set. So unless Nati
> and Ravi want to switch to using Swagger (or something else), WADL is
> the direction we're heading. I totally agree with Daryl that reading
> it is a PITA, and am finding (from my part) that the only definitive
> way to know about writing the docs and documenting the authoritative
> API is to read the underlying code. (which is what I suspect Nati
> likely did with the pull request that adds in WADL for the
> Nova/OpenCompute extension API)
 
 I wonder if it would be possible to generate much of the WADL from
 introspecting the code itself...surely the URL structure itself can be
 extracted from the paste setup, and the XML templates code I recently
 contributed could easily be traversed to provide at least a basic
 description of the output.  That could at least provide a starting point
 for generating WADLs...
 
 (Of course, I propose this, having little idea of what actually goes in
 a WADL, but still... ;)
>>> 
>>> I've worked with WADL and WSDL before, and yes, it is indeed possible to 
>>> generate the WADL by introspecting code. (with a few decorators/annotations 
>>> assisting)
>>> This is what Sandy Walsh is suggesting, and I highly, highly recommend this 
>>> approach.  Otherwise you have to either generate code from an external 
>>> WADL, which makes the code a mess, or keep the WADL in sync with the code 
>>> manually (bleh).  The big advantage of generating WADLs from the code is 
>>> that you then get a machine-readable description of what the *code* thinks 
>>> the interface is, not what you *hope* the interface is. That way,  if you 
>>> look at the generated WADL, and if the interface isn't what it *should* be 
>>> (as in "Where did THAT resource come from, and where did the foobar param 
>>> on that GET method go?"), you know you have a bug to fix.
>>> 
>>> --
>>>Monsyne M. Dragon
>>>OpenStack/Nova
>>>cell 210-441-0965
>>>work x 5014190
>>> 
>>> 
>>> ___
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>> 
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>> 
> 
> 
> 
> -- 
> Nachi Ueno
> email:nati.u...@gmail.com
> twitter:http://twitter.com/nati
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

--
Mark Nottingham   http://www.mnot.net/




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Push vs Polling (from Versioning Thread)

+1 Dragon

From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net 
[openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of 
Monsyne Dragon [mdra...@rackspace.com]
Sent: Thursday, October 27, 2011 4:14 PM
To: George Reese
Cc: 
Subject: Re: [Openstack] Push vs Polling (from Versioning Thread)

On Oct 27, 2011, at 11:38 AM, George Reese wrote:

> Sent from my iPhone
>
> On Oct 27, 2011, at 11:26, Bryan Taylor  wrote:
>
>> On 10/27/2011 10:36 AM, George Reese wrote:
>>
>>> #3 Push scales a hell of a lot better than having tools polling a cloud
>>> constantly. It doesn't matter whether it is polling the API, polling a
>>> feed, or polling a message queue. Polling is one of the most unscalable
>>> things you can do in any distributed systems scenario. Calling it a feed
>>> doesn't magically solve the problem. Actually, it's quite hard on its
>>> own in an IaaS scenario and has scaling issues independent of the
>>> polling issue.
>>
>> I disagree. The web was designed specifically to solve the distributed 
>> scaling problem and it's based on HTTP polling. It scales pretty well. The 
>> argument against polling not scaling inevitably neglects using caching 
>> properly.
>>
>
> The web was not designed to deal with a bunch of clients needing to
> know about infrastructure changes the instant they happen.

True.  This whole issue is the reason Nova's existing notification system  is 
designed  as a push system.  Currently it's used to push error notifications 
and usage info, but there is no reason it could not eventually also provide 
notifications to end users.  After watching demos of large cloud users where 
they were polling apis to see when their instances were ready (and often 
spinning up new ones when that didn't happen fast enough)
I kept that use case in mind when coding the notifications system.

> And API data should not be cached. The Rackspace API used to do that,
> and it created a mess.
>
>> Push doesn't scaled because it requires the server to know about every 
>> client and track conversational state with them.
>
> No, it doesn't. You push changes as they occur to a message queue. A
> separate system tracks subscribers and sends them out. There is no
> conversational state if done right.

Indeed. this is how the notifications system is/can work right now. If you turn 
notifications on, nova pushes them to a rabbit queue.  A separate app, namely a 
hub using the standard PubSubHubbub protocol, (plus an external rabbit queue -> 
feed generator app we wrote, called Yagi) manages the subscriptions and pings 
the subscribers.

>
>> If you need reliability, this requires persisting that conversational state. 
>> In order to allow this to happen you have to have some kind of registration 
>> protocol for clients. If some fraction of those clients are flaky, the 
>> conversational state tracking will kill you because each client consumes 
>> resources and so flaky clients = resource leak.

Existing PSH hubs are designed to handle large numbers of potentially 
unreliable clients.

>>
>> Push wins when you need very low latency delivery, high message throughput 
>> to individual consumers, or server side guarantees of delivery to individual 
>> consumers, but not for scaling to a large number of clients in a climate of 
>> an elastic infrastructure.
>>
>>> Push notifications are the only mechanism for solving the scaling issue.
>>> You push any changes to a message queue. Agents pick up the changes and
>>> send them on to subscriber endpoints. Not that hard.
>>
>> Not that hard with a few fairly reliable clients. Very hard with a web scale 
>> set of unreliable clients while I simultaneously need to scale the back end.
>

Actually, we are already implementing this at 200+node scale in nova, since 
this is  how we are handling the collection of  usage data for billing.  At the 
moment is seems to be working reasonably well. We are not using the  PSH hubs 
atm, since we are pushing to a few internal consumers via AtomPub, and don't 
need the complex subscription management, but from our point of view, pinging a 
hub is no different from what we already do, and the hub handles the 
subscription details.

It would be nice to one day support notifications of end-users, as I think it 
would be of great benefit to them. There's work that would need to be done 
around hub/Yagi auth, and I think there is some bigger fish to fry, nova 
functionality-wise at the moment, but it is something to keep in mind.

--
Monsyne M. Dragon
OpenStack/Nova
cell 210-441-0965
work x 5014190

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Pos

[Openstack] OpenStack Dashboard Cactus tag

2011-10-27 Thread Devin Carlen

Hello all,

When we transitioned from Launchpad to Github, we didn't carry over the tag for 
Cactus.  I've had a few people express interest lately in continuing to use it, 
so we have retagged for Cactus.  You can grab it here if you need:


> https://github.com/4P/horizon/zipball/2011.2



Devin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] describing APIs for OpenStack consumers

We may need to change the way the implementations are written to require 
explicit keyword parameters. No more sneaking into the . This may also 
mean that something more robust than the routes library may be needed?

Sadly, annotations of some form are probably going to be needed since we can't 
introspect the return types. The joy of dynamically typed languages.

Perhaps even parsing the doc string? Wait ... come back! As ghastly as it 
sounds, it does keep the interface definition with the implementation, so the 
likelihood of them going out of sync is lessened. I'm hesitant to suggest an 
external IDL form, but perhaps the other approach is to hand-craft the WADL and 
generate the code stubs from that? (not a fan)

-S


From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net 
[openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of 
Nati Ueno [nati.u...@gmail.com]
Sent: Thursday, October 27, 2011 2:50 PM
To: Ziad Sawalha
Cc: 
Subject: Re: [Openstack] describing APIs for OpenStack consumers

Hi folks

I tried to generate WADL from nova code.
I could get all resource URI and method from Routes object.
However, I could not get input parameters from code.
(The api method accesses body argument directly. This is also bad for
input validation QA effort.)

But If we use some annotations, it may be solved.
Also, It looks possible to generate Resource definitions from model class.

2011/10/26 Ziad Sawalha :
> So you would do a diff if the generated WADL against the expected WADL. That 
> would mean we use both. I think that's a reasonable approach.
>
>
>
> On Oct 26, 2011, at 12:31 PM, "Monsyne Dragon"  wrote:
>
>>
>> On Oct 26, 2011, at 10:48 AM, Kevin L. Mitchell wrote:
>>
>>> On Tue, 2011-10-25 at 15:30 -0700, Joseph Heck wrote:
 It sounds like even though most of us hate WADL, it's what we're
 expending effort after to make a consolidated API set. So unless Nati
 and Ravi want to switch to using Swagger (or something else), WADL is
 the direction we're heading. I totally agree with Daryl that reading
 it is a PITA, and am finding (from my part) that the only definitive
 way to know about writing the docs and documenting the authoritative
 API is to read the underlying code. (which is what I suspect Nati
 likely did with the pull request that adds in WADL for the
 Nova/OpenCompute extension API)
>>>
>>> I wonder if it would be possible to generate much of the WADL from
>>> introspecting the code itself...surely the URL structure itself can be
>>> extracted from the paste setup, and the XML templates code I recently
>>> contributed could easily be traversed to provide at least a basic
>>> description of the output.  That could at least provide a starting point
>>> for generating WADLs...
>>>
>>> (Of course, I propose this, having little idea of what actually goes in
>>> a WADL, but still... ;)
>>
>> I've worked with WADL and WSDL before, and yes, it is indeed possible to 
>> generate the WADL by introspecting code. (with a few decorators/annotations 
>> assisting)
>> This is what Sandy Walsh is suggesting, and I highly, highly recommend this 
>> approach.  Otherwise you have to either generate code from an external WADL, 
>> which makes the code a mess, or keep the WADL in sync with the code manually 
>> (bleh).  The big advantage of generating WADLs from the code is that you 
>> then get a machine-readable description of what the *code* thinks the 
>> interface is, not what you *hope* the interface is. That way,  if you look 
>> at the generated WADL, and if the interface isn't what it *should* be (as in 
>> "Where did THAT resource come from, and where did the foobar param on that 
>> GET method go?"), you know you have a bug to fix.
>>
>> --
>>Monsyne M. Dragon
>>OpenStack/Nova
>>cell 210-441-0965
>>work x 5014190
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>



--
Nachi Ueno
email:nati.u...@gmail.com
twitter:http://twitter.com/nati

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

Ah yes,

API reference pages that span all the projects.  That's totally doable, I know 
we had plans for doing such a thing, but not sure where those plans are.  We 
were planing on using WADL for that.  Maybe we should get together with some of 
the doc folks Anne, David to come up with a strategy. How about we discuss at 
the next Doc Team Meeting  http://wiki.openstack.org/Meetings/DocTeamMeeting

-jOrGe W.


On Oct 27, 2011, at 1:20 PM, Joseph Heck wrote:

Jorge -

It's way back the beginning of this thread - A consolidated single website with 
API docs as HTML pages that is easy for developers to consume. I'm looking 
forward to seeing the WADL parser, already on that thread with David Cramer 
directly. I can wait until he's got it in github, which he said would likely be 
next week.

The docs generated on doc.openstack.org are all in 
docbook format - neat, but not what I'm after. As I mentioned some 40 msgs back 
(now quite lost, I'm sure), what I'm looking to create is something like these 
sites provide:

https://dev.twitter.com/docs/api
http://developer.netflix.com/docs/REST_API_Reference
http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
http://upcoming.yahoo.com/services/api/

That we can generate (ideally dynamically, but I'm not wedded to that) from the 
API's of all of Nova, Glance, Keystone and Quantum - both what we've labelled 
as core and extensions.

My goal isn't to make, parse, or manually read WADL's - it's to make this set 
of web pages. If WADL helps me get there expediently, I'm all over it.

-joe

On Oct 27, 2011, at 11:03 AM, Jorge Williams wrote:
As I stated in previous emails, we are pulling data from the WADL to grab 
human-consumable REST API docs that live at 
docs.openstack.org today.  We can certainly expand 
that capability to create a unified API documentation set rather than 
individual guides.  A lot of the hard work for parsing is already done, and 
we'll be releasing a WADL normalizer that puts the WADL in an easer to process 
form.

Joe, I'd love to hear more about what you're trying to accomplish.  Maybe we 
can help you leverage the tools we have to accomplish them.

-jOrGe W.


On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:

Yeah, that's what I've been poking at and the original start of this rather 
lengthy thread. Unfortunately, WADL, while it appears complete, is rather 
obnoxious for pulling out data. Or more accurately, I haven't fully understood 
the WADL specification in order to write a WADL parser to allow me to do just 
that. I'm poking at it now, but my original goal wasn't to write an XML parser 
but to just create a unified API documentation set on a web site to make it 
easier to consume OpenStack services.

-joe

On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
It would be great if we could do some kind of transform of the IDL to generate 
(some of) the human-consumable REST API documentation that lives at 
docs.openstack.org. That would simplify the task of 
keeping those docs up to date.

Lorin
--
Lorin Hochstein, Computer Scientist
USC Information Sciences Institute
703.812.3710
http://www.east.isi.edu/~lorin


On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
Sounds awesome!

I've done an application like this in the past where an entire web UI was data 
driven using a custom IDL. It had to have presentation hints associated with it 
(acceptable values, display widget, etc). Not something WADL supports 
inherently I'm sure. But, I know from experience this can work.

I don't really care what the IDL is, so long as we don't have to write a parser 
for it in 10 different languages ... which is why XML/JSON hold such appeal 
(although JSON in C keeps me awake at night).

-S


From: Mark Nottingham [m...@mnot.net]
Sent: Thursday, October 27, 2011 10:38 AM
To: Sandy Walsh
Cc: Mellquist, Peter; Joseph Heck; 
openstack@lists.launchpad.net
Subject: Re: [Openstack] +1,  All services should have WADLs

I'm totally on board with having the interface being machine-consumable at 
runtime -- see the previous discussion on versioning and extensibility -- but 
WADL isn't really designed for this. I'm sketching up something more 
appropriate, and will be able to talk about it soon (hopefully).


___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

_

Re: [Openstack] Push vs Polling (from Versioning Thread)

2011-10-27 Thread Monsyne Dragon

On Oct 27, 2011, at 11:38 AM, George Reese wrote:

> Sent from my iPhone
> 
> On Oct 27, 2011, at 11:26, Bryan Taylor  wrote:
> 
>> On 10/27/2011 10:36 AM, George Reese wrote:
>> 
>>> #3 Push scales a hell of a lot better than having tools polling a cloud
>>> constantly. It doesn't matter whether it is polling the API, polling a
>>> feed, or polling a message queue. Polling is one of the most unscalable
>>> things you can do in any distributed systems scenario. Calling it a feed
>>> doesn't magically solve the problem. Actually, it's quite hard on its
>>> own in an IaaS scenario and has scaling issues independent of the
>>> polling issue.
>> 
>> I disagree. The web was designed specifically to solve the distributed 
>> scaling problem and it's based on HTTP polling. It scales pretty well. The 
>> argument against polling not scaling inevitably neglects using caching 
>> properly.
>> 
> 
> The web was not designed to deal with a bunch of clients needing to
> know about infrastructure changes the instant they happen.

True.  This whole issue is the reason Nova's existing notification system  is 
designed  as a push system.  Currently it's used to push error notifications 
and usage info, but there is no reason it could not eventually also provide 
notifications to end users.  After watching demos of large cloud users where 
they were polling apis to see when their instances were ready (and often 
spinning up new ones when that didn't happen fast enough)
I kept that use case in mind when coding the notifications system.   

> And API data should not be cached. The Rackspace API used to do that,
> and it created a mess.
> 
>> Push doesn't scaled because it requires the server to know about every 
>> client and track conversational state with them.
> 
> No, it doesn't. You push changes as they occur to a message queue. A
> separate system tracks subscribers and sends them out. There is no
> conversational state if done right.

Indeed. this is how the notifications system is/can work right now. If you turn 
notifications on, nova pushes them to a rabbit queue.  A separate app, namely a 
hub using the standard PubSubHubbub protocol, (plus an external rabbit queue -> 
feed generator app we wrote, called Yagi) manages the subscriptions and pings 
the subscribers.   

> 
>> If you need reliability, this requires persisting that conversational state. 
>> In order to allow this to happen you have to have some kind of registration 
>> protocol for clients. If some fraction of those clients are flaky, the 
>> conversational state tracking will kill you because each client consumes 
>> resources and so flaky clients = resource leak.

Existing PSH hubs are designed to handle large numbers of potentially 
unreliable clients. 

>> 
>> Push wins when you need very low latency delivery, high message throughput 
>> to individual consumers, or server side guarantees of delivery to individual 
>> consumers, but not for scaling to a large number of clients in a climate of 
>> an elastic infrastructure.
>> 
>>> Push notifications are the only mechanism for solving the scaling issue.
>>> You push any changes to a message queue. Agents pick up the changes and
>>> send them on to subscriber endpoints. Not that hard.
>> 
>> Not that hard with a few fairly reliable clients. Very hard with a web scale 
>> set of unreliable clients while I simultaneously need to scale the back end.
> 

Actually, we are already implementing this at 200+node scale in nova, since 
this is  how we are handling the collection of  usage data for billing.  At the 
moment is seems to be working reasonably well. We are not using the  PSH hubs 
atm, since we are pushing to a few internal consumers via AtomPub, and don't 
need the complex subscription management, but from our point of view, pinging a 
hub is no different from what we already do, and the hub handles the 
subscription details. 

It would be nice to one day support notifications of end-users, as I think it 
would be of great benefit to them. There's work that would need to be done 
around hub/Yagi auth, and I think there is some bigger fish to fry, nova 
functionality-wise at the moment, but it is something to keep in mind. 

--
Monsyne M. Dragon
OpenStack/Nova 
cell 210-441-0965
work x 5014190

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] describing APIs for OpenStack consumers

2011-10-27 Thread Kevin L. Mitchell

On Thu, 2011-10-27 at 10:50 -0700, Nati Ueno wrote:
> I tried to generate WADL from nova code.
> I could get all resource URI and method from Routes object.
> However, I could not get input parameters from code.
> (The api method accesses body argument directly. This is also bad for
> input validation QA effort.)
> 
> But If we use some annotations, it may be solved.
> Also, It looks possible to generate Resource definitions from model class.

I've also considered that my templates code could be adapted to perform
deserialization as well as serialization.  You'd have to add a bit more
data to its structure to pull that trick off, but it'd show you the
exact structure of the XML input and output for automation tricks like
this.
-- 
Kevin L. Mitchell 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

Jorge - 

It's way back the beginning of this thread - A consolidated single website with 
API docs as HTML pages that is easy for developers to consume. I'm looking 
forward to seeing the WADL parser, already on that thread with David Cramer 
directly. I can wait until he's got it in github, which he said would likely be 
next week.

The docs generated on doc.openstack.org are all in docbook format - neat, but 
not what I'm after. As I mentioned some 40 msgs back (now quite lost, I'm 
sure), what I'm looking to create is something like these sites provide:

https://dev.twitter.com/docs/api
http://developer.netflix.com/docs/REST_API_Reference
http://code.google.com/p/bitly-api/wiki/ApiDocumentation#REST_API
http://upcoming.yahoo.com/services/api/

That we can generate (ideally dynamically, but I'm not wedded to that) from the 
API's of all of Nova, Glance, Keystone and Quantum - both what we've labelled 
as core and extensions.

My goal isn't to make, parse, or manually read WADL's - it's to make this set 
of web pages. If WADL helps me get there expediently, I'm all over it.

-joe

On Oct 27, 2011, at 11:03 AM, Jorge Williams wrote:
> As I stated in previous emails, we are pulling data from the WADL to grab 
> human-consumable REST API docs that live at docs.openstack.org today.  We can 
> certainly expand that capability to create a unified API documentation set 
> rather than individual guides.  A lot of the hard work for parsing is already 
> done, and we'll be releasing a WADL normalizer that puts the WADL in an easer 
> to process form.
> 
> Joe, I'd love to hear more about what you're trying to accomplish.  Maybe we 
> can help you leverage the tools we have to accomplish them.
> 
> -jOrGe W.
> 
> 
> On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:
> 
>> Yeah, that's what I've been poking at and the original start of this rather 
>> lengthy thread. Unfortunately, WADL, while it appears complete, is rather 
>> obnoxious for pulling out data. Or more accurately, I haven't fully 
>> understood the WADL specification in order to write a WADL parser to allow 
>> me to do just that. I'm poking at it now, but my original goal wasn't to 
>> write an XML parser but to just create a unified API documentation set on a 
>> web site to make it easier to consume OpenStack services.
>> 
>> -joe
>> 
>> On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
>>> It would be great if we could do some kind of transform of the IDL to 
>>> generate (some of) the human-consumable REST API documentation that lives 
>>> at docs.openstack.org. That would simplify the task of keeping those docs 
>>> up to date.
>>> 
>>> Lorin
>>> --
>>> Lorin Hochstein, Computer Scientist
>>> USC Information Sciences Institute
>>> 703.812.3710
>>> http://www.east.isi.edu/~lorin
>>> 
>>> 
>>> On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
 Sounds awesome!

 I've done an application like this in the past where an entire web UI was 
 data driven using a custom IDL. It had to have presentation hints 
 associated with it (acceptable values, display widget, etc). Not something 
 WADL supports inherently I'm sure. But, I know from experience this can 
 work.

 I don't really care what the IDL is, so long as we don't have to write a 
 parser for it in 10 different languages ... which is why XML/JSON hold 
 such appeal (although JSON in C keeps me awake at night).

 -S

 From: Mark Nottingham [m...@mnot.net]
 Sent: Thursday, October 27, 2011 10:38 AM
 To: Sandy Walsh
 Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
 Subject: Re: [Openstack] +1,  All services should have WADLs

 I'm totally on board with having the interface being machine-consumable at 
 runtime -- see the previous discussion on versioning and extensibility -- 
 but WADL isn't really designed for this. I'm sketching up something more 
 appropriate, and will be able to talk about it soon (hopefully).

 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
>>> 
>>> ___
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>> 
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@l

Re: [Openstack] describing APIs for OpenStack consumers

2011-10-27 Thread Nati Ueno

Hi folks

I tried to generate WADL from nova code.
I could get all resource URI and method from Routes object.
However, I could not get input parameters from code.
(The api method accesses body argument directly. This is also bad for
input validation QA effort.)

But If we use some annotations, it may be solved.
Also, It looks possible to generate Resource definitions from model class.

2011/10/26 Ziad Sawalha :
> So you would do a diff if the generated WADL against the expected WADL. That 
> would mean we use both. I think that's a reasonable approach.
>
>
>
> On Oct 26, 2011, at 12:31 PM, "Monsyne Dragon"  wrote:
>
>>
>> On Oct 26, 2011, at 10:48 AM, Kevin L. Mitchell wrote:
>>
>>> On Tue, 2011-10-25 at 15:30 -0700, Joseph Heck wrote:
 It sounds like even though most of us hate WADL, it's what we're
 expending effort after to make a consolidated API set. So unless Nati
 and Ravi want to switch to using Swagger (or something else), WADL is
 the direction we're heading. I totally agree with Daryl that reading
 it is a PITA, and am finding (from my part) that the only definitive
 way to know about writing the docs and documenting the authoritative
 API is to read the underlying code. (which is what I suspect Nati
 likely did with the pull request that adds in WADL for the
 Nova/OpenCompute extension API)
>>>
>>> I wonder if it would be possible to generate much of the WADL from
>>> introspecting the code itself...surely the URL structure itself can be
>>> extracted from the paste setup, and the XML templates code I recently
>>> contributed could easily be traversed to provide at least a basic
>>> description of the output.  That could at least provide a starting point
>>> for generating WADLs...
>>>
>>> (Of course, I propose this, having little idea of what actually goes in
>>> a WADL, but still... ;)
>>
>> I've worked with WADL and WSDL before, and yes, it is indeed possible to 
>> generate the WADL by introspecting code. (with a few decorators/annotations 
>> assisting)
>> This is what Sandy Walsh is suggesting, and I highly, highly recommend this 
>> approach.  Otherwise you have to either generate code from an external WADL, 
>> which makes the code a mess, or keep the WADL in sync with the code manually 
>> (bleh).  The big advantage of generating WADLs from the code is that you 
>> then get a machine-readable description of what the *code* thinks the 
>> interface is, not what you *hope* the interface is. That way,  if you look 
>> at the generated WADL, and if the interface isn't what it *should* be (as in 
>> "Where did THAT resource come from, and where did the foobar param on that 
>> GET method go?"), you know you have a bug to fix.
>>
>> --
>>    Monsyne M. Dragon
>>    OpenStack/Nova
>>    cell 210-441-0965
>>    work x 5014190
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Nachi Ueno
email:nati.u...@gmail.com
twitter:http://twitter.com/nati

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

As I stated in previous emails, we are pulling data from the WADL to grab 
human-consumable REST API docs that live at 
docs.openstack.org today.  We can certainly expand 
that capability to create a unified API documentation set rather than 
individual guides.  A lot of the hard work for parsing is already done, and 
we'll be releasing a WADL normalizer that puts the WADL in an easer to process 
form.

Joe, I'd love to hear more about what you're trying to accomplish.  Maybe we 
can help you leverage the tools we have to accomplish them.

-jOrGe W.


On Oct 27, 2011, at 10:51 AM, Joseph Heck wrote:

Yeah, that's what I've been poking at and the original start of this rather 
lengthy thread. Unfortunately, WADL, while it appears complete, is rather 
obnoxious for pulling out data. Or more accurately, I haven't fully understood 
the WADL specification in order to write a WADL parser to allow me to do just 
that. I'm poking at it now, but my original goal wasn't to write an XML parser 
but to just create a unified API documentation set on a web site to make it 
easier to consume OpenStack services.

-joe

On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
It would be great if we could do some kind of transform of the IDL to generate 
(some of) the human-consumable REST API documentation that lives at 
docs.openstack.org. That would simplify the task of 
keeping those docs up to date.

Lorin
--
Lorin Hochstein, Computer Scientist
USC Information Sciences Institute
703.812.3710
http://www.east.isi.edu/~lorin


On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
Sounds awesome!

I've done an application like this in the past where an entire web UI was data 
driven using a custom IDL. It had to have presentation hints associated with it 
(acceptable values, display widget, etc). Not something WADL supports 
inherently I'm sure. But, I know from experience this can work.

I don't really care what the IDL is, so long as we don't have to write a parser 
for it in 10 different languages ... which is why XML/JSON hold such appeal 
(although JSON in C keeps me awake at night).

-S


From: Mark Nottingham [m...@mnot.net]
Sent: Thursday, October 27, 2011 10:38 AM
To: Sandy Walsh
Cc: Mellquist, Peter; Joseph Heck; 
openstack@lists.launchpad.net
Subject: Re: [Openstack] +1,  All services should have WADLs

I'm totally on board with having the interface being machine-consumable at 
runtime -- see the previous discussion on versioning and extensibility -- but 
WADL isn't really designed for this. I'm sketching up something more 
appropriate, and will be able to talk about it soon (hopefully).


___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone "Why does it? What does?" questions

2011-10-27 Thread Anne Gentle

Hi Joe -
The place I've found this credential add is in the keystone data script in
devstack. See:

https://github.com/cloudbuilders/devstack/blob/master/files/keystone_data.sh

lines 42 and 43.

Anne
*Anne Gentle*
a...@openstack.org
 my blog  | my
book|
LinkedIn  |
Delicious|
Twitter 
On Thu, Oct 27, 2011 at 11:22 AM, Joseph Heck  wrote:

> On Oct 27, 2011, at 9:19 AM, Vishvananda Ishaya wrote:
> > On Oct 25, 2011, at 12:05 PM, Joseph Heck wrote:
> >> Q: What's the keystone-manage command for "credential add" do? There's
> also no corresponding delete or disable - is this password update for the
> passwords that are set on "keystone-manage user add"? If not, how are those
> passwords updated?
> >>
> >> Q: What are "type" and "key" as related to "credential add" command, and
> what are they intended to do?
> >
> > We use the credential add command to store ec2 credentials.  The idea
> behind credentials could be that a user would have a number of different
> credentials of different types.  Oauth keys, ec2 secret and access, etc.  I
> hacked the ability to add the ec2 credentials in because we needed it to
> make the ec2 api work in nova.  The idea was that more robust support would
> be added through admin api extensions later.
>
> Thanks Vish! Any quick pointers to nova code (or where to find them) so I
> can see how you're using it today? I want to slap this into the docs (which
> are anemic in this area at the moment, since I didn't grok it)
>
> -joe
>
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] nova-compute often stop since of libvirt-bin

2011-10-27 Thread Jesse Andrews

What version of libvirt bin/d/python are you running?

on debian/ubuntu do:

   dpkg -l | grep libvirt

On Thu, Oct 27, 2011 at 8:49 AM, Vishvananda Ishaya
 wrote:
> I haven't seen this happen.  Sounds like it may be a libvirt bug.
> On Oct 27, 2011, at 7:48 AM, mao weijie wrote:
>
> I have found nova-compute often stopped that no update info in log and
> service status is XXX. Then I restart nova-compute, but it is still stop in
> "DEBUG nova.virt.libvirt_conn [-] Connecting to libvirt: qemu:///system from
> (pid=31811) _get_connection /data/nova/nova/virt/libvirt/connection.py:205"
> Everytime I need to stop libvirt-bin and start libvirt-bin, then I can start
> nova-compute successfully.
> Does anyone meet this case?
>
> 非淡泊無以明志，非寧靜無以致遠
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone "Why does it? What does?" questions

Perfect, thanks!

-joe

On Oct 27, 2011, at 9:38 AM, Vishvananda Ishaya wrote:
> See the current trunk of devstack which creates the creds and has an 'openrc' 
> file that is sourceable and allows euca2ools to work again. There is also a 
> bit of code in nova-manage to export users projects and keys to keystone 
> which uses it. We don't really do anything with them in the nova code per se.
> 
> Vish
> 
> On Oct 27, 2011, at 9:22 AM, Joseph Heck wrote:
> 
>> On Oct 27, 2011, at 9:19 AM, Vishvananda Ishaya wrote:
>>> On Oct 25, 2011, at 12:05 PM, Joseph Heck wrote:
 Q: What's the keystone-manage command for "credential add" do? There's 
 also no corresponding delete or disable - is this password update for the 
 passwords that are set on "keystone-manage user add"? If not, how are 
 those passwords updated?
 
 Q: What are "type" and "key" as related to "credential add" command, and 
 what are they intended to do?
>>> 
>>> We use the credential add command to store ec2 credentials.  The idea 
>>> behind credentials could be that a user would have a number of different 
>>> credentials of different types.  Oauth keys, ec2 secret and access, etc.  I 
>>> hacked the ability to add the ec2 credentials in because we needed it to 
>>> make the ec2 api work in nova.  The idea was that more robust support would 
>>> be added through admin api extensions later.
>> 
>> Thanks Vish! Any quick pointers to nova code (or where to find them) so I 
>> can see how you're using it today? I want to slap this into the docs (which 
>> are anemic in this area at the moment, since I didn't grok it)
>> 
>> -joe
>> 
>> 
>> 
>> 
> 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility


On 10/27/2011 11:04 AM, George Reese wrote:

You know what it has to do with API versioning? It has to do with people
proposing bad versioning ideas to support esoteric stuff WHEN THE BASIC
STUFF AIN'T THERE YET.


Dude. Calm down. The fact that somebody proposed something you don't 
agree with doesn't make it a "bad" idea.



curl is the appropriate mechanism for manually interacting with an API
for development purpose. A browser is a very limited tool for
interacting with the HTTP protocol and should not be the boundary of
what an API should support.


When I click on a link in an email or a IM, curl isn't what opens it. 
Since we try to support HATEOAS it may take a sequence of links to get 
to where I want to be. Once I open payload #1, curl doesn't allow me to 
click around. Which is why, I don't think an effort to persuade the 
developers of the world to abandon browsers for curl as the way to 
introspect a RESTful web service isn't likely to get very far.



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone "Why does it? What does?" questions

2011-10-27 Thread Vishvananda Ishaya

See the current trunk of devstack which creates the creds and has an 'openrc' 
file that is sourceable and allows euca2ools to work again. There is also a bit 
of code in nova-manage to export users projects and keys to keystone which uses 
it. We don't really do anything with them in the nova code per se.

Vish

On Oct 27, 2011, at 9:22 AM, Joseph Heck wrote:

> On Oct 27, 2011, at 9:19 AM, Vishvananda Ishaya wrote:
>> On Oct 25, 2011, at 12:05 PM, Joseph Heck wrote:
>>> Q: What's the keystone-manage command for "credential add" do? There's also 
>>> no corresponding delete or disable - is this password update for the 
>>> passwords that are set on "keystone-manage user add"? If not, how are those 
>>> passwords updated?
>>> 
>>> Q: What are "type" and "key" as related to "credential add" command, and 
>>> what are they intended to do?
>> 
>> We use the credential add command to store ec2 credentials.  The idea behind 
>> credentials could be that a user would have a number of different 
>> credentials of different types.  Oauth keys, ec2 secret and access, etc.  I 
>> hacked the ability to add the ec2 credentials in because we needed it to 
>> make the ec2 api work in nova.  The idea was that more robust support would 
>> be added through admin api extensions later.
> 
> Thanks Vish! Any quick pointers to nova code (or where to find them) so I can 
> see how you're using it today? I want to slap this into the docs (which are 
> anemic in this area at the moment, since I didn't grok it)
> 
> -joe
> 
> 
> 
> 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Push vs Polling (from Versioning Thread)

Sent from my iPhone

On Oct 27, 2011, at 11:26, Bryan Taylor  wrote:

> On 10/27/2011 10:36 AM, George Reese wrote:
>
>> #3 Push scales a hell of a lot better than having tools polling a cloud
>> constantly. It doesn't matter whether it is polling the API, polling a
>> feed, or polling a message queue. Polling is one of the most unscalable
>> things you can do in any distributed systems scenario. Calling it a feed
>> doesn't magically solve the problem. Actually, it's quite hard on its
>> own in an IaaS scenario and has scaling issues independent of the
>> polling issue.
>
> I disagree. The web was designed specifically to solve the distributed 
> scaling problem and it's based on HTTP polling. It scales pretty well. The 
> argument against polling not scaling inevitably neglects using caching 
> properly.
>

The web was not designed to deal with a bunch of clients needing to
know about infrastructure changes the instant they happen.

And API data should not be cached. The Rackspace API used to do that,
and it created a mess.

> Push doesn't scaled because it requires the server to know about every client 
> and track conversational state with them.

No, it doesn't. You push changes as they occur to a message queue. A
separate system tracks subscribers and sends them out. There is no
conversational state if done right.

> If you need reliability, this requires persisting that conversational state. 
> In order to allow this to happen you have to have some kind of registration 
> protocol for clients. If some fraction of those clients are flaky, the 
> conversational state tracking will kill you because each client consumes 
> resources and so flaky clients = resource leak.
>
> Push wins when you need very low latency delivery, high message throughput to 
> individual consumers, or server side guarantees of delivery to individual 
> consumers, but not for scaling to a large number of clients in a climate of 
> an elastic infrastructure.
>
>> Push notifications are the only mechanism for solving the scaling issue.
>> You push any changes to a message queue. Agents pick up the changes and
>> send them on to subscriber endpoints. Not that hard.
>
> Not that hard with a few fairly reliable clients. Very hard with a web scale 
> set of unreliable clients while I simultaneously need to scale the back end.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Push vs Polling (from Versioning Thread)


On 10/27/2011 10:36 AM, George Reese wrote:


#3 Push scales a hell of a lot better than having tools polling a cloud
constantly. It doesn't matter whether it is polling the API, polling a
feed, or polling a message queue. Polling is one of the most unscalable
things you can do in any distributed systems scenario. Calling it a feed
doesn't magically solve the problem. Actually, it's quite hard on its
own in an IaaS scenario and has scaling issues independent of the
polling issue.


I disagree. The web was designed specifically to solve the distributed 
scaling problem and it's based on HTTP polling. It scales pretty well. 
The argument against polling not scaling inevitably neglects using 
caching properly.


Push doesn't scaled because it requires the server to know about every 
client and track conversational state with them. If you need 
reliability, this requires persisting that conversational state. In 
order to allow this to happen you have to have some kind of registration 
protocol for clients. If some fraction of those clients are flaky, the 
conversational state tracking will kill you because each client consumes 
resources and so flaky clients = resource leak.


Push wins when you need very low latency delivery, high message 
throughput to individual consumers, or server side guarantees of 
delivery to individual consumers, but not for scaling to a large number 
of clients in a climate of an elastic infrastructure.



Push notifications are the only mechanism for solving the scaling issue.
You push any changes to a message queue. Agents pick up the changes and
send them on to subscriber endpoints. Not that hard.


Not that hard with a few fairly reliable clients. Very hard with a web 
scale set of unreliable clients while I simultaneously need to scale the 
back end.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone "Why does it? What does?" questions

2011-10-27 Thread Vishvananda Ishaya

On Oct 25, 2011, at 12:05 PM, Joseph Heck wrote:

> 
> Q: What's the keystone-manage command for "credential add" do? There's also 
> no corresponding delete or disable - is this password update for the 
> passwords that are set on "keystone-manage user add"? If not, how are those 
> passwords updated?
> 
> Q: What are "type" and "key" as related to "credential add" command, and what 
> are they intended to do?
> 

We use the credential add command to store ec2 credentials.  The idea behind 
credentials could be that a user would have a number of different credentials 
of different types.  Oauth keys, ec2 secret and access, etc.  I hacked the 
ability to add the ec2 credentials in because we needed it to make the ec2 api 
work in nova.  The idea was that more robust support would be added through 
admin api extensions later.

> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone "Why does it? What does?" questions

On Oct 27, 2011, at 9:19 AM, Vishvananda Ishaya wrote:
> On Oct 25, 2011, at 12:05 PM, Joseph Heck wrote:
>> Q: What's the keystone-manage command for "credential add" do? There's also 
>> no corresponding delete or disable - is this password update for the 
>> passwords that are set on "keystone-manage user add"? If not, how are those 
>> passwords updated?
>> 
>> Q: What are "type" and "key" as related to "credential add" command, and 
>> what are they intended to do?
> 
> We use the credential add command to store ec2 credentials.  The idea behind 
> credentials could be that a user would have a number of different credentials 
> of different types.  Oauth keys, ec2 secret and access, etc.  I hacked the 
> ability to add the ec2 credentials in because we needed it to make the ec2 
> api work in nova.  The idea was that more robust support would be added 
> through admin api extensions later.

Thanks Vish! Any quick pointers to nova code (or where to find them) so I can 
see how you're using it today? I want to slap this into the docs (which are 
anemic in this area at the moment, since I didn't grok it)

-joe





___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

You know what it has to do with API versioning? It has to do with people 
proposing bad versioning ideas to support esoteric stuff WHEN THE BASIC STUFF 
AIN'T THERE YET. 

curl is the appropriate mechanism for manually interacting with an API for 
development purpose. A browser is a very limited tool for interacting with the 
HTTP protocol and should not be the boundary of what an API should support.

On Oct 27, 2011, at 10:59 AM, Bryan Taylor wrote:

> On 10/27/2011 08:56 AM, George Reese wrote:
> 
>> THE API SHOULD NOT BE SERVING ATOM CONTENT!!!
> 
> What!? Atom is a fine way to represent a collection. Especially one that is 
> append only.
> 
>> There's a nasty habit within the OpenStack community of trying to boil
>> the ocean. And here we are navel gazing over feeds and crap when the API
>> can't yet support the most basic of functionality.
> 
> What does that have to do with versioning?
> 
>> I've worked with just about every damn cloud API out there. I have a
>> very solid grasp on how cloud APIs get consumed, what people have done
>> right, what people have done wrong, and where we need innovation.
>> 
>> We need innovation in pushing data to API consumers.
> 
> I disagree, but this isn't the thread to debate push vs pull as it has 
> nothing to do with versioning.
> 
>> Version and content desired belong in the headers for request and
>> response. The imaginary crap you are dealing with a) don't require them
>> in a URL unless you are pulling it from the URL bar of a browser, which
>> is NOT an API use case and b) doesn't help you deal with the core
>> functionality of the API that is now OVER A YEAR BEHIND where it should be.
> 
> Developers of the API need a way to see the payloads they are developing 
> against. They need it to understand what the data looks like to consume it in 
> code and they need it to troubleshoot data specific problems. Since our data 
> is URI centric, it's quite reasonable for a developer to expect to put those 
> URIs in a browser and get something useful.
> 
> Whether a particular team has met it's expected roadmap to back it's API with 
> a working implementation seems completely unrelated to the best way to do 
> versioning to maximize the ability of clients to leverage backwards 
> compatible changes.

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility


On 10/27/2011 08:56 AM, George Reese wrote:


THE API SHOULD NOT BE SERVING ATOM CONTENT!!!


What!? Atom is a fine way to represent a collection. Especially one that 
is append only.



There's a nasty habit within the OpenStack community of trying to boil
the ocean. And here we are navel gazing over feeds and crap when the API
can't yet support the most basic of functionality.


What does that have to do with versioning?


I've worked with just about every damn cloud API out there. I have a
very solid grasp on how cloud APIs get consumed, what people have done
right, what people have done wrong, and where we need innovation.

We need innovation in pushing data to API consumers.


I disagree, but this isn't the thread to debate push vs pull as it has 
nothing to do with versioning.



Version and content desired belong in the headers for request and
response. The imaginary crap you are dealing with a) don't require them
in a URL unless you are pulling it from the URL bar of a browser, which
is NOT an API use case and b) doesn't help you deal with the core
functionality of the API that is now OVER A YEAR BEHIND where it should be.


Developers of the API need a way to see the payloads they are developing 
against. They need it to understand what the data looks like to consume 
it in code and they need it to troubleshoot data specific problems. 
Since our data is URI centric, it's quite reasonable for a developer to 
expect to put those URIs in a browser and get something useful.


Whether a particular team has met it's expected roadmap to back it's API 
with a working implementation seems completely unrelated to the best way 
to do versioning to maximize the ability of clients to leverage 
backwards compatible changes.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

Yeah, that's what I've been poking at and the original start of this rather 
lengthy thread. Unfortunately, WADL, while it appears complete, is rather 
obnoxious for pulling out data. Or more accurately, I haven't fully understood 
the WADL specification in order to write a WADL parser to allow me to do just 
that. I'm poking at it now, but my original goal wasn't to write an XML parser 
but to just create a unified API documentation set on a web site to make it 
easier to consume OpenStack services.

-joe

On Oct 27, 2011, at 8:04 AM, Lorin Hochstein wrote:
> It would be great if we could do some kind of transform of the IDL to 
> generate (some of) the human-consumable REST API documentation that lives at 
> docs.openstack.org. That would simplify the task of keeping those docs up to 
> date.
> 
> Lorin
> --
> Lorin Hochstein, Computer Scientist
> USC Information Sciences Institute
> 703.812.3710
> http://www.east.isi.edu/~lorin
> 
> 
> On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:
>> Sounds awesome!
>> 
>> I've done an application like this in the past where an entire web UI was 
>> data driven using a custom IDL. It had to have presentation hints associated 
>> with it (acceptable values, display widget, etc). Not something WADL 
>> supports inherently I'm sure. But, I know from experience this can work.
>> 
>> I don't really care what the IDL is, so long as we don't have to write a 
>> parser for it in 10 different languages ... which is why XML/JSON hold such 
>> appeal (although JSON in C keeps me awake at night).
>> 
>> -S
>> 
>> 
>> From: Mark Nottingham [m...@mnot.net]
>> Sent: Thursday, October 27, 2011 10:38 AM
>> To: Sandy Walsh
>> Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
>> Subject: Re: [Openstack] +1,  All services should have WADLs
>> 
>> I'm totally on board with having the interface being machine-consumable at 
>> runtime -- see the previous discussion on versioning and extensibility -- 
>> but WADL isn't really designed for this. I'm sketching up something more 
>> appropriate, and will be able to talk about it soon (hopefully).
>> 
>> 
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] nova-compute often stop since of libvirt-bin

2011-10-27 Thread Vishvananda Ishaya

I haven't seen this happen.  Sounds like it may be a libvirt bug.
On Oct 27, 2011, at 7:48 AM, mao weijie wrote:

> I have found nova-compute often stopped that no update info in log and 
> service status is XXX. Then I restart nova-compute, but it is still stop in 
> "DEBUG nova.virt.libvirt_conn [-] Connecting to libvirt: qemu:///system from 
> (pid=31811) _get_connection /data/nova/nova/virt/libvirt/connection.py:205"
> Everytime I need to stop libvirt-bin and start libvirt-bin, then I can start 
> nova-compute successfully.
> 
> Does anyone meet this case?
> 
> 
> 非淡泊無以明志，非寧靜無以致遠
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

The complete lack of evolution of the OSAPI combined with the irrational 
resistance to the EC2 API has struck a nerve with me.

#1 Feature coverage in the OSAPI is atrocious. And I don't get the feeling 
there's anyone seriously doing anything about it. Of course, you can always 
say, "George, it's an Open Source project. If you don't like it, feel free to 
fix it." Of course, I'm not worrying about all kinds of bizarre OpenStack 
projects that have nothing to do with building a basic, functional cloud 
platform either.

#2 The Netflix API is not an IaaS or PaaS API. Different problems for different 
audiences and different use cases.

#3 Push scales a hell of a lot better than having tools polling a cloud 
constantly. It doesn't matter whether it is polling the API, polling a feed, or 
polling a message queue. Polling is one of the most unscalable things you can 
do in any distributed systems scenario. Calling it a feed doesn't magically 
solve the problem. Actually, it's quite hard on its own in an IaaS scenario and 
has scaling issues independent of the polling issue.

Push notifications are the only mechanism for solving the scaling issue. You 
push any changes to a message queue. Agents pick up the changes and send them 
on to subscriber endpoints. Not that hard.

#4 The use cases you mention don't demand API versioning in the URI.

#5 For all the reasons I have already outlined, API versioning in the URI is a 
bad idea. I say this, again, as someone who made the dumb mistake of putting 
version in the URI.

#6 Unstructured content does not belong in an API except in select 
circumstances in which you are providing mechanisms for machine to machine 
transfer and the machines themselves don't care about the underlying 
unstructured content. For example, a document repository pulling a PDF report 
from a system and storing it in the doc repository. In such cases, it again 
does not need the meta-data information in the URI. Keep meta-data in 
structures for describing meta-data. The URI is not such a structure.

On EC2 vs OSAPI, I actually have no opinion. I do have an opinion that you pick 
one and make it work. We have a situation in which certain parties don't like 
the one that works (EC2), but they spend a bunch of time doing nothing on their 
alternative solution (OSAPI). And I say this as someone who things the 
structure of the Rackspace API on which OSAPI is based is one of the more 
elegant, well-designed APIs. But a well-designed API with no feature coverage 
isn't an API.

-George

On Oct 27, 2011, at 9:29 AM, Jorge Williams wrote:
> 
> Wow, I seem to have struck a nerve there George which was not my intent.  To 
> add some context, I was under the impression that we're simply having a 
> discussion here about versioning and extensibility in general, in order to 
> help shape our future APIs in a consistent manner.  I don't think that we're 
> re-inventing our current API design, more than discussing how it should 
> evolve. We're also not talking about any particular API (identity, compute, 
> image, object store), but simply discussing things in general in the hopes of 
> moving to some consistency.
> 
> Overall, I like the idea of push but it's difficult to scale and techniques 
> like atom have proven themselves.  I don't think that the use case is 
> extraneous or imaginary, in fact, I presented two APIs that use the 
> technique, today in production, at scale.  In the netflix case, I'm simply 
> pointing out that they go out of their way to support variants, precisely to 
> support the browser case.
> 
> -jOrGe W.
> 
> 

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese

smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility


On 10/26/2011 11:19 PM, Mark Nottingham wrote:

To be truly RESTful at the level of the Fielding article (which I
actually think is the best description of HATEOAS there is) you
shouldn't have these variants at all.  I worry about us trying to
put lipstick on the pig -- all these variants are a crummy
compromise to work around broken browsers that do allow changing
the accepts header.



It's perfectly legitimate to link to a resource that has only one
representation, such as having /foo.html to allow people to
specifically refer to the HTML version of the /foo resource. That's
effectively agent-driven negotiation, and it's just as valid
(RESTful, if you will) as server-driven negotiation. Nothing that Roy
has said contradicts that, because a core principle of REST (if we
really want to go there) is that important things have identity, and
that links are what drive things (agent-driven negotiation is just
linking, really).


It harms visibility, a key goal of REST, especially if we are doing this 
on every resource. If I do a PUT to /foo.xml the intermediaries have no 
way to know that I'm really affecting /foo .


I've made the point before that I can define a resource to be anything I 
want, including a particular representation of another resource. I'm not 
opposed to using variants generally on the web, and Jorge's pdf example 
is pretty much impossible to do without them. I'm just questioning 
whether it makes sense to use them to solve our developer API 
introspection problem within our web service APIs.



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] starting a service with reporting updates to the db

2011-10-27 Thread Craig Vyvial

I noticed that in the Nova Service start method the now=False seem to wait
and update heartbeat of the services after the first interval. I was curious
is there a reason for this? It looks like this would not matter too much but
the fact that the option is available to update now i would think this
should be set to True.

I think the initial thought behind this may have been when starting up the
services it would create the service and there for be available on create
but with the last report time of the service is not available until after
the interval is updated.

pulse.start(interval=self.report_interval, now=False)
periodic.start(interval=self.periodic_interval, now=False)

Should i mark this as a bug and submit a patch for it?
Thoughts?

Thanks, Craig Vyvial
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

2011-10-27 Thread Lorin Hochstein

It would be great if we could do some kind of transform of the IDL to generate 
(some of) the human-consumable REST API documentation that lives at 
docs.openstack.org. That would simplify the task of keeping those docs up to 
date.

Lorin
--
Lorin Hochstein, Computer Scientist
USC Information Sciences Institute
703.812.3710
http://www.east.isi.edu/~lorin




On Oct 27, 2011, at 9:54 AM, Sandy Walsh wrote:

> Sounds awesome!
> 
> I've done an application like this in the past where an entire web UI was 
> data driven using a custom IDL. It had to have presentation hints associated 
> with it (acceptable values, display widget, etc). Not something WADL supports 
> inherently I'm sure. But, I know from experience this can work.
> 
> I don't really care what the IDL is, so long as we don't have to write a 
> parser for it in 10 different languages ... which is why XML/JSON hold such 
> appeal (although JSON in C keeps me awake at night).
> 
> -S
> 
> 
> 
> 
> From: Mark Nottingham [m...@mnot.net]
> Sent: Thursday, October 27, 2011 10:38 AM
> To: Sandy Walsh
> Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
> Subject: Re: [Openstack] +1,  All services should have WADLs
> 
> I'm totally on board with having the interface being machine-consumable at 
> runtime -- see the previous discussion on versioning and extensibility -- but 
> WADL isn't really designed for this. I'm sketching up something more 
> appropriate, and will be able to talk about it soon (hopefully).
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

2011-10-27 Thread Ed Leafe

On Oct 27, 2011, at 8:56 AM, George Reese wrote:

> There's a nasty habit within the OpenStack community of trying to boil the 
> ocean. And here we are navel gazing over feeds and crap when the API can't 
> yet support the most basic of functionality.

+1


-- Ed Leafe


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

2011-10-27 Thread Dolph Mathews

+11

On Thu, Oct 27, 2011 at 8:56 AM, George Reese wrote:

>
> Version and content desired belong in the headers for request and response.
> The imaginary crap you are dealing with a) don't require them in a URL
> unless you are pulling it from the URL bar of a browser, which is NOT an API
> use case and b) doesn't help you deal with the core functionality of the API
> that is now OVER A YEAR BEHIND where it should be.
>
> -George
>
> --
> George Reese - Chief Technology Officer, enStratus
> e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217
>  f: +1.612.338.5041
> enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus
> - http://www.enstratus.com
> To schedule a meeting with me: http://tungle.me/GeorgeReese
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] nova-compute often stop since of libvirt-bin

2011-10-27 Thread mao weijie

I have found nova-compute often stopped that no update info in log and service 
status is XXX. Then I restart nova-compute, but it is still stop in "DEBUG 
nova.virt.libvirt_conn [-] Connecting to libvirt: qemu:///system from 
(pid=31811) _get_connection /data/nova/nova/virt/libvirt/connection.py:205"
Everytime I need to stop libvirt-bin and start libvirt-bin, then I can start 
nova-compute successfully.

Does anyone meet this case?


非淡泊無以明志，非寧靜無以致遠

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

On Oct 27, 2011, at 8:56 AM, George Reese wrote:

On Oct 27, 2011, at 8:11 AM, Jorge Williams wrote:

Response inline:

On Oct 27, 2011, at 12:50 AM, Bryan Taylor wrote:

On 10/26/2011 04:45 PM, Jorge Williams wrote:

On Oct 26, 2011, at 1:19 PM, Bryan Taylor wrote:

So no pdfs or excel spreadsheets without conneg.

But PDFs and excel spreadsheets are precisely why you want variants!

Reports and spreadsheets are presentation layer resources that should come from
control panels and dashboards and not from a web services API layer.

In fact, it's with some reluctance that I even suggested having HTML in the
services layer, but we said an API goal was to target developers eyeballing our
data formats in a browser. HTML is the best media type to use for this,
leveraging the element, perhaps with some syntax highlighting eye candy.

"Here's my usage stats for 2009...

http://usage.api.acme.com/v1.0/jorgew/2009/usage.pdf";

That shouldn't be coming directly from an openstack API.

We're actually building a usage service on top of OpenStack and we don't have
any PDFs in it. Dashboards, control panels, BI systems etc, should host that
resource, not our APIs.

You mean to tell me that I can't send that out as an e-mail? Instead I
have to say

"Please run this command to see my usage stats for 2009

Our use case is to show *developers* what the openstack API payloads look like,
not to deal with arbitrary end user presentation desires.

curl -H "Accept: application/vnd.acme.com+pdf;version=1.0"
http://usage.api.acme.com/jorgew/2009/usage";

That seems silly to me, we're missing an important feature, the ability
to click.

We are adding an important feature by leaving it out: separation of
presentation and data.

In this case you can think of the PDF or excel spreadsheet as simply an
alternate representation of the data. Providing these alternate
representations can lower barriers for a lot of clients and personally I think
they make sense in some cases. It's a pattern I've seen used quite
successfully.

They may be user-centric representations of the data, but from an API
perspective (again, "application programming interface"), any such unstructured
data is just a blob. There may be valid reasons for delivering blobs through
the API (though as Bryan pointed out, in general, there aren't). But when it is
done, it is done irrespective of content type or version of the underlying
content unless that data is accompanied with meta-data. Under no circumstances
is it about the ability to pull that data through a browser.

That said, I'm not to worried about generating PDFs from our current APIs
because we're just not likely to run into that use case.

What I'm more worried about is being able to support things like feeds. A feed
can be an alternate representation of an existing collection of servers, for
example, and here again we have to deal with the browser as a user agent, that
may not participate in the content negotiation process as we'd like. The
element approach your suggesting won't work in this case either.
The current, load balancer, API uses feeds as an alternate representation for
most collection types so that you can track changes, here's an example call.

http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/List_Virtual_IPs-d1e2809.html

The API uses a variant (.atom).

You also see this pattern in stuff like the Netflix API as well See
/users/{user_id}/feeds in:

http://developer.netflix.com/docs/read/REST_API_Reference

Here the parameter output=atom and a (read only) token is placed in the URI as
well, so that one can get access to the feed from a browser.

THE API SHOULD NOT BE SERVING ATOM CONTENT!!!

I am so damn irritated with the state of the OpenStack API.

There's a nasty habit within the OpenStack community of trying to boil the
ocean. And here we are navel gazing over feeds and crap when the API can't yet
support the most basic of functionality.

I've worked with just about every damn cloud API out there. I have a very solid
grasp on how cloud APIs get consumed, what people have done right, what people
have done wrong, and where we need innovation.

We need innovation in pushing data to API consumers.

We don't need people re-inventing API design best practices to suit extraneous
use cases.

Version and content desired belong in the headers for request and response. The
imaginary crap you are dealing with a) don't require them in a URL unless you
are pulling it from the URL bar of a browser, which is NOT an API use case and
b) doesn't help you deal with the core functionality of the API that is now
OVER A YEAR BEHIND where it should be.

-George

Wow, I seem to have struck a nerve there George which was not my intent. To
add some context, I was under the impression that we're simply having a
discussion here about versioning and extensibility in general, in order to help
shape our future APIs in a consistent manner. I don't think

Re: [Openstack] +1, All services should have WADLs

Sounds awesome!

I've done an application like this in the past where an entire web UI was data 
driven using a custom IDL. It had to have presentation hints associated with it 
(acceptable values, display widget, etc). Not something WADL supports 
inherently I'm sure. But, I know from experience this can work.

I don't really care what the IDL is, so long as we don't have to write a parser 
for it in 10 different languages ... which is why XML/JSON hold such appeal 
(although JSON in C keeps me awake at night).

-S




From: Mark Nottingham [m...@mnot.net]
Sent: Thursday, October 27, 2011 10:38 AM
To: Sandy Walsh
Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
Subject: Re: [Openstack] +1,  All services should have WADLs

I'm totally on board with having the interface being machine-consumable at 
runtime -- see the previous discussion on versioning and extensibility -- but 
WADL isn't really designed for this. I'm sketching up something more 
appropriate, and will be able to talk about it soon (hopefully).


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

On Oct 27, 2011, at 8:11 AM, Jorge Williams wrote:

> Response inline:
> 
> On Oct 27, 2011, at 12:50 AM, Bryan Taylor wrote:
> 
>> On 10/26/2011 04:45 PM, Jorge Williams wrote:
>>> 
>>> On Oct 26, 2011, at 1:19 PM, Bryan Taylor wrote:
>>> 
 So no pdfs or excel spreadsheets without conneg.
>>> 
>>> But PDFs and excel spreadsheets are precisely why you want variants!
>> 
>> Reports and spreadsheets are presentation layer resources that should come 
>> from control panels and dashboards and not from a web services API layer.
>> 
>> In fact, it's with some reluctance that I even suggested having HTML  in the 
>> services layer, but we said an API goal was to target developers eyeballing 
>> our data formats in a browser. HTML is the best media type to use for this, 
>> leveraging the  element, perhaps with some syntax highlighting eye 
>> candy.
>> 
>>> "Here's my usage stats for 2009...
>>> 
>>> http://usage.api.acme.com/v1.0/jorgew/2009/usage.pdf";
>> 
>> That shouldn't be coming directly from an openstack API.
>> 
>> We're actually building a usage service on top of OpenStack and we don't 
>> have any PDFs in it. Dashboards, control panels, BI systems etc, should host 
>> that resource, not our APIs.
>> 
>>> You mean to tell me that I can't send that out as an e-mail? Instead I
>>> have to say
>>> 
>>> "Please run this command to see my usage stats for 2009
>> 
>> Our use case is to show *developers* what the openstack API payloads look 
>> like, not to deal with arbitrary end user presentation desires.
>> 
>>> curl -H "Accept: application/vnd.acme.com+pdf;version=1.0"
>>> http://usage.api.acme.com/jorgew/2009/usage";
>> 
>>> That seems silly to me, we're missing an important feature, the ability
>>> to click.
>> 
>> We are adding an important feature by leaving it out: separation of 
>> presentation and data.
>> 
> 
> In this case you can think of the PDF or excel spreadsheet as simply an 
> alternate representation of the data.  Providing these alternate 
> representations can lower barriers for a lot of clients and personally I 
> think they make sense in some cases.  It's a pattern I've seen used quite 
> successfully.
> 

They may be user-centric representations of the data, but from an API 
perspective (again, "application programming interface"), any such unstructured 
data is just a blob. There may be valid reasons for delivering blobs through 
the API (though as Bryan pointed out, in general, there aren't). But when it is 
done, it is done irrespective of content type or version of the underlying 
content unless that data is accompanied with meta-data. Under no circumstances 
is it about the ability to pull that data through a browser.

> That said,  I'm not to worried about generating PDFs from our current APIs 
> because we're just not likely to run into that use case.
> 
> What I'm more worried about is being able to support things like feeds. A 
> feed can be an alternate representation of an existing collection of servers, 
> for example, and here again we have to deal with the browser as a user agent, 
> that may not participate in the content negotiation process as we'd like.  
> The  element approach your suggesting won't work in this case either.
> The current, load balancer, API uses feeds as an alternate representation for 
> most collection types so that you can track changes, here's an example call.
> 
> http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/List_Virtual_IPs-d1e2809.html
> 
> The API uses a variant (.atom).
> 
> You also see this pattern in stuff like the Netflix API as well See 
> /users/{user_id}/feeds in:
> 
> http://developer.netflix.com/docs/read/REST_API_Reference
> 
> Here the parameter output=atom and a (read only) token is placed in the URI 
> as well, so that one can get access to the feed from a browser.

THE API SHOULD NOT BE SERVING ATOM CONTENT!!!

I am so damn irritated with the state of the OpenStack API. 

There's a nasty habit within the OpenStack community of trying to boil the 
ocean. And here we are navel gazing over feeds and crap when the API can't yet 
support the most basic of functionality.

I've worked with just about every damn cloud API out there. I have a very solid 
grasp on how cloud APIs get consumed, what people have done right, what people 
have done wrong, and where we need innovation.

We need innovation in pushing data to API consumers.

We don't need people re-inventing API design best practices to suit extraneous 
use cases.

Version and content desired belong in the headers for request and response. The 
imaginary crap you are dealing with a) don't require them in a URL unless you 
are pulling it from the URL bar of a browser, which is NOT an API use case and 
b) doesn't help you deal with the core functionality of the API that is now 
OVER A YEAR BEHIND where it should be.

-George

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217

Re: [Openstack] +1, All services should have WADLs


On 28/10/2011, at 12:31 AM, Sandy Walsh wrote:

> I think that would be ideal.
> 
> The key to API adoption will come not only from ease-of-use but from 
> availability of language bindings. Manually creating and maintaining the 
> Python binding is well and good, but having to create and maintain bindings 
> for Java, C#, C++, C, Ruby, PHP, etc is going to be a full time job for many 
> parties. Not to mention versioning support.
> 
> The beauty of having a machine consumable format is to automate the changes 
> in the bindings as soon as the WADL (or whatever) changes. All clients move 
> lock-step with the server changes. As I mentioned before, it's even better 
> when the server WADL is auto-generated from the implementation.
> 
> Yes, there are technical issues around this. The greatest being the "sugar" 
> on the user interface. For example, knowing that something is a password or 
> calendar field. Or using the output from one call as input to another. But 
> for basic   ->  type calls, I know this is 
> do-able. At the very least we should be able to create a nice object model 
> for clients to use. 
> 
> Is WADL rich enough to support this? That's what we need to investigate.

The problem is that people are used to using WSDL and WADL as ways to do code 
generation, not using them dynamically at runtime. This makes everything very 
brittle.

I'm totally on board with having the interface being machine-consumable at 
runtime -- see the previous discussion on versioning and extensibility -- but 
WADL isn't really designed for this. I'm sketching up something more 
appropriate, and will be able to talk about it soon (hopefully).

Cheers,


> 
> -S
> 
> 
> From: Mark Nottingham [m...@mnot.net]
> Sent: Thursday, October 27, 2011 10:16 AM
> To: Sandy Walsh
> Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
> Subject: Re: [Openstack] +1,  All services should have WADLs
> 
> On 26/10/2011, at 11:17 PM, Sandy Walsh wrote:
> 
>> As discussed at the summit, I agree there should be some form of IDL (WADL 
>> being the likely candidate for REST), I think manually crafting/maintaining 
>> a WADL (or XML in general) is a fools errand. This stuff is made for machine 
>> consumption and should be machine generated. Whatever solution we adopt, we 
>> should keep that requirement in mind.
> 
> I've stayed out of the discussions about WADL mostly, because up till now 
> they've been centred on creating documentation and running tests. As far as 
> that goes, these are reasonable things to do with it.
> 
> What do you mean by "machine consumption" -- are you saying that you want 
> clients to automatically generate bindings?
> 
> Cheers,
> 
> 
> --
> Mark Nottingham   http://www.mnot.net/
> 
> 
> 

--
Mark Nottingham   http://www.mnot.net/




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs

I think that would be ideal.

The key to API adoption will come not only from ease-of-use but from 
availability of language bindings. Manually creating and maintaining the Python 
binding is well and good, but having to create and maintain bindings for Java, 
C#, C++, C, Ruby, PHP, etc is going to be a full time job for many parties. Not 
to mention versioning support.

The beauty of having a machine consumable format is to automate the changes in 
the bindings as soon as the WADL (or whatever) changes. All clients move 
lock-step with the server changes. As I mentioned before, it's even better when 
the server WADL is auto-generated from the implementation.

Yes, there are technical issues around this. The greatest being the "sugar" on 
the user interface. For example, knowing that something is a password or 
calendar field. Or using the output from one call as input to another. But for 
basic   ->  type calls, I know this is do-able. 
At the very least we should be able to create a nice object model for clients 
to use. 

Is WADL rich enough to support this? That's what we need to investigate.

-S

From: Mark Nottingham [m...@mnot.net]
Sent: Thursday, October 27, 2011 10:16 AM
To: Sandy Walsh
Cc: Mellquist, Peter; Joseph Heck; openstack@lists.launchpad.net
Subject: Re: [Openstack] +1,  All services should have WADLs

On 26/10/2011, at 11:17 PM, Sandy Walsh wrote:

> As discussed at the summit, I agree there should be some form of IDL (WADL 
> being the likely candidate for REST), I think manually crafting/maintaining a 
> WADL (or XML in general) is a fools errand. This stuff is made for machine 
> consumption and should be machine generated. Whatever solution we adopt, we 
> should keep that requirement in mind.

I've stayed out of the discussions about WADL mostly, because up till now 
they've been centred on creating documentation and running tests. As far as 
that goes, these are reasonable things to do with it.

What do you mean by "machine consumption" -- are you saying that you want 
clients to automatically generate bindings?

Cheers,

--
Mark Nottingham   http://www.mnot.net/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility


On 27/10/2011, at 3:19 PM, Mark Nottingham wrote:

>> I floated the idea a while back that we get rid of variants altogether and 
>> instead use an HTML representation to offer the user a choice of how to view 
>> the information that includes  elements with JSON and XML formatted 
>> text. It could also have a nice human readable representation so that users 
>> could directly navigate around in the browser. Remember that the only use 
>> case justifying variants are human developers who want to see what the JSON 
>> or XML will look like.
>> If we do this, then we have GET /server/1234 and everything is clean. Jorge 
>> pointed out one slight problem with this:  can only display textual 
>> representations. So no pdfs or excel spreadsheets without conneg. If we 
>> could live with this, getting rid of the variants altogether seems like a 
>> win.
> 
> 
> Why not just return HTML that uses XHR to pull in the JSON and XML on-demand, 
> when the Accept header prefers HTML? 

Actually, I take it back -- we probably shouldn't be putting any HTML on the 
same host:port as the API, at all -- the cost of a successful XSS against the 
raw API is too high, even if there are ways to mitigate the risk. 

:

> The researchers said Amazon was also vulnerable to cross-site scripting (XSS) 
> attacks that could have allowed users logged onto its online store to hijack 
> an AWS session, using injected JavaScript code. The researchers demonstrated 
> the vulnerability, only possible because signing into Amazon store 
> automatically creates a concurrent AWS cloud service session automatically, 
> at an ACM workshop on cloud security during a presentation entitled All Your 
> Clouds are Belong to us.



--
Mark Nottingham   http://www.mnot.net/




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] API Versioning and Extensibility

Response inline:

On Oct 27, 2011, at 12:50 AM, Bryan Taylor wrote:

> On 10/26/2011 04:45 PM, Jorge Williams wrote:
>> 
>> On Oct 26, 2011, at 1:19 PM, Bryan Taylor wrote:
>> 
>>> So no pdfs or excel spreadsheets without conneg.
>> 
>> But PDFs and excel spreadsheets are precisely why you want variants!
> 
> Reports and spreadsheets are presentation layer resources that should come 
> from control panels and dashboards and not from a web services API layer.
> 
> In fact, it's with some reluctance that I even suggested having HTML  in the 
> services layer, but we said an API goal was to target developers eyeballing 
> our data formats in a browser. HTML is the best media type to use for this, 
> leveraging the  element, perhaps with some syntax highlighting eye candy.
> 
>> "Here's my usage stats for 2009...
>> 
>> http://usage.api.acme.com/v1.0/jorgew/2009/usage.pdf";
> 
> That shouldn't be coming directly from an openstack API.
> 
> We're actually building a usage service on top of OpenStack and we don't have 
> any PDFs in it. Dashboards, control panels, BI systems etc, should host that 
> resource, not our APIs.
> 
>> You mean to tell me that I can't send that out as an e-mail? Instead I
>> have to say
>> 
>> "Please run this command to see my usage stats for 2009
> 
> Our use case is to show *developers* what the openstack API payloads look 
> like, not to deal with arbitrary end user presentation desires.
> 
>> curl -H "Accept: application/vnd.acme.com+pdf;version=1.0"
>> http://usage.api.acme.com/jorgew/2009/usage";
> 
>> That seems silly to me, we're missing an important feature, the ability
>> to click.
> 
> We are adding an important feature by leaving it out: separation of 
> presentation and data.
> 

In this case you can think of the PDF or excel spreadsheet as simply an 
alternate representation of the data.  Providing these alternate 
representations can lower barriers for a lot of clients and personally I think 
they make sense in some cases.  It's a pattern I've seen used quite 
successfully.

That said,  I'm not to worried about generating PDFs from our current APIs 
because we're just not likely to run into that use case.

What I'm more worried about is being able to support things like feeds. A feed 
can be an alternate representation of an existing collection of servers, for 
example, and here again we have to deal with the browser as a user agent, that 
may not participate in the content negotiation process as we'd like.  The  
element approach your suggesting won't work in this case either.

The current, load balancer, API uses feeds as an alternate representation for 
most collection types so that you can track changes, here's an example call.

http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/List_Virtual_IPs-d1e2809.html

The API uses a variant (.atom).

You also see this pattern in stuff like the Netflix API as well See 
/users/{user_id}/feeds in:

http://developer.netflix.com/docs/read/REST_API_Reference

Here the parameter output=atom and a (read only) token is placed in the URI as 
well, so that one can get access to the feed from a browser.

-jOrGe W.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] +1, All services should have WADLs