Re: [openstack-dev] [congress] Jenkins failure

2014-08-19 Thread Tim Hinrichs
Some docs got merged last night.  I think that if you fetch the latest and 
rebase that you’ll end up with conflicts that you need to resolve manually.  
The Readme file has changed a lot, so I’m sure git doesn’t know how to merge 
your changes in.

Tim

On Aug 18, 2014, at 10:21 PM, Rajdeep Dua 
rajdeep@gmail.commailto:rajdeep@gmail.com wrote:

my branch already has the latest changes.
it is not able to merge two rst files hence it failed


On Tue, Aug 19, 2014 at 10:02 AM, Akash Gangil 
akashg1...@gmail.commailto:akashg1...@gmail.com wrote:
This change was unable to be automatically merged with the current state of 
the repository. Please rebase your change and upload a new patchset.

Rebase and commit?


On Tue, Aug 19, 2014 at 12:10 AM, Rajdeep Dua 
rajdeep@gmail.commailto:rajdeep@gmail.com wrote:
One of my CL - updation of  README.rst seems to be failing jenkins

https://review.openstack.org/#/c/114896/1

Any idea how to get this passed?

Thanks
Rajdeep

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Akash

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Policy Enforcement logic

2014-08-21 Thread Tim Hinrichs
Hi Madhu,

For the alpha release (due soon), we’re focusing on just monitoring policy 
violations—we’ve disabled all the enforcement code in master.  (Though we never 
actually hooked up the enforcement policy to the real world, so all Congress 
has ever done is compute what actions to take to enforce policy.)  There’s a 
ton of interest in enforcement, so we’re planning to add enforcement features 
to the beta release.

Tim


On Aug 21, 2014, at 7:07 AM, Madhu Mohan 
mmo...@mvista.commailto:mmo...@mvista.com wrote:

Hi,

I am quite new to the Congress and Openstack as well and this question may seem 
very trivial and basic.

I am trying to figure out the policy enforcement logic,

Can some body help me understand how exactly, a policy enforcement action is 
taken.

From the example policy there is an action defined as:

action(disconnect_network)
nova:network-(vm, network) :- disconnect_network(vm, network)

I assume that this statement when applied would translate to deletion of entry 
in the database.

But, how does this affect the actual setup (i.e) How is this database update 
translated to actual disconnection of the VM from the network.
How does nova know that it has to disconnect the VM from the network ?

Thanks and Regards,
Madhu Mohan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Policy Enforcement logic

2014-08-21 Thread Tim Hinrichs
Hi Jay,

We have a tutorial in review right now.  It should be merged in a couple of 
days.  Thanks for the suggestion!

Tim


On Aug 21, 2014, at 7:54 AM, Jay Lau 
jay.lau@gmail.commailto:jay.lau@gmail.com wrote:

I know that Congress is still under development, but it is better that it can 
provide some info for How to use it just like docker 
https://wiki.openstack.org/wiki/Docker , this might attract more people 
contributing to it.


2014-08-21 22:07 GMT+08:00 Madhu Mohan 
mmo...@mvista.commailto:mmo...@mvista.com:
Hi,

I am quite new to the Congress and Openstack as well and this question may seem 
very trivial and basic.

I am trying to figure out the policy enforcement logic,

Can some body help me understand how exactly, a policy enforcement action is 
taken.

From the example policy there is an action defined as:

action(disconnect_network)
nova:network-(vm, network) :- disconnect_network(vm, network)

I assume that this statement when applied would translate to deletion of entry 
in the database.

But, how does this affect the actual setup (i.e) How is this database update 
translated to actual disconnection of the VM from the network.
How does nova know that it has to disconnect the VM from the network ?

Thanks and Regards,
Madhu Mohan




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Ramp-up strategy

2014-08-21 Thread Tim Hinrichs
Hi Madhu,

We have an end-user tutorial in review right now.  That should help you get 
started understanding the end-to-end flow a bit better.  Look for it to be 
merged today or tomorrow.

Tim



On Aug 21, 2014, at 2:44 AM, Madhu Mohan 
mmo...@mvista.commailto:mmo...@mvista.com wrote:

Hi,

Since a few weeks I am trying to get a hold on Congress code base and 
understand the flow.

Here is a brief summary what I am trying out:

Prepared a dummy client to send the policy strings to congress_server listening 
at the path /policies. This is now changed to v1/policies. I am using 
POST request to send the policy string to the server.

The call to server somehow seems to get converted to an action with the name 
create_policies
Added a new API create_policies in the api model policy_model.py which gets 
the policy string in params.

I am able to call compile.parse() and runtime.initialize() functions from this 
API.
The compilation produces a result in the format below:

Rule(head=[Literal(table=u'error', arguments=[Variable(name=u'vm')], 
negated=False)], body=[Literal(table=u'nova:virtual_machine', 
arguments=[Variable(name=u'vm')],.

I am not really sure about how to go about from here to see the policies 
actually getting applied and monitored.

Any resource or instructions on getting through the code flow will be of great 
help to proceed further.

Thanks in Advance,
Madhu Mohan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] Upcoming alpha release

2014-08-21 Thread Tim Hinrichs
Hi all,

We're aiming for an alpha release of Congress tomorrow (Friday).  If you have a 
spare server and a little time, it’d be great if you could try it out: install 
it, write some policies, run tests, etc.  If you could send some feedback along 
the following lines, that would be helpful.

1. What problems did you run into?  File a bug if you like, or drop me an email.

2. Which operating system did you use, and was the install successful or not?

Here are some docs that we hope are enough to get you started.

- README with install instructions:
https://github.com/stackforge/congress/blob/master/README.rst

- Tutorial in the form of an end-to-end example:
https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst

- Troubleshooting guide:
https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rst

Thanks!
Tim


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Policy Enforcement logic

2014-08-21 Thread Tim Hinrichs
The tutorial is now merged.

https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst



Tim

On Aug 21, 2014, at 3:02 PM, Jay Lau 
jay.lau@gmail.commailto:jay.lau@gmail.com wrote:

Hi Tim,

That's great! Does the tutorial is uploaded to Gerrit for review?

Thanks.


2014-08-21 23:56 GMT+08:00 Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com:
Hi Jay,

We have a tutorial in review right now.  It should be merged in a couple of 
days.  Thanks for the suggestion!

Tim


On Aug 21, 2014, at 7:54 AM, Jay Lau 
jay.lau@gmail.commailto:jay.lau@gmail.com wrote:

I know that Congress is still under development, but it is better that it can 
provide some info for How to use it just like docker 
https://wiki.openstack.org/wiki/Docker , this might attract more people 
contributing to it.


2014-08-21 22:07 GMT+08:00 Madhu Mohan 
mmo...@mvista.commailto:mmo...@mvista.com:
Hi,

I am quite new to the Congress and Openstack as well and this question may seem 
very trivial and basic.

I am trying to figure out the policy enforcement logic,

Can some body help me understand how exactly, a policy enforcement action is 
taken.

From the example policy there is an action defined as:

action(disconnect_network)
nova:network-(vm, network) :- disconnect_network(vm, network)

I assume that this statement when applied would translate to deletion of entry 
in the database.

But, how does this affect the actual setup (i.e) How is this database update 
translated to actual disconnection of the VM from the network.
How does nova know that it has to disconnect the VM from the network ?

Thanks and Regards,
Madhu Mohan




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all] Alpha release of Congress

2014-09-03 Thread Tim Hinrichs
Hi all,

The alpha release of Congress is now available!  We'd love any and all feedback.

Components and Features
- Support for policy monitoring
- Policy engine implementation
- Message-passing architecture
- Drivers for Nova and Neutron
- Devstack integration
- Documentation
- REST API

README (with install instructions):
https://github.com/stackforge/congress/blob/master/README.rst

Tutorial:
https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst

Troubleshooting:
https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rst

Contributors (IRC, Reviews, Code):
Sergio Cazzolato (sjcazzol)
Peter Balland (pballand)
Mohammad Banikazemi (banix)
Rajdeep Dua (rajdeep)
Conner Ferguson (Radu)
Tim Hinrichs (thinrichs)
Gokul Kandiraju (gokul)
Harrison Kelly (harrisonkelly)
Prabhakar Kudva (kudva)
Susanta Nanda (skn)
Sean Roberts (sarob)
Iben Rodriguez (iben)
Aaron Rosen (arosen)
Derick Winkworth (cloudtoad)
Alex Yip (alexsyip)


Sincerely,
The Congress team


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Alpha release of Congress

2014-09-04 Thread Tim Hinrichs
Congress isn’t incubated yet, so it won’t be a part of the official release.

There’s a command-line interface in review right now.  The horizon integration 
will be along not long after.

Tim

On Sep 3, 2014, at 7:28 PM, Baohua Yang 
yangbao...@gmail.commailto:yangbao...@gmail.com wrote:

Congrats!
Will appear in J or K?
And will there's an interface in horizon?
Thanks!


On Thu, Sep 4, 2014 at 5:41 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:
Hi all,

The alpha release of Congress is now available!  We'd love any and all feedback.

Components and Features
- Support for policy monitoring
- Policy engine implementation
- Message-passing architecture
- Drivers for Nova and Neutron
- Devstack integration
- Documentation
- REST API

README (with install instructions):
https://github.com/stackforge/congress/blob/master/README.rsthttps://urldefense.proofpoint.com/v1/url?u=https://github.com/stackforge/congress/blob/master/README.rstk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=6iCMeALTCrx1m6hWlIjoRMUKoRQoAX1IF1NOgy3X2Mo%3D%0As=b457bfe87b47b60df3fc9e8e2e5ebcbaa930693a5eb3d948210792ec428c83bd

Tutorial:
https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rsthttps://urldefense.proofpoint.com/v1/url?u=https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rstk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=6iCMeALTCrx1m6hWlIjoRMUKoRQoAX1IF1NOgy3X2Mo%3D%0As=2780defe2dde27a72faff00ba4120122e9954511d66f9a2b1b802e8d2e00f8c4

Troubleshooting:
https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rsthttps://urldefense.proofpoint.com/v1/url?u=https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rstk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=6iCMeALTCrx1m6hWlIjoRMUKoRQoAX1IF1NOgy3X2Mo%3D%0As=da150867c461b3e743a3e32560417e454fbcbf42f8a2217ecb8576176daae325

Contributors (IRC, Reviews, Code):
Sergio Cazzolato (sjcazzol)
Peter Balland (pballand)
Mohammad Banikazemi (banix)
Rajdeep Dua (rajdeep)
Conner Ferguson (Radu)
Tim Hinrichs (thinrichs)
Gokul Kandiraju (gokul)
Harrison Kelly (harrisonkelly)
Prabhakar Kudva (kudva)
Susanta Nanda (skn)
Sean Roberts (sarob)
Iben Rodriguez (iben)
Aaron Rosen (arosen)
Derick Winkworth (cloudtoad)
Alex Yip (alexsyip)


Sincerely,
The Congress team


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Best wishes!
Baohua
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

2014-09-12 Thread Tim Hinrichs
+1

Tim

On Sep 12, 2014, at 11:47 AM, Sean Roberts seanrobert...@gmail.com wrote:

 +1
 
 ~sean
 
 On Sep 12, 2014, at 11:28 AM, Peter Balland pball...@vmware.com wrote:
 
 Hello,
 
 I would like to nominate Aaron Rosen for the congress-core team.
 
 Aaron has been involved in congress for the last few months providing
 valuable reviews as well as leading the keystone integration and
 congressclient work.
 
 References:
 
 https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress,
 n,z
 
 https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
 ngressclient,n,z
 
 https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
 ss,n,z
 
 Please respond with +1's or any concerns.
 
 Thanks,
 
 Peter
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] [policy] Policy-group relationship

2013-12-17 Thread Tim Hinrichs
Inline.

- Original Message -
| From: Mohammad Banikazemi m...@us.ibm.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Cc: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Tuesday, December 17, 2013 11:42:53 AM
| Subject: Re: [openstack-dev] [neutron] [policy] Policy-group relationship
| 
| 
| 
| 
| Stephen Wong s3w...@midokura.com wrote on 12/15/2013 12:00:32 PM:
| 
|  From: Stephen Wong s3w...@midokura.com
|  To: OpenStack Development Mailing List (not for usage questions)
|  openstack-dev@lists.openstack.org,
|  Date: 12/15/2013 12:04 PM
|  Subject: Re: [openstack-dev] [neutron] [policy] Policy-group
|  relationship
|  
|  Hi Mohammad,
|  
|  Good writeup, one minor comment at the end below (look for
|  [s3wong]).
|  
|  On Thu, Dec 12, 2013 at 3:42 PM, Mohammad Banikazemi
|  m...@us.ibm.com wrote:
|   Continuing the discussion we had earlier today during the Neutron
|   Group
|   Policy weekly meeting [0], I would like to initiate a couple of
|   email
|   threads and follow up on a couple of important issues we need to
|   agree on so
|   we can move forward. In this email thread, I would like to
|   discuss the
|   policy-group relationship.
|   
|   I want to summarize the discussion we had during the meeting [1]
|   and see if
|   we have reached an agreement:
|   
|   There are two models for expressing the relationship between
|   Groups and
|   Policies that were discussed:
|   1- Policies are defined for a source Group and a destination
|   Group
|   2- Groups specify the Policies they provide and the Policies
|   they
|   consume
|   
|   As expressed during the IRC meeting, both models have strong
|   support and we
|   decided to have a resource model that can be used to express both
|   models.
|   The solution we came up with was rather simple:
|   
|   Update the resource model (shown in the attribute tables and the
|   taxonomy in
|   the google doc [2]) such that policy can refer to a list of
|   source Groups
|   and a list of destination Groups.
|   This boils down to having two attributes namely, src_groups and
|   destination_groups (both list of uuid-str type) replacing the
|   current
|   attributes src_group and dest_group, respectively.
|   
|   This change simply allows the support for both models. For
|   supporting model
|   1, specify a single source Group and a single destination Group.
|   For model
|   2, specify the producers of a Policy in the source Group list and
|   specify
|   the consumers of the Policy in the destination Group list.
|  
|  [s3wong] this is interesting. Let's say we have two groups A  B,
|  and
|  A wants to send traffic to B, so in this case, B is providing a
|  policy
|  which A will consume. In your proposal above, I would have to put A
|  in
|  destination group list and B in source group list although the
|  traffic
|  direction is the reverse. Would that cause a bit of a confusion?
|  
| 
| Yeah, that's a good point. Producers are the destination of traffic
| flows.
| So should we have them under destination group? Even that is a bit
| confusing.
| May be we need different names for the two groups.
| 
| -Mohammad
| 

If we're not sure what the 2 groups represent (source and destination), perhaps 
that means we ought to have any number of groups and not assign them names.  A 
policy would then be applied to any number of groups, and it would be up to the 
policy itself to dictate source/destination semantics if that is what the 
groups represent.

For example, if we had a DENY action, it might take several arguments, one of 
which is a source and one of which is a destination.  Then by applying groups 
properly to that DENY action, we could dictate which group is playing the role 
of SOURCE and which group is playing the role of DESTINATION.

Tim

|  Thanks,
|  - Stephen
|  
|  
|   
|   If there is agreement, I will update the taxonomy and the
|   attribute tables
|   in the doc.
|   
|   Best,
|   
|   Mohammad
|   
|   
|   [0] https://wiki.openstack.org/wiki/Meetings/Neutron_Group_Policy
|   [1]
|   http://eavesdrop.openstack.org/meetings/networking_policy/2013/
|  networking_policy.2013-12-12-16.01.log.html
|   [2]
|   https://docs.google.com/document/d/
|  1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit#heading=h.x1h06xqhlo1n
|   (Note the new additions are at the end of the document.)
|   
|   
|   ___
|   OpenStack-dev mailing list
|   OpenStack-dev@lists.openstack.org
|   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
|   
|  
|  ___
|  OpenStack-dev mailing list
|  OpenStack-dev@lists.openstack.org
|  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
|  
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 

Re: [openstack-dev] [neutron][policy] Policy-Rules discussions based on Dec.12 network policy meeting

2013-12-17 Thread Tim Hinrichs


- Original Message -
| From: Prasad Vellanki prasad.vella...@oneconvergence.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Monday, December 16, 2013 2:11:37 PM
| Subject: Re: [openstack-dev] [neutron][policy] Policy-Rules discussions based 
on Dec.12 network policy meeting
| 
| 
| 
| Hi
| Please see inline 
| 
| 
| 
| On Sun, Dec 15, 2013 at 8:49 AM, Stephen Wong  s3w...@midokura.com 
| wrote:
| 
| 
| Hi,
| 
| During Thursday's group-policy meeting[1], there are several
| policy-rules related issues which we agreed should be posted on the
| mailing list to gather community comments / consensus. They are:
| 
| (1) Conflict resolution between policy-rules
| --- a priority field was added to the policy-rules attributes
| list[2]. Is this enough to resolve conflict across policy-rules (or
| even across policies)? Please state cases where a cross policy-rules
| conflict can occur.
| --- conflict resolution was a major discussion point during
| Thursday's meeting - and there was even suggestion on setting
| priority
| on endpoint groups; but I would like to have this email thread
| focused
| on conflict resolution across policy-rules in a single policy first.
| 

There was interest in having a single policy that could include different 
actions so that a single flow might be both redirected and QOSed 
simultaneously.  For me this rules out a total ordering on the policy 
statements.  Here's a proposal that relies on the fact that we're fixing the 
meaning of actions within the language: the language specifies a partial order 
on the *actions*.  For example, DENY takes precedence over ALLOW, so if we both 
ALLOW and DENY, then the conflict resolution dictates DENY wins. But {DENY, 
ALLOW} and QOS and REDIRECT are all unrelated, so there is no problem with a 
policy that both DENYs and QOSes and REDIRECTs.

| (2) Default policy-rule actions
| --- there seems to be consensus from the community that we need to
| establish some basic set of policy-rule actions upon which all
| plugins/drivers would have to support
| --- just to get the discussion going, I am proposing:
| 
| 
| 
| Or should this be a query the plugin for supported actions and thus
| the user knows what functionality the plugin can support. Hence
| there is no default supported list.
| 

I think the important part is that the language defines what the actions mean.  
Whether each plugin supports them all is a different issue.  If the language 
doesn't define the meaning of the actions, there's no way for anyone to use the 
language.  We might be able to write down policies, but we don't know what 
those policies actually mean because 2 plugins might assign very different 
meanings to the same action name.

| 
| 
| a.) action_type: 'security' action: 'allow' | 'drop'
| b.) action_type: 'qos' action: {'qos_class': {'critical' |
| 'low-priority' | 'high-priority' |
| 
| 'low-immediate' | 'high-immediate' |
| 
| 'expedite-forwarding'}
| (a subset of DSCP values - hopefully in language that can
| be well understood by those performing application deployments)
| c.) action_type:'redirect' action: {UUID, [UUID]...}
| (a list of Neutron objects to redirect to, and the list
| should contain at least one element)
| 
| 
| 
| 
| I am not sure making the UUIDs a list of neutron objects or endpoints
| will work well. It seems that it should more higher level such as
| list of services that form a chain. Lets say one forms a chain of
| services, firewall, IPS, LB. It would be tough to expect user to
| derive the neutron ports create a chain of them. It could be a VM
| UUID.
| 
| 

Perhaps we could use our usual group mechanism here and say that the redirect 
action operates on 3 groups: source, destination, and the group to which we 
want to redirect.

| 
| Please discuss. In the document, there is also 'rate-limit' and
| 'policing' for 'qos' type, but those can be optional instead of
| required for now
| 

It would be nice if we had some rationale for deciding which actions to include 
and which to leave out.  Maybe if we found a 
standard/spec/collection-of-use-cases and included exactly the same actions.  
Or if we go with the action-based conflict resolution scheme from (1), we might 
want to think about whether we have at least complementary actions (e.g. ALLOW 
and DENY, WAYPOINT -- route traffic through a group of middleboxes-- and FORBID 
-- prohibit traffic from passing through middleboxes).

| (3) Prasad asked for clarification on 'redirect' action, I propose to
| add the following text to document regarding 'redirect' action:
| 
| 'redirect' action is used to mirror traffic to other destinations
| - destination can be another endpoint group, a service chain, a port,
| or a network. Note that 'redirect' action type can be used with other
| forwarding related action type such as 'security'; therefore, it is
| entirely possible that one can specify {'security':'deny'} and still

[openstack-dev] [neutron] [policy] Complementary or alternative semantics?

2014-01-06 Thread Tim Hinrichs

Over the holidays I realized there's something about the proposed Neutron 
policy API that I don't understand.  Is the proposed API complementary to the 
core API, or is it intended to be an alternative?  By complementary, I mean 
that a user can create a bunch of networks, subnets, and ports and then 
constrain how those things interoperate by writing policy.  By an alternative, 
I mean that a user must choose either networks/subnets/ports or policy, but 
cannot choose both.  I had always assumed we were talking about a complementary 
API but wanted to double-check.

Thanks,
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron] [policy] More conflict resolution

2014-01-23 Thread Tim Hinrichs
Hi all,

Today on the Neutron-policy IRC, we ran into some issues that we wanted all 
your feedback on (and as I was writing this note several more occurred to me). 
It's the problem of conflict resolution: different policies making different 
decisions about what to do.  We have a conflict resolution scheme for when 
conflicts arise within a single policy, but we have yet to solve the problem of 
conflicts arising because different policies make different decisions.  The 
important thing to know is that each policy is attached to a tenant (and a 
single tenant can have multiple policies).

1) We need conflict resolution for the following scenarios.  Suggestions?

- Two policies for a single tenant have a conflict.

- Two policies for different tenants with different Keystone roles have a 
conflict.  For example, we expect that admin policies should supersede 
non-admin policies (i.e. that if the admin policy makes a decision to either 
allow or drop a packet, then the non-admin's policy is ignored; otherwise, the 
non-admin's policy makes the final decision).  Are there other roles we need to 
think about?

- Two policies for different tenants with the same Keystone roles have a 
conflict.  For example, if there are two admins whose policies conflict, which 
one wins?

2) Do we perform conflict resolution at the time policies are created (i.e. at 
the API level) or do we resolve conflicts more dynamically at run-time?  

To me, API-level conflict resolution doesn't seem like a good option.  Suppose 
a non-admin writes a perfectly reasonable policy.  Then a month later an admin 
writes a policy that conflicts with the non-admin policy.  There's no way to 
reject the non-admin's policy at this point (and we can't reject the admin's 
policy).  It seems the best we can do is inform the non-admin (via email?) that 
her policy has been overruled.  But if we do that, it may be possible for a 
tenant to learn what the admin's policy is--whether that is a security problem 
or not I don't know.

3) What do we do when roles change, e.g. a non-admin user gets promoted to an 
admin user or vice versa?  And how do we find out about role changes if the 
users do not log in after their policies have been created?  That is, 
role-changes seem to affect the overall policy that is enforced at any point in 
time and thus role-changes ought to be factored into policy enforcement.  

Role-changes make me even more dubious that API-level conflict resolution is a 
good choice.


Hopefully that's a reasonable summary.  Others will chime in where not.

Thoughts?
Thanks,
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] [policy] More conflict resolution

2014-01-29 Thread Tim Hinrichs
 to predict how the policy changes will 
affect the network.  Demoting admin rules to non-admin priority has a similar 
effect.  Hmmm... leaving the priorities the same seems problematic, but 
changing the priorities seems problematic too; not sure what to do here.

Tim

| 
| my 2 cents,
| Manuel
| 
| From: Tim Hinrichs [thinri...@vmware.com]
| Sent: Thursday, January 23, 2014 6:57 PM
| To: OpenStack Development Mailing List (not for usage questions)
| Subject: [openstack-dev] [neutron] [policy] More conflict resolution
| 
| Hi all,
| 
| Today on the Neutron-policy IRC, we ran into some issues that we
| wanted all your feedback on (and as I was writing this note several
| more occurred to me). It's the problem of conflict resolution:
| different policies making different decisions about what to do.  We
| have a conflict resolution scheme for when conflicts arise within a
| single policy, but we have yet to solve the problem of conflicts
| arising because different policies make different decisions.  The
| important thing to know is that each policy is attached to a tenant
| (and a single tenant can have multiple policies).
| 
| 1) We need conflict resolution for the following scenarios.
|  Suggestions?
| 
| - Two policies for a single tenant have a conflict.
| 
| - Two policies for different tenants with different Keystone roles
| have a conflict.  For example, we expect that admin policies should
| supersede non-admin policies (i.e. that if the admin policy makes a
| decision to either allow or drop a packet, then the non-admin's
| policy is ignored; otherwise, the non-admin's policy makes the final
| decision).  Are there other roles we need to think about?
| 
| - Two policies for different tenants with the same Keystone roles
| have a conflict.  For example, if there are two admins whose
| policies conflict, which one wins?
| 
| 2) Do we perform conflict resolution at the time policies are created
| (i.e. at the API level) or do we resolve conflicts more dynamically
| at run-time?
| 
| To me, API-level conflict resolution doesn't seem like a good option.
|  Suppose a non-admin writes a perfectly reasonable policy.  Then a
| month later an admin writes a policy that conflicts with the
| non-admin policy.  There's no way to reject the non-admin's policy
| at this point (and we can't reject the admin's policy).  It seems
| the best we can do is inform the non-admin (via email?) that her
| policy has been overruled.  But if we do that, it may be possible
| for a tenant to learn what the admin's policy is--whether that is a
| security problem or not I don't know.
| 
| 3) What do we do when roles change, e.g. a non-admin user gets
| promoted to an admin user or vice versa?  And how do we find out
| about role changes if the users do not log in after their policies
| have been created?  That is, role-changes seem to affect the overall
| policy that is enforced at any point in time and thus role-changes
| ought to be factored into policy enforcement.
| 
| Role-changes make me even more dubious that API-level conflict
| resolution is a good choice.
| 
| 
| Hopefully that's a reasonable summary.  Others will chime in where
| not.
| 
| Thoughts?
| Thanks,
| Tim
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=GCQOsyh5WyZr4Lx2t%2Bs5Kt6S9fWkBV9dfeL%2B1xUUBag%3D%0As=c06f67fbd826026f4abc79fda10240d25bd756e237f4c121a43e06de8cdfc5e1
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=GCQOsyh5WyZr4Lx2t%2Bs5Kt6S9fWkBV9dfeL%2B1xUUBag%3D%0As=c06f67fbd826026f4abc79fda10240d25bd756e237f4c121a43e06de8cdfc5e1
| 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] IRC meeting moved

2014-02-25 Thread Tim Hinrichs
Hi all,

Last week we moved the Congress IRC meeting time to every other Tuesday at 
17:00 UTC in #openstack-meeting-3 (that's an hour earlier than it was 
previously).  But we neglected to mail out the new time, and it doesn't look 
like anyone remembered the time change.  So I'll hang around at both our old 
and new time slots this week.

Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] status and feedback

2014-02-25 Thread Tim Hinrichs
Hi all,

I wanted to send out a quick status update on Congress and start a discussion 
about short-term goals.

1) Logistics

IRC Meeting time
Tuesday 1700 UTC in openstack-meeting-3
Every other week starting Feb 25
(Note this is a new meeting time/location.)

2) We have two design docs, which we would like feedback on.

Toplevel design doc:
https://docs.google.com/a/vmware.com/document/d/1f2xokl9Tc47aV67KEua0PBRz4jsdSDLXth7dYe-jz6Q/edit

Data integration design doc:
https://docs.google.com/document/d/1K9RkQuBSPN7Z2TmKfok7mw3E24otEGo8Pnsemxd5544/edit

3) Short term goals.

I think it's useful to get an end-to-end system up and running to make it 
easier to communicate what we're driving at.  To that end, I'm suggesting we 
take the following policy and build up enough of Congress to make it work.

Every network connected to a VM must either be public or owned by someone in 
the same group as the VM owner

This example is compelling because it combines information from Neutron, Nova, 
and Keystone (or another group-management data-source such as ActiveDirectory). 
 To that end, I suggest focusing on the following tasks in the short-term.

- Data integration framework, including read-only support for 
Neutron/Nova/Keystone
- Exposing policy engine via API
- Policy engine error handling
- Simple scaling tests
- Basic user docs
- Execution framework: it would be nice if we could actually execute some of 
the actions we tell Congress about, but this is lowest priority on the list, 
for me.

Thoughts?
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

2014-02-25 Thread Tim Hinrichs
Hi Jay,

The Congress project aims to handle something similar to your use cases.  I 
just sent a note to the ML with a Congress status update with the tag 
[Congress].  It includes links to our design docs.  Let me know if you have 
trouble finding it or want to follow up.

Tim

- Original Message -
| From: Sylvain Bauza sylvain.ba...@gmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Tuesday, February 25, 2014 3:58:07 PM
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage
| compute/storage resource
| 
| 
| 
| Hi Jay,
| 
| 
| Currently, the Nova scheduler only acts upon user request (either
| live migration or boot an instance). IMHO, that's something Gantt
| should scope later on (or at least there could be some space within
| the Scheduler) so that Scheduler would be responsible for managing
| resources on a dynamic way.
| 
| 
| I'm thinking of the Pets vs. Cattles analogy, and I definitely think
| that Compute resources could be treated like Pets, provided the
| Scheduler does a move.
| 
| 
| -Sylvain
| 
| 
| 
| 2014-02-26 0:40 GMT+01:00 Jay Lau  jay.lau@gmail.com  :
| 
| 
| 
| 
| Greetings,
| 
| 
| Here I want to bring up an old topic here and want to get some input
| from you experts.
| 
| 
| Currently in nova and cinder, we only have some initial placement
| polices to help customer deploy VM instance or create volume storage
| to a specified host, but after the VM or the volume was created,
| there was no policy to monitor the hypervisors or the storage
| servers to take some actions in the following case:
| 
| 
| 1) Load Balance Policy: If the load of one server is too heavy, then
| probably we need to migrate some VMs from high load servers to some
| idle servers automatically to make sure the system resource usage
| can be balanced.
| 
| 2) HA Policy: If one server get down for some hardware failure or
| whatever reasons, there is no policy to make sure the VMs can be
| evacuated or live migrated (Make sure migrate the VM before server
| goes down) to other available servers to make sure customer
| applications will not be affect too much.
| 
| 3) Energy Saving Policy: If a single host load is lower than
| configured threshold, then low down the frequency of the CPU to save
| energy; otherwise, increase the CPU frequency. If the average load
| is lower than configured threshold, then shutdown some hypervisors
| to save energy; otherwise, power on some hypervisors to load
| balance. Before power off a hypervisor host, the energy policy need
| to live migrate all VMs on the hypervisor to other available
| hypervisors; After Power on a hypervisor host, the Load Balance
| Policy will help live migrate some VMs to the new powered
| hypervisor.
| 
| 4) Customized Policy: Customer can also define some customized
| policies based on their specified requirement.
| 
| 5) Some run-time policies for block storage or even network.
| 
| 
| 
| I borrow the idea from VMWare DRS (Thanks VMWare DRS), and there
| indeed many customers want such features.
| 
| 
| 
| I have filed a bp here [1] long ago, but after some discussion with
| Russell, we think that this should not belong to nova but other
| projects. Till now, I did not find a good place where we can put
| this in, can any of you show some comments?
| 
| 
| 
| [1]
| https://blueprints.launchpad.net/nova/+spec/resource-optimization-service
| 
| --
| 
| 
| Thanks,
| 
| Jay
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
| 
| 
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=XDB3hT4WE2iDrNVK0sQ8qKooX2r1T4E%2BVHek3GREhnE%3D%0As=e2346cd017c9d8108c12a101892492e2ac75953e4a5ea5c17394c775cf086d7f
| 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

2014-02-26 Thread Tim Hinrichs
Hi Jay and Sylvain,

The solver-scheduler sounds like a good fit to me as well.  It clearly 
provisions resources in accordance with policy.  Does it monitor those 
resources and adjust them if the system falls out of compliance with the policy?

I mentioned Congress for two reasons. (i) It does monitoring.  (ii) There was 
mention of compute, networking, and storage, and I couldn't tell if the idea 
was for policy that spans OS components or not.  Congress was designed for 
policies spanning OS components.

Tim

- Original Message -
| From: Jay Lau jay.lau@gmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Tuesday, February 25, 2014 10:13:14 PM
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage
| compute/storage resource
| 
| 
| 
| 
| 
| Thanks Sylvain and Tim for the great sharing.
| 
| @Tim, I also go through with Congress and have the same feeling with
| Sylvai, it is likely that Congress is doing something simliar with
| Gantt providing a holistic way for deploying. What I want to do is
| to provide some functions which is very similar with VMWare DRS that
| can do some adaptive scheduling automatically.
| 
| @Sylvain, can you please show more detail for what Pets vs. Cattles
| analogy means?
| 
| 
| 
| 
| 2014-02-26 9:11 GMT+08:00 Sylvain Bauza  sylvain.ba...@gmail.com  :
| 
| 
| 
| Hi Tim,
| 
| 
| As per I'm reading your design document, it sounds more likely
| related to something like Solver Scheduler subteam is trying to
| focus on, ie. intelligent agnostic resources placement on an
| holistic way [1]
| IIRC, Jay is more likely talking about adaptive scheduling decisions
| based on feedback with potential counter-measures that can be done
| for decreasing load and preserving QoS of nodes.
| 
| 
| That said, maybe I'm wrong ?
| 
| 
| [1] https://blueprints.launchpad.net/nova/+spec/solver-scheduler
| 
| 
| 
| 2014-02-26 1:09 GMT+01:00 Tim Hinrichs  thinri...@vmware.com  :
| 
| 
| 
| 
| Hi Jay,
| 
| The Congress project aims to handle something similar to your use
| cases. I just sent a note to the ML with a Congress status update
| with the tag [Congress]. It includes links to our design docs. Let
| me know if you have trouble finding it or want to follow up.
| 
| Tim
| 
| 
| 
| - Original Message -
| | From: Sylvain Bauza  sylvain.ba...@gmail.com 
| | To: OpenStack Development Mailing List (not for usage questions)
| |  openstack-dev@lists.openstack.org 
| | Sent: Tuesday, February 25, 2014 3:58:07 PM
| | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal
| | for OpenStack run time policy to manage
| | compute/storage resource
| | 
| | 
| | 
| | Hi Jay,
| | 
| | 
| | Currently, the Nova scheduler only acts upon user request (either
| | live migration or boot an instance). IMHO, that's something Gantt
| | should scope later on (or at least there could be some space within
| | the Scheduler) so that Scheduler would be responsible for managing
| | resources on a dynamic way.
| | 
| | 
| | I'm thinking of the Pets vs. Cattles analogy, and I definitely
| | think
| | that Compute resources could be treated like Pets, provided the
| | Scheduler does a move.
| | 
| | 
| | -Sylvain
| | 
| | 
| | 
| | 2014-02-26 0:40 GMT+01:00 Jay Lau  jay.lau@gmail.com  :
| | 
| | 
| | 
| | 
| | Greetings,
| | 
| | 
| | Here I want to bring up an old topic here and want to get some
| | input
| | from you experts.
| | 
| | 
| | Currently in nova and cinder, we only have some initial placement
| | polices to help customer deploy VM instance or create volume
| | storage
| | to a specified host, but after the VM or the volume was created,
| | there was no policy to monitor the hypervisors or the storage
| | servers to take some actions in the following case:
| | 
| | 
| | 1) Load Balance Policy: If the load of one server is too heavy,
| | then
| | probably we need to migrate some VMs from high load servers to some
| | idle servers automatically to make sure the system resource usage
| | can be balanced.
| | 
| | 2) HA Policy: If one server get down for some hardware failure or
| | whatever reasons, there is no policy to make sure the VMs can be
| | evacuated or live migrated (Make sure migrate the VM before server
| | goes down) to other available servers to make sure customer
| | applications will not be affect too much.
| | 
| | 3) Energy Saving Policy: If a single host load is lower than
| | configured threshold, then low down the frequency of the CPU to
| | save
| | energy; otherwise, increase the CPU frequency. If the average load
| | is lower than configured threshold, then shutdown some hypervisors
| | to save energy; otherwise, power on some hypervisors to load
| | balance. Before power off a hypervisor host, the energy policy need
| | to live migrate all VMs on the hypervisor to other available
| | hypervisors; After Power on a hypervisor host

Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

2014-02-28 Thread Tim Hinrichs
Hi Jay,

I think the Solver Scheduler is a better fit for your needs than Congress 
because you know what kinds of constraints and enforcement you want.  I'm not 
sure this topic deserves an entire design session--maybe just talking a bit at 
the summit would suffice (I *think* I'll be attending).

Tim

- Original Message -
| From: Jay Lau jay.lau@gmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Wednesday, February 26, 2014 6:30:54 PM
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage
| compute/storage resource
| 
| 
| 
| 
| 
| 
| Hi Tim,
| 
| I'm not sure if we can put resource monitor and adjust to
| solver-scheduler (Gantt), but I have proposed this to Gantt design
| [1], you can refer to [1] and search jay-lau-513.
| 
| IMHO, Congress does monitoring and also take actions, but the actions
| seems mainly for adjusting single VM network or storage. It did not
| consider migrating VM according to hypervisor load.
| 
| Not sure if this topic deserved to be a design session for the coming
| summit, but I will try to propose.
| 
| 
| 
| 
| [1] https://etherpad.openstack.org/p/icehouse-external-scheduler
| 
| 
| 
| Thanks,
| 
| 
| Jay
| 
| 
| 
| 2014-02-27 1:48 GMT+08:00 Tim Hinrichs  thinri...@vmware.com  :
| 
| 
| Hi Jay and Sylvain,
| 
| The solver-scheduler sounds like a good fit to me as well. It clearly
| provisions resources in accordance with policy. Does it monitor
| those resources and adjust them if the system falls out of
| compliance with the policy?
| 
| I mentioned Congress for two reasons. (i) It does monitoring. (ii)
| There was mention of compute, networking, and storage, and I
| couldn't tell if the idea was for policy that spans OS components or
| not. Congress was designed for policies spanning OS components.
| 
| 
| Tim
| 
| - Original Message -
| 
| | From: Jay Lau  jay.lau@gmail.com 
| | To: OpenStack Development Mailing List (not for usage questions)
| |  openstack-dev@lists.openstack.org 
| 
| 
| | Sent: Tuesday, February 25, 2014 10:13:14 PM
| | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal
| | for OpenStack run time policy to manage
| | compute/storage resource
| | 
| | 
| | 
| | 
| | 
| | Thanks Sylvain and Tim for the great sharing.
| | 
| | @Tim, I also go through with Congress and have the same feeling
| | with
| | Sylvai, it is likely that Congress is doing something simliar with
| | Gantt providing a holistic way for deploying. What I want to do is
| | to provide some functions which is very similar with VMWare DRS
| | that
| | can do some adaptive scheduling automatically.
| | 
| | @Sylvain, can you please show more detail for what Pets vs.
| | Cattles
| | analogy means?
| | 
| | 
| | 
| | 
| | 2014-02-26 9:11 GMT+08:00 Sylvain Bauza  sylvain.ba...@gmail.com 
| | :
| | 
| | 
| | 
| | Hi Tim,
| | 
| | 
| | As per I'm reading your design document, it sounds more likely
| | related to something like Solver Scheduler subteam is trying to
| | focus on, ie. intelligent agnostic resources placement on an
| | holistic way [1]
| | IIRC, Jay is more likely talking about adaptive scheduling
| | decisions
| | based on feedback with potential counter-measures that can be done
| | for decreasing load and preserving QoS of nodes.
| | 
| | 
| | That said, maybe I'm wrong ?
| | 
| | 
| | [1] https://blueprints.launchpad.net/nova/+spec/solver-scheduler
| | 
| | 
| | 
| | 2014-02-26 1:09 GMT+01:00 Tim Hinrichs  thinri...@vmware.com  :
| | 
| | 
| | 
| | 
| | Hi Jay,
| | 
| | The Congress project aims to handle something similar to your use
| | cases. I just sent a note to the ML with a Congress status update
| | with the tag [Congress]. It includes links to our design docs. Let
| | me know if you have trouble finding it or want to follow up.
| | 
| | Tim
| | 
| | 
| | 
| | - Original Message -
| | | From: Sylvain Bauza  sylvain.ba...@gmail.com 
| | | To: OpenStack Development Mailing List (not for usage
| | | questions)
| | |  openstack-dev@lists.openstack.org 
| | | Sent: Tuesday, February 25, 2014 3:58:07 PM
| | | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A
| | | proposal
| | | for OpenStack run time policy to manage
| | | compute/storage resource
| | | 
| | | 
| | | 
| | | Hi Jay,
| | | 
| | | 
| | | Currently, the Nova scheduler only acts upon user request (either
| | | live migration or boot an instance). IMHO, that's something Gantt
| | | should scope later on (or at least there could be some space
| | | within
| | | the Scheduler) so that Scheduler would be responsible for
| | | managing
| | | resources on a dynamic way.
| | | 
| | | 
| | | I'm thinking of the Pets vs. Cattles analogy, and I definitely
| | | think
| | | that Compute resources could be treated like Pets, provided the
| | | Scheduler does a move.
| | | 
| | | 
| | | -Sylvain
| | | 
| | | 
| | | 
| | | 2014-02-26 0:40 GMT+01

Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

2014-03-05 Thread Tim Hinrichs
Hi Gokul, 

Thanks for working out how all these policy initiatives relate to each other. 
I'll be spending some time diving into the ones I hadn't heard about. 

I made some additional comments about Congress below. 

Tim 

- Original Message -

From: Jay Lau jay.lau@gmail.com 
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org 
Sent: Wednesday, March 5, 2014 7:31:55 AM 
Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage compute/storage resource 

Hi Gokul, 




2014-03-05 3:30 GMT+08:00 Gokul Kandiraju  gokul4o...@gmail.com  : 





Dear All, 

We are working on a framework where we want to monitor the system and take 
certain actions when specific events or situations occur. Here are two examples 
of ‘different’ situations: 

Example 1: A VM’s-Owner and N/W’s-owner are different == this could mean a 
violation == we need to take some action 

Example 2: A simple policy such as (VM-migrate of all VMs on possible node 
failure) OR (a more complex Energy Policy that may involve optimization). 

Both these examples need monitoring and actions to be taken when certain events 
happen (or through polling). However, the first one falls into the Compliance 
domain with Boolean conditions getting evaluated while the second one may 
require a more richer set of expression allowing for sequences or algorithms. 





So far, based on this discussion, it seems that these are *separate* 
initiatives in the community. I am understanding the Congress project to be in 
the domain of Boolean conditions (used for Compliance, etc.) where as the 
Run-time-policies (Jay's proposal) where policies can be expressed as rules, 
algorithms with higher-level goals. Is this understanding correct? 

Also, looking at all the mails, this is what I am reading: 

1. Congress -- Focused on Compliance [ is that correct? ] (Boolean constraints 
and logic) 



[Tim] Your characterization of boolean constraints for Congress is probably a 
good one. Congress won't be solving optimization/numeric problems any time soon 
if ever. However, I could imagine that down the road we could tell Congress 
here's the policy (optimization or Boolean) that we want to enforce, and it 
would carve off say the Load-balancing part of the policy and send it to the 
Runtime-Policies component; or it would carve off the placement policy and send 
it to the SolverScheduler. Not saying I know how to do this today, but that's 
always been part of the goal for Congress: to have a central point for admins 
to control the global policy being enforced throughout the datacenter/cloud. 

The other delta here is that the Congress policy language is general-purpose, 
so there's not a list of policy types that it will handle (Load Balancing, 
Placement, Energy). That generality comes with a price: that Congress must rely 
on other enforcement points, such as the ones below, to handle complicated 
policy enforcement problems. 



blockquote



2. Runtime-Policies -- Jay’s mail -- Focused on Runtime policies for Load 
Balancing, Availability, Energy, etc. (sequences of actions, rules, algorithms) 

/blockquote

[Jay] Yes, exactly. 

blockquote





3. SolverScheduler -- Focused on Placement [ static or runtime ] and will be 
invoked by the (above) policy engines 




4. Gantt – Focused on (Holistic) Scheduling 

/blockquote

[Jay] For 3 and 4, I was always thinking Gantt is doing something for 
implementing SolverScheduler, not sure if run time policy can be included. 

blockquote





5. Neat -- seems to be a special case of Runtime-Policies (policies based on 
Load) 



Would this be correct understanding? We need to understand this to contribute 
to the right project. :) 



Thanks! 

-Gokul 


On Fri, Feb 28, 2014 at 5:46 PM, Jay Lau  jay.lau@gmail.com  wrote: 

blockquote

Hi Yathiraj and Tim, 

Really appreciate your comments here ;-) 

I will prepare some detailed slides or documents before summit and we can have 
a review then. It would be great if OpenStack can provide DRS features. 

Thanks, 

Jay 



2014-03-01 6:00 GMT+08:00 Tim Hinrichs  thinri...@vmware.com  : 


blockquote
Hi Jay, 

I think the Solver Scheduler is a better fit for your needs than Congress 
because you know what kinds of constraints and enforcement you want. I'm not 
sure this topic deserves an entire design session--maybe just talking a bit at 
the summit would suffice (I *think* I'll be attending). 

Tim 

- Original Message - 
| From: Jay Lau  jay.lau@gmail.com  
| To: OpenStack Development Mailing List (not for usage questions)  
openstack-dev@lists.openstack.org  
| Sent: Wednesday, February 26, 2014 6:30:54 PM 
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage 
| compute/storage resource 
| 
| 
| 
| 
| 
| 
| Hi Tim, 
| 
| I'm not sure if we can put resource monitor and adjust to 
| solver-scheduler

[openstack-dev] [Congress] Policy types

2014-03-12 Thread Tim Hinrichs
Hi all, 

We started a discussion on IRC yesterday that I'd like to continue. The main 
question is what kind of policy does a Congress user actually write? I can see 
three options. The first two focus on actions (API calls that make changes to 
the state of the cloud) and the last focuses on just the cloud state. (By 
state of the cloud I mean all the information Congress can see about all the 
cloud services it is managing, e.g. all the information we can get through API 
calls to Nova, Neutron, Cinder, Heat, ...). 

1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed by 
other cloud services (for each state of the cloud) 
2) Condition Action: which *actions* Congress should execute (for each state of 
the cloud) 
3) Classification (currently supported in Congress): which *states* violate 
real-world policy. [For those of you who have read docs/white-papers/etc. I'm 
using Classification in this note to mean the combination of the current 
Classification and Action Description policies.] 

The important observation is that each of these policies could contain 
different information from each of the others. 

- Access Control vs Condition Action. The Access Control policy tells *other 
cloud services* which actions they are *allowed* to execute. The Condition 
Action policy tells *Congress* which actions it *must* execute. These policies 
differ because they constrain different sets of cloud services. 

- Access Control vs. Classification. The Access Control policy might permit 
some users to violate the Classification policy in some situations (e.g. to fix 
violation A, we might need to cause violation B before eliminating both). These 
policies differ because a violation in one policy might be be a violation in 
the other. 

- Classification vs. Condition Action. The Classification policy might imply 
which actions *could* eliminate a given violation, but the Condition Action 
policy would dictate which of those actions *should* be executed (e.g. the 
Classification policy might tell us that disconnecting a network and deleting a 
VM would both eliminate a particular violation, but the Condition Action policy 
would tell us which to choose). And the Condition Action policy need not 
eliminate all the violations present in the Classification policy. Again these 
policies differ because a violation in one policy might not be a violation in 
the other. 

I'm proposing that for the first release of Congress we support all 3 of these 
policies. When a user inserts/deletes a policy statement, she chooses which 
policy it belongs to. All would be written in basically the same syntax but 
would be used in 3 different scenarios: 

- Prevention: If a component wants to consult Congress before taking action to 
see if that action is allowed, Congress checks the Access Control policy. 

- Reaction: When Congress learns of a change in the cloud's state, it checks 
the Condition Action policy to see which actions should be executed (if any). 

- Monitoring: If a user wants to simply check if the cloud's state is in 
compliance and monitor compliance over time, she writes and queries the 
Classification policy. 

There are several benefits to this proposal. 
- It allows users to choose any of the policy types, if they only want one of 
them. From our discussions with potential users, most seem to want one of these 
3 policy types (and are uninterested in the others). 
- It makes the introduction to Congress relatively simple. We describe 3 
different uses of policy (Prevention, Reaction, Monitoring) and then explain 
which policy to use in which case. 
- This allows us to focus on implementing a single policy-engine technology (a 
Datalog policy language and evaluation algorithms), which gives us the 
opportunity to make it solid. 

There are drawbacks to this proposal as well. 
- We will have 3 separate policies that are conceptually very similar. As the 
policies grow larger, it will become increasingly difficult to keep the 
policies synchronized. This problem can be mitigated to some extent by having 
all 3 share a library of policy statements that they all apply in different 
ways (and such a library mechanism is already implemented). 
- As cloud services change their behavior, policies may need to be re-written. 
For example, right now Nova does not consult Congress before creating a VM; 
thus, to enforce policy surrounding VMs, the best we could do is write a 
Condition-Action policy that adjusts VM configuration when it learns about new 
VMs being created. If we later make Nova consult with Congress before creating 
a VM, we need to write an Access-control policy that puts the proper controls 
in place. 

These drawbacks were the original motivation for supporting only the 
Classification policy and attempting to derive the Access Control and Condition 
Action policies from it. But given that we can't always derive the proper 
Access Control and Condition Action policies from the 

Re: [openstack-dev] [Congress] Policy types

2014-03-12 Thread Tim Hinrichs
Hi Prabhakar, 

Thanks for the feedback. I'd be interested to hear what other policy types you 
have in mind. 

To answer your questions... 

We're planning on extending our policy language in such a way that you can use 
Python functions as conditions (atom in the grammar) in rules. That's on my 
todo-list but didn't mention it yesterday as we were short on time. There will 
be some syntactic restrictions so that we can properly execute those Python 
functions (i.e. we need to always be able to compute the inputs to the 
function). I had thought it was just an implementation detail I hadn't gotten 
around to (all Datalog implementations I've seen have such things), but it 
sounds like it's worth writing up a proposal and sending it around before 
implementing. If that's a pressing concern for you, let me know and I'll bump 
it up the stack (a little). If you'd like, feel free to draft a proposal (or 
remind me to do it once in a while). 

As for actions, I typically think of them as API calls to other OS components 
like Nova. But they could just as easily be Python functions. But I would want 
to avoid an action that changes Congress's internal data structures directly 
(e.g. adding a new policy statement). Such actions have caused trouble in the 
past for policy languages (though for declarative programming languages like 
Prolog they are less problematic). I don't think there's anyway we can stop 
people from creating such actions, but I think we should advocate against them. 

Tim 

- Original Message -

From: prabhakar Kudva nandava...@hotmail.com 
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org 
Sent: Wednesday, March 12, 2014 11:34:04 AM 
Subject: Re: [openstack-dev] [Congress] Policy types 

Hi Tim, All, 

I was in the discussion yesterday (kudva), and would like to start gradually 
contributing to the code base. 

So, this discussion below is based on my limited exploration of Congress 
code, running it. I am trying some small pieces to implement to familiarize. 
Please view it as such. As I start adding code, I am sure, my thoughts will 
be more evolved. 

I agree with the three types you outline. I also agree that these will grow. 
We are already thinking of expanding congress for various other types of 
policies. But those would be a manageable start. 

Regarding the comment below. I was wondering if all conditions, and actions 
could be both: 
1. python functions (for conditions they eval 
2. policy primitives. 

The advantage of 1, is that it is just executed and a True or False returned 
by Python for conditions. For actions, python functions are executed to respond 
to conditions. 
This controls the growth of policies and adding more primitives, and makes it 
flexible (say 
to use alarms, monitors, os clients, nova actions etc). 

The advantage of 2, is the ability to use unification (as in unify.py) and do 
some logic reduction. This gives us the full strength of extensive and mature 
logic reasoning and reduction methods. 

One possibility is that it checks which one the two it is and does the 
appropriate 
evaluation for condition and action. 


There are drawbacks to this proposal as well. 
- We will have 3 separate policies that are conceptually very similar. As the 
policies grow larger, it will become increasingly difficult to keep the 
policies synchronized. This problem can be mitigated to some extent by having 
all 3 share a library of policy statements that they all apply in different 
ways (and such a library mechanism is already implemented). 
- As cloud services change their behavior, policies may need to be re-written. 
For example, right now Nova does not consult Congress before creating a VM; 
thus, to enforce policy surrounding VMs, the best we could do is write a 
Condition-Action policy that adjusts VM configuration when it learns about new 
VMs being created. If we later make Nova consult with Congress before 
creating a VM, we need to write an Access-control policy that puts the proper 
controls in place. 

Thanks, 

Prabhakar Kudva 




Date: Wed, 12 Mar 2014 10:05:23 -0700 
From: thinri...@vmware.com 
To: openstack-dev@lists.openstack.org 
Subject: [openstack-dev] [Congress] Policy types 

Hi all, 

We started a discussion on IRC yesterday that I'd like to continue. The main 
question is what kind of policy does a Congress user actually write? I can see 
three options. The first two focus on actions (API calls that make changes to 
the state of the cloud) and the last focuses on just the cloud state. (By 
state of the cloud I mean all the information Congress can see about all the 
cloud services it is managing, e.g. all the information we can get through API 
calls to Nova, Neutron, Cinder, Heat, ...). 

1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed by 
other cloud services (for each state of the cloud) 
2) Condition Action: which *actions* Congress should execute (for each 

Re: [openstack-dev] [Congress][Data Integration]

2014-03-12 Thread Tim Hinrichs
Hi Rajdeep, 

This is an great problem to work on because it confronts one of the assumptions 
we're making in Congress: that cloud services can be represented as a 
collection of tables in a reasonable way. You're asking good questions here. 

More responses inline. 

Tim 


- Original Message -



From: Rajdeep Dua rajdeep@gmail.com 
To: openstack-dev@lists.openstack.org 
Sent: Wednesday, March 12, 2014 11:54:28 AM 
Subject: [openstack-dev] [Congress][Data Integration] 

Need some guidance on how to convert nested types into flat tuples. 
Also should we reorder the tuple values in a particular sequence? 




Order of tuples doesn't matter. Order of columns (values) within a tuple 
doesn't really matter either, except that all tuples must use the same order 
and the policies we write must know which column is which. 

blockquote


Thanks 
Rajdeep 

As an example i have shown networks and ports tuples with some nested types 

networks - tuple format 
--- 

keys (for reference) 

{'status','subnets', 
'name','test-network','provider:physical_network','admin_state_up', 
'tenant_id','provider:network_type','router:external', 
'shared',id,'provider:segmentation_id'} 

values 
--- 
('ACTIVE', ['4cef03d0-1d02-40bb-8c99-2f442aac6ab0'], 'test-network', None, 
True, 
'570fe78a1dc54cffa053bd802984ede2', 'gre', False, False, 
'240ff9df-df35-43ae-9df5-27fae87f2492', 4) 


/blockquote

Here we'd want to pull the List out and replace it with an ID. Then create 
another table that shows which subnets belong to the list with that ID. (You 
can think of the ID as a pointer to the list---in the C/C++ sense.) So 
something like... 

network( 'ACTIVE', 'ID1', 'test-network', None, True, 
'570fe78a1dc54cffa053bd802984ede2', 'gre', False, False, 
'240ff9df-df35-43ae-9df5-27fae87f2492', 4) 

element('ID1', '4cef03d0-1d02-40bb-8c99-2f442aac6ab0') 
element('ID1', another subnet if one existed) 

The other thing to think about is whether we want 1 table with 10 columns or we 
want 10 tables with 2 columns each. In this example, we would have... 

network('net1') 
network.status('net1', 'ACTIVE' ) 
network.subnets('net1', 'ID1') 
network.name('net1', 'test-network') 
... 

The period is just another character in the tablename. Nothing fancy happening 
here. 

The ports example below would need a similar flattening. To handle 
dictionaries, I would use the dot-notation shown above. 
A single Neutron API call might populate several Congress tables. 

Tim 

blockquote


ports - tuple format 
 
keys (for reference) 

{'status','binding:host_id', 'name', 'allowed_address_pairs', 
'admin_state_up', 'network_id', 
'tenant_id', 'extra_dhcp_opts': [], 
'binding:vif_type', 'device_owner', 
'binding:capabilities', 'mac_address', 
'fixed_ips' , 'id', 'security_groups', 
'device_id'} 

Values 

('ACTIVE', 'havana', '', [], True, '240ff9df-df35-43ae-9df5-27fae87f2492', 
'570fe78a1dc54cffa053bd802984ede2', [], 'ovs', 'network:router_interface', 
{'port_filter': True}, 'fa:16:3e:ab:90:df', [{'subnet_id': 
'4cef03d0-1d02-40bb-8c99-2f442aac6ab0', 'ip_address': '90.0.0.1'}], 
'0a2ce569-85a8-45ec-abb3-0d4b34ff69ba', [], 
'864e4acf-bf8e-4664-8cf7-ad5daa95681e') 

/blockquote

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=A86YVKfBX5U3g6F7eNScJYjr6Qwjv4dyDyVxE9Im8g8%3D%0As=0345ab3711a58ec1ebcee08649f047826cec593f57e9843df0fec2f8cfb03b42
 

blockquote



/blockquote
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Policy types

2014-03-13 Thread Tim Hinrichs
Hi Prabhakar,

I'm not sure the functionality is split between 'policy' and 'server' as 
cleanly as you describe.

The 'policy' directory contains the Policy Engine.  At its core, the policy 
engine has a generic Datalog implementation that could feasibly be used by 
other OS components.  (I don't want to think about pulling it out into Oslo 
though.  There are just too many other things going on and no demand yet.)  But 
there are also Congress-specific things in that directory, e.g. the class 
Runtime in policy/runtime.py will be the one that we hook up external API calls 
to.

The 'server' directory contains the code for the API web server that calls into 
the Runtime class.

So if you're digging through code, I'd suggest focusing on the 'policy' 
directory and looking at compile.py (responsible for converting Datalog rules 
written as strings into an internal representation) and runtime.py (responsible 
for everything else).  The docs I mentioned in the IRC should have a decent 
explanation of the functions in Runtime that the API web server will hook into. 
 

Be warned though that unless someone raises some serious objections to the 
proposal that started this thread, we'll be removing some of the more 
complicated functions from Runtime.  The compile.py code won't change (much).  
All of the 3 new theories will be instances of MaterializedViewTheory.  That's 
also the code that must change to add in the Python functions we talked about 
(more specifically see MaterializedViewTheory::propagate_rule(), which calls 
TopDownTheory::top_down_evaluation(), which is what will need modification).

Tim
 



- Original Message -
| From: prabhakar Kudva nandava...@hotmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Wednesday, March 12, 2014 1:38:55 PM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| 
| 
| 
| Hi Tim,
| 
| Thanks for your comments.
| Would be happy to contribute to the propsal and code.
| 
| The existing code already reflects the thoughts below, and got me
| in the line of ideas. Please orrect me if I am wrong as I am
| learning with these discussions:
| 
| One part (reflected by code in policy directory is the generic
| condition- action engine which could take logic primitives and
| (in the future) python functions, evaluate the conditions and
| execute the action. This portable core engine be used for any kind of
| policy enforcement
| (as by other OS projects), such as for data center monitoring and
| repair,
| service level enforcement, compliance policies, optimization (energy,
| performance) etc... at any level of the stack. This core engine seems
| possibly
| a combination of logic reasoning/unification and python function
| evaluation, and python code actions.
| 
| Second part (reflected by code in server) are the applications
| for various purposes. These could be project specific, task specific.
| We could add a diverse set of examples. The example I have worked
| with seems closer to compliance (as in net owner, vm owner check),
| and we will add more.
| 
| Prabhakar
| 
| 
| 
| Date: Wed, 12 Mar 2014 12:33:35 -0700
| From: thinri...@vmware.com
| To: openstack-dev@lists.openstack.org
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| 
| 
| Hi Prabhakar,
| 
| 
| Thanks for the feedback. I'd be interested to hear what other policy
| types you have in mind.
| 
| 
| To answer your questions...
| 
| 
| We're planning on extending our policy language in such a way that
| you can use Python functions as conditions (atom in the grammar)
| in rules. That's on my todo-list but didn't mention it yesterday as
| we were short on time. There will be some syntactic restrictions so
| that we can properly execute those Python functions (i.e. we need to
| always be able to compute the inputs to the function). I had thought
| it was just an implementation detail I hadn't gotten around to (all
| Datalog implementations I've seen have such things), but it sounds
| like it's worth writing up a proposal and sending it around before
| implementing. If that's a pressing concern for you, let me know and
| I'll bump it up the stack (a little). If you'd like, feel free to
| draft a proposal (or remind me to do it once in a while).
| 
| 
| As for actions, I typically think of them as API calls to other OS
| components like Nova. But they could just as easily be Python
| functions. But I would want to avoid an action that changes
| Congress's internal data structures directly (e.g. adding a new
| policy statement). Such actions have caused trouble in the past for
| policy languages (though for declarative programming languages like
| Prolog they are less problematic). I don't think there's anyway we
| can stop people from creating such actions, but I think we should
| advocate against them.
| 
| 
| Tim
| 
| 
| 
| From: prabhakar Kudva nandava...@hotmail.com
| To: OpenStack Development Mailing List (not for usage 

Re: [openstack-dev] [Congress] Policy types

2014-03-17 Thread Tim Hinrichs
Hi Prabhakar,

One big piece we're missing in terms of code right now is the Data Integration 
component.  The job of this component is to integrate data sources available in 
the cloud so that tables like nova:virtual_machine, neutron:owner, etc. reflect 
the information stored in Nova, Neutron, etc.  Rajdeep is making progress on 
that (he's got some code up on review that we're iterating on), and Peter had 
worked on integrating AD a while back.

Typically I've seen the Python functions (which I usually call 'builtins') 
integrated into a Datalog system have explicit declarations (e.g. inputs, 
outputs, etc.).  This is good if we need to do deeper analysis of the policy 
(which is one of the reasons to use a policy language) and typically requires 
information about how that builtin works.  

I dug through some old (Lisp) code to see how I've done this in the past.

// (defbuiltin datalog-name lisp-function list of types of args list of 
types of returns [internal])
(defbuiltin plus + (integer integer) integer)
(defbuiltin minus - (integer integer) integer)
(defbuiltin times * (integer integer) integer)
(defbuiltin div (lambda (x y) (floor (/ x y))) (integer integer) integer)
(defbuiltin lt numlessp (integer integer) nil)
(defbuiltin lte numleqp (integer integer) nil)
(defbuiltin gte numgeqp (integer integer) nil)
(defbuiltin gt numgreaterp (integer integer) nil)

But maybe you're right in that we could do away with these explicit 
declarations and just assume that everything that (i) is calleable, (ii) not 
managed by the Data Integration component, and (iii) does not appear in the 
head of a rule is a builtin.  My only worry is that I could imagine silent and 
weird problems showing up, e.g. someone forgot to define a table with rules and 
there happens to be a function in Python by that name.  Or someone supplies the 
wrong number of arguments, and we just get an error from Python, which we'd 
have no direct way to communicate to the policy-writer, i.e. there's no 
compile-time argument-length checking.

The other thing I've seen done is to have a single builtin 'evaluate' that lets 
us call an arbitrary Python function, e.g.

p(x, y) :- q(x), evaluate(mypyfunc(x), y)

Then we wouldn't need to declare the functions.  Errors would still be silent.  
But it would be clear whether we were using a Python function or not.

Thoughts?
Tim




- Original Message -
| From: prabhakar Kudva nandava...@hotmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Saturday, March 15, 2014 8:28:27 PM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| 
| 
| Hi Tim,
| 
| Here is a small change I wanted to try in runtime.py
| It may already exist in MaterializedViewTheory, but wasn't clear to
| me.
| Checking to see if this is something that:1. makes sense 2. already
| exists
| 3. worth implementing? in that order.
| 
| Let's take the example from private_public_network.classify
| 
| error(vm) :- nova:virtual_machine(vm), nova:network(vm, network),
| not neutron:public_network(network),
| neutron:owner(network, netowner), nova:owner(vm, vmowner), not
| same_group(netowner, vmowner)
| 
| same_group(user1, user2) :- cms:group(user1, group), cms:group(user2,
| group)
| 
| nova:virtual_machine(vm1)
| nova:virtual_machine(vm2)
| nova:virtual_machine(vm3)
| nova:network(vm1, net_private)
| nova:network(vm2, net_public)
| neutron:public_network(net_public)
| nova:owner(vm1, tim)
| nova:owner(vm2, pete)
| nova:owner(vm3, pierre)
| neutron:owner(net_private, martin)
| 
| 
| In this example, if as in Scenario 1:
| 
| Cloud services at our disposal:
| nova:virtual_machine(vm)
| nova:network(vm, network)
| nova:owner(vm, owner)
| neutron:public_network(network)
| neutron:owner(network, owner)
| cms:group(user, group)
| 
| are all python functions called through some nova/neutron api,
| then, we just execute them to get a true/false value in runtime.py
| They should be first checked to make sure they are python functions
| and
| not condition primitives using 'callable' and os.dir or some such
| combination.
| 
| If not, and they are assertions made in the file, not directly
| related to OS
| state, then in Scenario 2
| 
| nova:owner(vm1, tim)
| nova:owner(vm2, pete)
| nova:owner(vm3, pierre)
| 
| Are assertions made in the file. In a dynamic environment,
| a python function could query an OS client to actually find the
| current owner,
| since some other OS command could have been used to change the owner
| without an entry being made in this file, i.e., without explicitly
| informing Congres.
| This may not occur currently with vms, but may be implemented
| in a future release. Similar other examples are possible
| 
https://ask.openstack.org/en/question/5582/how-to-change-ownership-between-tenants-of-volume/
| https://blueprints.launchpad.net/cinder/+spec/volume-transfer
| 
| 
| So, I was thinking that python_nova_owner(vm1), is first checked as
| a 

Re: [openstack-dev] [Congress] Policy types

2014-03-18 Thread Tim Hinrichs
Hi Prabhakar,

No IRC meeting this week.  Our IRC is every *other* week, and we had it last 
week.

Though there's been enough activity of late that maybe we should consider 
making it weekly.

I'll address the rest later.

Tim

- Original Message -
| From: prabhakar Kudva nandava...@hotmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Monday, March 17, 2014 7:45:53 PM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Tim,
|  
| Definitely would like to learn more about the Data Integration component
| and how best to contribute. Can't find it in the latest git, perhaps we can
| discuss during the meeting tomorrow. Where can I find some code and/or
| document?
|  
| I like the idea of using an 'evauate' wrapper.  And the idea of a builtin
| library of functions.  Having strict rules for builtins and an evaluate
| wrapper
| to do type checking (primitive or builtin) and so on reduces the silent
| errors or weird outcomes.
|  
| I also agree, raw python using methods like 'calleable' (or other ways
| of checking if the function can be called in python) may not always give
| definitive answers. Even the definition of callable says, that a True only
| says that the object 'appears' calleable.
|  
| So silent and weird problems are possible if we try to execute. On the other
| hand,
| if we are looking at module which contains only user defined functions or
| builtine
| , i.e., there  are strict rules to what exactly constitute allowed-functions,
| then there
| is a higher likelihood that it is correct.  But still a likelihood.
|  
| I like the 'evaluate' idea which could do the checking within
| to make sure the function is indeed calleable, and silent and weird side
| effects
| are filtered inside the 'evaluate'. In addition, it can do checking if it is
| a  primitive or some other valid format. Needs more thought on this.
|  
| Let's discuss implementation paths if possible at the meeting. Would like to
| carve out a small implementation goal either in the 'data integration' line
| or
| the discussion above.
|  
| Prabhakar
|  
|  
|  
| 
|  
|  Date: Mon, 17 Mar 2014 08:57:05 -0700
|  From: thinri...@vmware.com
|  To: openstack-dev@lists.openstack.org
|  Subject: Re: [openstack-dev] [Congress] Policy types
|  
|  Hi Prabhakar,
|  
|  One big piece we're missing in terms of code right now is the Data
|  Integration component.  The job of this component is to integrate data
|  sources available in the cloud so that tables like nova:virtual_machine,
|  neutron:owner, etc. reflect the information stored in Nova, Neutron, etc.
|  Rajdeep is making progress on that (he's got some code up on review that
|  we're iterating on), and Peter had worked on integrating AD a while back.
|  
|  Typically I've seen the Python functions (which I usually call 'builtins')
|  integrated into a Datalog system have explicit declarations (e.g. inputs,
|  outputs, etc.).  This is good if we need to do deeper analysis of the
|  policy (which is one of the reasons to use a policy language) and
|  typically requires information about how that builtin works.
|  
|  I dug through some old (Lisp) code to see how I've done this in the past.
|  
|  // (defbuiltin datalog-name lisp-function list of types of args list
|  of types of returns [internal])
|  (defbuiltin plus + (integer integer) integer)
|  (defbuiltin minus - (integer integer) integer)
|  (defbuiltin times * (integer integer) integer)
|  (defbuiltin div (lambda (x y) (floor (/ x y))) (integer integer) integer)
|  (defbuiltin lt numlessp (integer integer) nil)
|  (defbuiltin lte numleqp (integer integer) nil)
|  (defbuiltin gte numgeqp (integer integer) nil)
|  (defbuiltin gt numgreaterp (integer integer) nil)
|  
|  But maybe you're right in that we could do away with these explicit
|  declarations and just assume that everything that (i) is calleable, (ii)
|  not managed by the Data Integration component, and (iii) does not appear
|  in the head of a rule is a builtin.  My only worry is that I could imagine
|  silent and weird problems showing up, e.g. someone forgot to define a
|  table with rules and there happens to be a function in Python by that
|  name.  Or someone supplies the wrong number of arguments, and we just get
|  an error from Python, which we'd have no direct way to communicate to the
|  policy-writer, i.e. there's no compile-time argument-length checking.
|  
|  The other thing I've seen done is to have a single builtin 'evaluate' that
|  lets us call an arbitrary Python function, e.g.
|  
|  p(x, y) :- q(x), evaluate(mypyfunc(x), y)
|  
|  Then we wouldn't need to declare the functions.  Errors would still be
|  silent.  But it would be clear whether we were using a Python function or
|  not.
|  
|  Thoughts?
|  Tim
|  
|  
|  
|  
|  - Original Message -
|  | From: prabhakar Kudva nandava...@hotmail.com
|  | To: OpenStack Development Mailing List (not for 

Re: [openstack-dev] [Congress] Policy types

2014-03-18 Thread Tim Hinrichs
Hi Prabhakar,

Found time for a more detailed response.  Comments are inline.

Tim

- Original Message -
| From: Tim Hinrichs thinri...@vmware.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Tuesday, March 18, 2014 9:31:34 AM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Prabhakar,
| 
| No IRC meeting this week.  Our IRC is every *other* week, and we had it last
| week.
| 
| Though there's been enough activity of late that maybe we should consider
| making it weekly.
| 
| I'll address the rest later.
| 
| Tim
| 
| - Original Message -
| | From: prabhakar Kudva nandava...@hotmail.com
| | To: OpenStack Development Mailing List (not for usage questions)
| | openstack-dev@lists.openstack.org
| | Sent: Monday, March 17, 2014 7:45:53 PM
| | Subject: Re: [openstack-dev] [Congress] Policy types
| | 
| | Hi Tim,
| |  
| | Definitely would like to learn more about the Data Integration component
| | and how best to contribute. Can't find it in the latest git, perhaps we can
| | discuss during the meeting tomorrow. Where can I find some code and/or
| | document?

There are no docs yet, and the only code in git is a quick-and-dirty 
ActiveDirectory integration.  But conceptually there are two pieces of the data 
integration story.

1. We're representing the data stored in each cloud service (that is pertinent 
to policy) as a collection of TABLES.  Each table is a collection of rows that 
all have the same number of columns.  Each value in the table is a scalar 
(number or string).   

If you look at the last IRC, Rajdeep has started working on translating Neutron 
data into this table format.  There are a couple of emails on the mailing list 
as well where we had some discussion.  Here's his change set.

https://review.openstack.org/#/c/77835/

2. The policy engine will need to ask for the contents of tables (or updates to 
tables).  Or have the contents/updates pushed to it periodically.  The 
infrastructure for doing this is what Peter (pballand) volunteered to do in IRC 
last week.  Let's wait on this to see what he has to say.



| |  
| | I like the idea of using an 'evauate' wrapper.  And the idea of a builtin
| | library of functions.  Having strict rules for builtins and an evaluate
| | wrapper
| | to do type checking (primitive or builtin) and so on reduces the silent
| | errors or weird outcomes.
| |  
| | I also agree, raw python using methods like 'calleable' (or other ways
| | of checking if the function can be called in python) may not always give
| | definitive answers. Even the definition of callable says, that a True only
| | says that the object 'appears' calleable.
| |  
| | So silent and weird problems are possible if we try to execute. On the
| | other
| | hand,
| | if we are looking at module which contains only user defined functions or
| | builtine
| | , i.e., there  are strict rules to what exactly constitute
| | allowed-functions,
| | then there
| | is a higher likelihood that it is correct.  But still a likelihood.

Agreed: errors will always be possible.  The policy engine will treat them as 
failures (i.e. won't re-throw the error), so we just want to help the 
policy-writer as much as we can.

| |  
| | I like the 'evaluate' idea which could do the checking within
| | to make sure the function is indeed calleable, and silent and weird side
| | effects
| | are filtered inside the 'evaluate'. In addition, it can do checking if it
| | is
| | a  primitive or some other valid format. Needs more thought on this.
| |  
| | Let's discuss implementation paths if possible at the meeting. Would like
| | to
| | carve out a small implementation goal either in the 'data integration' line
| | or
| | the discussion above.


Great! 

One option is to coordinate with rajdeep and see if there's an OS component 
that you'd like to try to turn into tables.  His code is an example of what 
you'd be writing.

Or you could put together a proposal for adding Python builtins to the policy 
engine.  One thing to think about is that we'd want to make it easy to add 
additional builtins.  We'd also need to figure out what the syntax would look 
like from the policy-writer's point of view.  Here I've seen two options:

// +, *, and lt are builtins
p(x, y, z) :- q(x), r(z) (3*x)+2=y, lt(y, z)

// builtins are forced to look like any other table.
p(x, y, z) :- q(x), r(z), *(3, x, w), +(w, 2, y), lt(y, z)


The latter is a bit clunky, but it makes it easier to implement the policy 
engine.

Tim

   

| |  
| | Prabhakar
| |  
| |  
| |  
| | 
| |  
| |  Date: Mon, 17 Mar 2014 08:57:05 -0700
| |  From: thinri...@vmware.com
| |  To: openstack-dev@lists.openstack.org
| |  Subject: Re: [openstack-dev] [Congress] Policy types
| |  
| |  Hi Prabhakar,
| |  
| |  One big piece we're missing in terms of code right now is the Data
| |  Integration component.  The job of this component is to integrate data
| |  sources available

Re: [openstack-dev] [Congress] Policy types

2014-03-19 Thread Tim Hinrichs
Inline.

Tim

- Original Message -
| From: prabhakar Kudva nandava...@hotmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Wednesday, March 19, 2014 7:09:24 AM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Tim,
|  
| I am good with starting or own the Python __congress__builtin__ proposal, but
| will need help from the whole team to work through the details.
| Do you all envision, the data integration API (Pete's calls) to be part of
| this builtin?
| What are the exact aspects each member is interested in having in this
| builtin? It
| would be good to start a list and discuss.
| 1. Inherit all python builtins
| 2. Any builtins specific to datalog etc
| 3. builtins to test and manage primitives
|  

I'd say we should start simple.  The data types we support right now are 
integers, floats, and strings.  So let's add some basic arithmetic and string 
manipulation functions.  If we design this right, it will be easy to add more 
later.

Arithmetic: +, *, -, /, , , =, others?
Strings: concatenation, substring, indexof, others?

I'd say we definitely don't want class (3)--we don't want the policy to change 
the data upon which policy decisions are made, at least not directly.  
Conceptually, the tables derived from nova/Neutron/etc. represent *their* 
states.  We can change their states but only by executing *their* API calls, 
which if successful, will change their state, which will eventually be 
reflected in Congress tables.  The Condition-Action policy that I described 
below dictates which of their API calls Congress should execute and when; this 
policy *indirectly* will change the tables of Nova/Neutron/etc.  But I wouldn't 
want to change tables via builtins.



| Regarding data integration:
| My concern with periodically pushing the contents is consistency. Having two
| stores (the nova/OS
| database and a Congress specific increases the chances of the data being out
| of sync). For example
| nova:owner(vm1), might be changed in the nova database if the period is too
| large.  What would be
| the benefits of doing this?  The first one: 'ask for contents of the tables'
| reduces the consistency
| issue.  Do you see those API mapping betweeen table entries and OS calls as
| part of the builtin?

Conceptually, data integration handles the problem of grabbing information from 
other components.  Builtins handle the problem that some tables are  
hard/impossible to define within Datalog itself.  

More practically, I would NOT want a builtin that queries another cloud service 
because the policy writer does not (and is not supposed to) know the particular 
algorithms Congress will use to process those rules.  Congress could end up 
calling a builtin 100 times for a *single* computation of policy violations.
We wouldn't want to download all the data about all VM instances 100 times 
whenever someone asks for all violations.  So builtins are things that compute 
quickly.

I don't see a way around caching the contents of other OS components.  A single 
policy might utilize the same data but in different ways in multiple places in 
the policy.  The intent is that we'll always be in sync to the extent that is 
practically feasible.  And even if we were to ask for all the data that we need 
before each and every query someone asks, that data could still change before 
we answer the query, before the person asking the query looked at the answer, 
before the person asking the query made a decision based on the answer, etc.  
So staleness is a fact of life, and we're setting this up to minimize those 
problems by enabling cloud services to send us *updates* to their data whenever 
they get them.  If a cloud service does not send us updates, the best we can do 
is poll them periodically, which will make staleness a bigger problem.  But I 
don't see what else we can do.


|  
| 2. The policy engine will need to ask for the contents of tables (or
| updates to tables).  Or have the contents/updates pushed to it
| periodically.  The infrastructure for doing this is what Peter (pballand)
| volunteered to do in IRC last week.  Let's wait on this to see what he
| has to say.
|  
|  Date: Tue, 18 Mar 2014 12:56:10 -0700
|  From: thinri...@vmware.com
|  To: openstack-dev@lists.openstack.org
|  CC: rajde...@vmware.com
|  Subject: Re: [openstack-dev] [Congress] Policy types
|  
|  Hi Prabhakar,
|  
|  Found time for a more detailed response.  Comments are inline.
|  
|  Tim
|  
|  - Original Message -
|  | From: Tim Hinrichs thinri...@vmware.com
|  | To: OpenStack Development Mailing List (not for usage questions)
|  | openstack-dev@lists.openstack.org
|  | Sent: Tuesday, March 18, 2014 9:31:34 AM
|  | Subject: Re: [openstack-dev] [Congress] Policy types
|  | 
|  | Hi Prabhakar,
|  | 
|  | No IRC meeting this week.  Our IRC is every *other* week, and we had it
|  | last
|  | week.
|  | 
|  | Though there's been

Re: [openstack-dev] [Congress] Policy types

2014-03-19 Thread Tim Hinrichs
I went ahead and made an editing pass over our design doc to reflect the 
changes described below.  (I moved most of the old content into the final 
section focused on future work.)  Overall, I'd say it simplifies the story and 
implementation considerably.  We can always revert if anyone has serious 
objections, but it was a useful exercise to see if everything hangs together 
still (and it seems to).

I've also happened to talk with several different people about Congress since 
the last IRC, and the proposed changes are well-received.  

https://docs.google.com/a/vmware.com/document/d/1f2xokl9Tc47aV67KEua0PBRz4jsdSDLXth7dYe-jz6Q/edit#

Tim



- Original Message -
| From: Tim Hinrichs thinri...@vmware.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Wednesday, March 12, 2014 10:05:23 AM
| Subject: [Congress] Policy types
| 
| Hi all,
| 
| We started a discussion on IRC yesterday that I'd like to continue. The main
| question is what kind of policy does a Congress user actually write? I can
| see three options. The first two focus on actions (API calls that make
| changes to the state of the cloud) and the last focuses on just the cloud
| state. (By state of the cloud I mean all the information Congress can see
| about all the cloud services it is managing, e.g. all the information we can
| get through API calls to Nova, Neutron, Cinder, Heat, ...).
| 
| 1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed
| by other cloud services (for each state of the cloud)
| 2) Condition Action: which *actions* Congress should execute (for each state
| of the cloud)
| 3) Classification (currently supported in Congress): which *states* violate
| real-world policy. [For those of you who have read docs/white-papers/etc.
| I'm using Classification in this note to mean the combination of the
| current Classification and Action Description policies.]
| 
| The important observation is that each of these policies could contain
| different information from each of the others.
| 
| - Access Control vs Condition Action. The Access Control policy tells *other
| cloud services* which actions they are *allowed* to execute. The Condition
| Action policy tells *Congress* which actions it *must* execute. These
| policies differ because they constrain different sets of cloud services.
| 
| - Access Control vs. Classification. The Access Control policy might permit
| some users to violate the Classification policy in some situations (e.g. to
| fix violation A, we might need to cause violation B before eliminating
| both). These policies differ because a violation in one policy might be be a
| violation in the other.
| 
| - Classification vs. Condition Action. The Classification policy might imply
| which actions *could* eliminate a given violation, but the Condition Action
| policy would dictate which of those actions *should* be executed (e.g. the
| Classification policy might tell us that disconnecting a network and
| deleting a VM would both eliminate a particular violation, but the Condition
| Action policy would tell us which to choose). And the Condition Action
| policy need not eliminate all the violations present in the Classification
| policy. Again these policies differ because a violation in one policy might
| not be a violation in the other.
| 
| I'm proposing that for the first release of Congress we support all 3 of
| these policies. When a user inserts/deletes a policy statement, she chooses
| which policy it belongs to. All would be written in basically the same
| syntax but would be used in 3 different scenarios:
| 
| - Prevention: If a component wants to consult Congress before taking action
| to see if that action is allowed, Congress checks the Access Control policy.
| 
| - Reaction: When Congress learns of a change in the cloud's state, it checks
| the Condition Action policy to see which actions should be executed (if
| any).
| 
| - Monitoring: If a user wants to simply check if the cloud's state is in
| compliance and monitor compliance over time, she writes and queries the
| Classification policy.
| 
| There are several benefits to this proposal.
| - It allows users to choose any of the policy types, if they only want one of
| them. From our discussions with potential users, most seem to want one of
| these 3 policy types (and are uninterested in the others).
| - It makes the introduction to Congress relatively simple. We describe 3
| different uses of policy (Prevention, Reaction, Monitoring) and then explain
| which policy to use in which case.
| - This allows us to focus on implementing a single policy-engine technology
| (a Datalog policy language and evaluation algorithms), which gives us the
| opportunity to make it solid.
| 
| There are drawbacks to this proposal as well.
| - We will have 3 separate policies that are conceptually very similar. As the
| policies grow larger, it will become increasingly

Re: [openstack-dev] [Congress] Policy types

2014-03-27 Thread Tim Hinrichs
Hi Prabhakar,

I looked into this a while back.  pydatalog is a cool project, and I'd like to 
find time to study some of its algorithms and architecture a bit more.  

The reason we rolled our own version of Datalog is that we knew we'd want the 
freedom to try out non-standard Datalog algorithms, and there are algorithms we 
will likely need that aren't usually included (query rewriting, skolemization, 
conversion to DNF, etc.).  I suppose we could have modified pydatalog, but 
given that it is basically an extension to Python, it seemed there would be 
significant overhead in figuring out the code, making sure to keep our changes 
compatible with the old, etc.  Rolling our own gives us the freedom to build 
exactly what we need with minimal distractions, which is important since we 
won't really know what we need until we start getting feedback from users.  

But like I said, there are probably some good ideas in there, so if the way 
they deal with builtins is useful to us, great!  I'd just keep in mind that the 
benefit of using Datalog instead of Python is mainly that it *limits* what 
policy authors can say (without limiting it too much).  

Tim

- Original Message -
| From: prabhakar Kudva nandava...@hotmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Wednesday, March 26, 2014 9:04:22 AM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Tim, All,
|  
| As I was preparing the background for the proposal for the
| __Congress_builtins__,
| came across this link which uses Datalog with Python, and provides
| data integration facilities through SQLAlchemy.  The tool seems to have been
| used (from the web page claim) in production:
| Just want to run it by everyone to see if this is connected or useful in our
| builtin
| capability, and anything we can glean from it:
|  
| 
https://urldefense.proofpoint.com/v1/url?u=https://pypi.python.org/pypi/pyDatalogk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=WvQzzQA4hxW34TMc13h1FgKQQSxHsa1RgBpgfI5lO2U%3D%0As=6028d309c9c709635828e7dd33c3f614db1b6989544e05ad6a21da2e646144fe
| 
|  
https://urldefense.proofpoint.com/v1/url?u=https://sites.google.com/site/pydatalog/k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=WvQzzQA4hxW34TMc13h1FgKQQSxHsa1RgBpgfI5lO2U%3D%0As=fb152fdc60b5925bb47145dd54973b8184a2b47fb51c28cc86afd6ecbe02b2c5
|  
| 
https://urldefense.proofpoint.com/v1/url?u=https://sites.google.com/site/pydatalog/home/datalog-applicationsk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=WvQzzQA4hxW34TMc13h1FgKQQSxHsa1RgBpgfI5lO2U%3D%0As=a4cb6eed4bf9d7d0a0bfa811c3c8d72f64b973924697f0e5e9e00681403a7664
|  
| Thanks,
|  
| Prabhakar
|  
|  Date: Tue, 18 Mar 2014 12:56:10 -0700
|  From: thinri...@vmware.com
|  To: openstack-dev@lists.openstack.org
|  CC: rajde...@vmware.com
|  Subject: Re: [openstack-dev] [Congress] Policy types
|  
|  Hi Prabhakar,
|  
|  Found time for a more detailed response.  Comments are inline.
|  
|  Tim
|  
|  - Original Message -
|  | From: Tim Hinrichs thinri...@vmware.com
|  | To: OpenStack Development Mailing List (not for usage questions)
|  | openstack-dev@lists.openstack.org
|  | Sent: Tuesday, March 18, 2014 9:31:34 AM
|  | Subject: Re: [openstack-dev] [Congress] Policy types
|  | 
|  | Hi Prabhakar,
|  | 
|  | No IRC meeting this week.  Our IRC is every *other* week, and we had it
|  | last
|  | week.
|  | 
|  | Though there's been enough activity of late that maybe we should consider
|  | making it weekly.
|  | 
|  | I'll address the rest later.
|  | 
|  | Tim
|  | 
|  | - Original Message -
|  | | From: prabhakar Kudva nandava...@hotmail.com
|  | | To: OpenStack Development Mailing List (not for usage questions)
|  | | openstack-dev@lists.openstack.org
|  | | Sent: Monday, March 17, 2014 7:45:53 PM
|  | | Subject: Re: [openstack-dev] [Congress] Policy types
|  | | 
|  | | Hi Tim,
|  | |  
|  | | Definitely would like to learn more about the Data Integration
|  | | component
|  | | and how best to contribute. Can't find it in the latest git, perhaps we
|  | | can
|  | | discuss during the meeting tomorrow. Where can I find some code and/or
|  | | document?
|  
|  There are no docs yet, and the only code in git is a quick-and-dirty
|  ActiveDirectory integration.  But conceptually there are two pieces of the
|  data integration story.
|  
|  1. We're representing the data stored in each cloud service (that is
|  pertinent to policy) as a collection of TABLES.  Each table is a
|  collection of rows that all have the same number of columns.  Each value
|  in the table is a scalar (number or string).
|  
|  If you look at the last IRC, Rajdeep has started working on translating
|  Neutron data into this table format.  There are a couple of emails on the
|  mailing list as well where we had some

[openstack-dev] Congress: an open policy framework

2013-11-02 Thread Tim Hinrichs
Hi OpenStackers,

We've been working on an open policy framework for OpenStack that we're calling 
Congress.  We've been talking with OpenStack users and several of our partners 
to understand the kinds of rules and regulations they envision enforcing with a 
policy-based management framework.  Across the board they are interested in 
policies that span networking, compute, storage, etc.

The idea behind Congress is to have a single policy engine that integrates any 
collection of external authentication and data stores and allows cloud 
administrators to write policies over those data stores in a rich, declarative 
language.  The policy engine can either enforce the policy proactively (i.e. 
preventing policy violations before they occur) or reactively (identifying 
violations after they occur and taking corrective action) or a combination 
(proactively when possible and reactively when not).  The policy engine can 
also interact with the administrator, explaining the causes of violations, 
computing potential remediation plans, and simulating action executions to 
understand what violations those actions might cause.  

While the project is still in the early stages, we have identified a grammar 
for the policy language, implemented a policy engine, and written a proof of 
concept integration for ActiveDirectory.  We would love to get participation 
and feedback.  

Code (in the midst of moving to stackforge):
https://github.com/pballand/congress

Wiki:
https://wiki.openstack.org/wiki/Congress

We'll be in Hong Kong, so if you would like to meet up to discuss please e-mail 
Peter pball...@vmware.com and Pierre pett...@vmware.com.

-- The Congress Team

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Congress: an open policy framework

2013-11-13 Thread Tim Hinrichs
We're just getting started with Congress and understanding how it will 
integrate with the OS ecosystem, but here's our current thinking about how 
Congress relates to Oslo's policy engine and to Keystone.  Comments and 
suggestions are welcome.


Congress and Oslo

Three dimensions for comparison: policy language, data sources, and policy 
engine.  

We've always planned to make Congress compatible with existing policy languages 
like the one in oslo.  The plan is to build a front-end for a number of policy 
languages/formats, e.g. oslo-policy language, XACML, JSON, YAML, SQL, etc.  The 
idea being that the syntax/language you use is irrelevant as long as it can be 
mapped into Congress's native policy language.  As of now, Congress is using 
Datalog, which is a variant of SQL and is at least as expressive as all of the 
policy languages we've run across in the cloud domain, including the 
oslo-policy language.

In terms of the data sources you can reference in the policy, Congress is 
designed to enable policies that reference arbitrary data sources in the cloud. 
 For example, we could write a Nova authorization policy that permits a new VM 
to be created if that VM is connected to a network owned by a tenant (info 
stored in Neutron) where the VM owner (info in the request) is in the same 
group as the network owner (info stored in Keystone/LDAP).  Oslo's handles some 
of these data sources with its terminal rules, but it's not involved in data 
integration to the same extent Congress is.

In terms of policy engines, Congress is intended to enforce policies in 2 
different ways: proactively (stopping policy violations before they occur) and 
reactively (acting to eliminate a violation after it occurs).  Ideally we 
wouldn't need reactive enforcement, but there will always be cases where 
proactive enforcement is not possible (e.g. a DOS attack brings app latencies 
out of compliance).  The oslo-engine does proactive enforcement only--stopping 
API calls before they violate the policy.

One concrete integration idea would be to treat Congress as a plugin for the 
oslo-policy engine.  This wouldn't enable say Nova to write policies that take 
advantage of the expressiveness of Datalog, but it would give us backwards 
compatibility.

Congress and Keystone
--
I see Keystone as providing two pieces of functionality: authentication and 
group membership.  Congress has nothing to do with authentication and never 
will.  Groups, on the other hand, are things we end up defining when writing 
policies in Congress, so conceptually there's some overlap with Keystone.  I 
guess Congress could serve as a plugin/data source for Keystone and provide it 
with the groups defined within the policy.  This would allow a group to be 
defined using data sources not available to Keystone, e.g. we could define a 
group as all users who own a VM (info from Nova) connected to a network owned 
by someone (info from Neutron) in the same group (info from LDAP).  I don't 
know how useful or efficient this would be, and it's certainly not something 
we've designed Congress for.

Thoughts?
Tim





- Original Message -
| From: Dolph Mathews dolph.math...@gmail.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Monday, November 11, 2013 3:10:01 PM
| Subject: Re: [openstack-dev] Congress: an open policy framework
| 
| 
| 
| 
| 
| 
| On Mon, Nov 11, 2013 at 4:28 AM, Flavio Percoco  fla...@redhat.com 
| wrote:
| 
| 
| 
| On 02/11/13 21:31 -0700, Tim Hinrichs wrote:
| 
| 
| Hi OpenStackers,
| 
| We've been working on an open policy framework for OpenStack that
| we're calling Congress. We've been talking with OpenStack users and
| several of our partners to understand the kinds of rules and
| regulations they envision enforcing with a policy-based management
| framework. Across the board they are interested in policies that
| span networking, compute, storage, etc.
| 
| The idea behind Congress is to have a single policy engine that
| integrates any collection of external authentication and data stores
| and allows cloud administrators to write policies over those data
| stores in a rich, declarative language. The policy engine can either
| enforce the policy proactively (i.e. preventing policy violations
| before they occur) or reactively (identifying violations after they
| occur and taking corrective action) or a combination (proactively
| when possible and reactively when not). The policy engine can also
| interact with the administrator, explaining the causes of
| violations, computing potential remediation plans, and simulating
| action executions to understand what violations those actions might
| cause.
| 
| While the project is still in the early stages, we have identified a
| grammar for the policy language, implemented a policy engine, and
| written a proof of concept integration for ActiveDirectory. We would
| love

Re: [openstack-dev] [neutron] Group-based Policy Sub-team Meetings

2013-11-13 Thread Tim Hinrichs
Are there plans for a concrete policy language (e.g. a grammar and semantics) 
to be part of the proposal, or does each plugin to Neutron supply its own 
policy language?

I'm trying to envision how Heat would utilize the policy API.  If there's a 
concrete policy language, then Heat can take an app template, extract the 
networking-relevant policy, and express that policy in the concrete language.  
Then whatever plugin we're using for Neutron can implement that policy in any 
way it sees fit as long as it obeys the policy's semantics (according to the 
language--the semantics Heat intended).

But if there's no concrete policy language, how does Heat know which policy 
statements to send?  It doesn't know which plugin is being used for Neutron.  
So it doesn't even know which strings are valid policy statements.  Or are we 
assuming that Heat knows which plugin is being used for Neutron?  Or am I 
missing something?

Thanks,
Tim

- Original Message -
| From: Kyle Mestery (kmestery) kmest...@cisco.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Wednesday, November 13, 2013 9:57:54 AM
| Subject: Re: [openstack-dev] [neutron] Group-based Policy Sub-team Meetings
| 
| On Nov 13, 2013, at 10:36 AM, Stephen Wong s3w...@midokura.com
|  wrote:
| 
|  Hi Kyle,
|  
| So no meeting this Thursday?
|  
| I am inclined to skip this week's meeting due to the fact I haven't
| heard many
| replies yet. I think a good starting point for people would be to
| review the
| BP [1] and Design Document [2] and provide feedback where
| appropriate.
| We should start to formalize what the APIs will look like at next
| week's meeting,
| and the Design Document has a first pass at this.
| 
| Thanks,
| Kyle
| 
| [1]
| https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction
| [2]
| 
https://docs.google.com/document/d/1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit?usp=sharing
| 
|  Thanks,
|  - Stephen
|  
|  On Wed, Nov 13, 2013 at 7:11 AM, Kyle Mestery (kmestery)
|  kmest...@cisco.com wrote:
|  On Nov 13, 2013, at 8:58 AM, Stein, Manuel (Manuel)
|  manuel.st...@alcatel-lucent.com wrote:
|  
|  Kyle,
|  
|  I'm afraid your meeting vanished from the Meetings page [2] when
|  user amotiki reworked neutron meetings ^.^
|  Is the meeting for Thu 1600 UTC still on?
|  
|  Ack, thanks for the heads up here! I have re-added the meeting. I
|  only heard
|  back from one other person other than yourself, so at this point
|  I'm inclined
|  to wait until next week to hold our first meeting unless I hear
|  back from others.
|  
|  A few heads-up questions (couldn't attend the HK design summit
|  Friday meeting):
|  
|  1) In the summit session Etherpad [3], ML2 implementation
|  mentions insertion of arbitrary metadata to hint to underlying
|  implementation. Is that (a) the plug-ing reporting its
|  policy-bound realization? (b) the user further specifying what
|  should be used? (c) both? Or (d) none of that but just some
|  arbitrary message of the day?
|  
|  I believe that would be (a).
|  
|  2) Would policies _always_ map to the old Neutron entities?
|  E.g. when I have policies in place, can I query related
|  network/port, subnet/address, router elements on the API or are
|  there no equivalents created? Would the logical topology created
|  under the policies be exposed otherwise? for e.g.
|  monitoring/wysiwyg/troubleshoot purposes.
|  
|  No, this is up to the plugin/MechanismDriver implementation.
|  
|  3) Do the chain identifier in your policy rule actions match to
|  Service Chain UUID in Service Insertion, Chaining and API [4]
|  
|  That's one way to look at this, yes.
|  
|  4) Are you going to describe L2 services the way group policies
|  work? I mean, why would I need a LoadBalancer or Firewall
|  instance before I can insert it between two groups when all that
|  load balancing/firewalling requires is nothing but a policy for
|  group communication itself? - regardless the service instance
|  used to carry out the service.
|  
|  These are things I'd like to discuss at the IRC meeting each week.
|  The goal
|  would be to try and come up with some actionable items we can
|  drive towards
|  in both Icehouse-1 and Icehouse-2. Given how close the closing of
|  Icehouse-1
|  is, we need to focus on this very fast if we want to have a
|  measurable impact in
|  Icehouse-1.
|  
|  Thanks,
|  Kyle
|  
|  
|  Best, Manuel
|  
|  [2]
|  
https://wiki.openstack.org/wiki/Meetings#Neutron_Group_Policy_Sub-Team_Meeting
|  [3]
|  
https://etherpad.openstack.org/p/Group_Based_Policy_Abstraction_for_Neutron
|  [4]
|  
https://docs.google.com/document/d/1fmCWpCxAN4g5txmCJVmBDt02GYew2kvyRsh0Wl3YF2U/edit#
|  
|  -Original Message-
|  From: Kyle Mestery (kmestery) [mailto:kmest...@cisco.com]
|  Sent: Montag, 11. November 2013 19:41
|  To: OpenStack Development Mailing List (not for usage questions)
|  Subject: [openstack-dev] 

Re: [openstack-dev] Congress: an open policy framework

2013-11-14 Thread Tim Hinrichs
I completely agree that making Congress successful will rely crucially on 
addressing performance and scalability issues.  Some thoughts...

1. We're definitely intending to cache data locally to avoid repeated API 
calls.  In fact, a prototype cache is already in place.  We haven't yet hooked 
up API calls (other than to AD).  We envision some data sources pushing us data 
(updates) and some data sources requiring us to periodically pull.  

2. My main concern for scalability/performance is for proactive enforcement, 
where at least conceptually Congress is on the critical path for API calls.

One thought is that we could splice out, say, the network portion of the 
Congress policy and push it down into neutron, assuming neutron could enforce 
that policy.  This would at least eliminate cross-component communication.  It 
would require a policy engine on each of the OS components, but (a) there 
already is one on many components and (b) if there isn't, we can rely on 
reactive enforcement.

The downside with policy-caching on other OS components are the usual problems 
with staleness and data replication, e.g. maybe we'd end up copying all of 
nova's VM data into neutron so that neutron could enforce its policy.  But 
because we have reactive enforcement to rely on, we could always approximate 
the policy that we push down (conservatively) to catch the common mistakes and 
leave the remainder to reactive enforcement.  For example, we might be able to 
auto-generate the current policy.json files for each component from Congress's 
policy.

Keeping Congress out of the critical path for every API call is one of the 
reasons it was designed to do reactive enforcement as well as proactive 
enforcement.

3. Another option is to build high-performance optimizations for certain 
fragments of the policy language.  Then the cloud architect can decide whether 
she wants to utilize a more expressive language whose performance is worse or a 
less expressive language whose performance is better.

Tim




- Original Message -
| From: Flavio Percoco fla...@redhat.com
| To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
| Sent: Thursday, November 14, 2013 6:05:46 AM
| Subject: Re: [openstack-dev] Congress: an open policy framework
| 
| On 14/11/13 04:40 -0800, Morgan Fainberg wrote:
| On Wed, Nov 13, 2013 at 10:40 AM, Tim Hinrichs
| thinri...@vmware.com wrote:
|  We're just getting started with Congress and understanding how it
|  will integrate with the OS ecosystem, but here's our current
|  thinking about how Congress relates to Oslo's policy engine and
|  to Keystone.  Comments and suggestions are welcome.
| 
| 
|  Congress and Oslo
|  
|  Three dimensions for comparison: policy language, data sources,
|  and policy engine.
| 
|  We've always planned to make Congress compatible with existing
|  policy languages like the one in oslo.  The plan is to build a
|  front-end for a number of policy languages/formats, e.g.
|  oslo-policy language, XACML, JSON, YAML, SQL, etc.  The idea
|  being that the syntax/language you use is irrelevant as long as
|  it can be mapped into Congress's native policy language.  As of
|  now, Congress is using Datalog, which is a variant of SQL and is
|  at least as expressive as all of the policy languages we've run
|  across in the cloud domain, including the oslo-policy language.
| 
|  In terms of the data sources you can reference in the policy,
|  Congress is designed to enable policies that reference arbitrary
|  data sources in the cloud.  For example, we could write a Nova
|  authorization policy that permits a new VM to be created if that
|  VM is connected to a network owned by a tenant (info stored in
|  Neutron) where the VM owner (info in the request) is in the same
|  group as the network owner (info stored in Keystone/LDAP).
|   Oslo's handles some of these data sources with its terminal
|  rules, but it's not involved in data integration to the same
|  extent Congress is.
| 
|  In terms of policy engines, Congress is intended to enforce
|  policies in 2 different ways: proactively (stopping policy
|  violations before they occur) and reactively (acting to eliminate
|  a violation after it occurs).  Ideally we wouldn't need reactive
|  enforcement, but there will always be cases where proactive
|  enforcement is not possible (e.g. a DOS attack brings app
|  latencies out of compliance).  The oslo-engine does proactive
|  enforcement only--stopping API calls before they violate the
|  policy.
| 
| 
| Does this mean all policy decisions need to ask this new service?
| There are many policy checks that occur across even a given action
| (in
| some cases).  Could this have a significant performance implication
| on
| larger scale cloud deployments?  I like the idea of having reactive
| (DOS prevention) policy enforcement as well as external (arbitrary)
| data to help make policy decisions, I don't want

Re: [openstack-dev] Congress: an open policy framework

2013-11-18 Thread Tim Hinrichs
Inline.

- Original Message -
| From: Adam Young ayo...@redhat.com
| To: openstack-dev@lists.openstack.org
| Sent: Friday, November 15, 2013 2:46:02 PM
| Subject: Re: [openstack-dev] Congress: an open policy framework
|
| On 11/14/2013 11:39 AM, Tim Hinrichs wrote:
|  I completely agree that making Congress successful will rely
|  crucially on addressing performance and scalability issues.  Some
|  thoughts...
| 
|  1. We're definitely intending to cache data locally to avoid
|  repeated API calls.  In fact, a prototype cache is already in
|  place.  We haven't yet hooked up API calls (other than to AD).  We
|  envision some data sources pushing us data (updates) and some data
|  sources requiring us to periodically pull.
| So, I think that you should start with what we already have working.
| Policy distribution is a part of Keystone now.  Policy processing
| uses
| Oslo policy.py.  I am not sure why it would make sense to have a
| separate program.  I would be more than happy to house this work
| inside
| the existing Keystone architecture.
|
| 
|  2. My main concern for scalability/performance is for proactive
|  enforcement, where at least conceptually Congress is on the
|  critical path for API calls.
| 
|  One thought is that we could splice out, say, the network portion
|  of the Congress policy and push it down into neutron, assuming
|  neutron could enforce that policy.  This would at least eliminate
|  cross-component communication.  It would require a policy engine
|  on each of the OS components, but (a) there already is one on many
|  components and (b) if there isn't, we can rely on reactive
|  enforcement.
| 
|  The downside with policy-caching on other OS components are the
|  usual problems with staleness and data replication, e.g. maybe
|  we'd end up copying all of nova's VM data into neutron so that
|  neutron could enforce its policy.  But because we have reactive
|  enforcement to rely on, we could always approximate the policy
|  that we push down (conservatively) to catch the common mistakes
|  and leave the remainder to reactive enforcement.  For example, we
|  might be able to auto-generate the current policy.json files for
|  each component from Congress's policy.
| 
|  Keeping Congress out of the critical path for every API call is one
|  of the reasons it was designed to do reactive enforcement as well
|  as proactive enforcement.
|
| I think all these questions have come up multiple times with
| Keystone.
| There was a signifcant discussion about Configuration management,
| which
| would encompass policy.  We were discussing the Key Distribution
| Service
| as well. It bascially comes down to inside the undercloud versus
| outside (end user facing).
|
| Termie's suggestion was that we needed to consider using something
| like
| Zookeeper.  I think he is right.
|

I've heard different people mean different things by configuration mgmt.  
Some are talking about managing server configurations e.g. with Puppet.  Others 
are talking about managing the overall configuration of the cloud across 
components (e.g. every network connected to a VM must be owned by someone in 
the same group as the VMs owner).  Congress is intended to support whatever 
configuration management/policy you can write in terms of the cloud services 
you have available.  We don't differentiate between configuration management 
and policy.

| 
|  3. Another option is to build high-performance optimizations for
|  certain fragments of the policy language.  Then the cloud
|  architect can decide whether she wants to utilize a more
|  expressive language whose performance is worse or a less
|  expressive language whose performance is better.
|
| So we should not be writing yet another policy language.  We have a
| simple one in Open Stack already, and there are many, many ones out
| there already.  XACML is the standard for authorization decisions, as
| an
| example.  Puppet  falls into this role as well.
|

Not all languages are created equal, which is especially true of 
declarative/policy languages.  The one we've prototyped is more expressive than 
the languages you mention (though is general-purpose, not domain-specific like 
Puppet).  We're not inventing yet another policy language--we're using a core 
fragment of SQL that's been studied and used for 50 years.  We're simply 
advocating its use.  We're also building a policy engine around it for both 
proactive enforcement (stopping violations before they occur) and reactive 
enforcement (correcting violations after they occur).  We're also adding 
algorithms that help an admin understand her current policy as well as 
hypothetical changes to that policy and its data sources.  Finally, we're 
providing data integration functionality for the cloud services over which 
policy is written.

|
| I don't think this effort makes sense as a stand alone project.
| Either
| it should be part of Keystone if it just for the existing acces
| policy

[openstack-dev] [neutron] Group-based Policy language

2013-11-21 Thread Tim Hinrichs
At the Neutron group-based policy proposal meeting today, we discussed whether 
or not the proposal should include a concrete policy language.  We decided to 
send a note to the list to get additional feedback.

The proposed API extension includes the ability to insert/delete policy 
statements.  But we do not say which policy statements are valid.  The benefit 
of leaving the policy language unspecified is that each plugin can support a 
custom policy language, leading to maximum flexibility in terms of writing 
plugins.  The drawback of leaving the policy language unspecified is that 
there's no way for any person or other OS component to know which API calls are 
valid, unless we know which plugin is being used.  Said another way, the 
current proposal says there are API calls like insert-policy-statement and 
delete-policy-statement, but does not say which arguments are valid to give to 
those calls (and the valid arguments can differ from plugin to plugin).

The thought experiment we went through was to imagine writing a super 
stripped-down version of Heat that only builds applications with a DB tier and 
a Web tier, and the template for the app only specifies how many DB servers and 
how many Web servers we want.  We should be able to implement a function that 
takes the number of DB servers and the number of web servers as input and 
executes a sequence of Nova/Neutron API calls that deploys that app.  But 
without a concrete policy language, we can't use the Neutron policy API  b/c we 
don't know what arguments to give the insert-policy-statement call.

In the end, we discussed adding a concrete language to the proposal.  Does 
anyone see a better alternative?

Thanks,
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] Use Cases

2014-06-18 Thread Tim Hinrichs
Hi all,

We’ve been working on developing a list of use cases for Congress, which we’ve 
also started voting on so as to prioritize our efforts.  Please feel free take 
a look, leave feedback, add to the list, vote, whatever.

https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit

If you add a use case, please use the one called “Public/private networks with 
group membership” as a guide.  The policy doesn’t need to be written in the 
policy language syntax, but it needs to be precise enough for us to figure out 
if it can be written in that language.

For those of you who have already contributed use cases, could you take a look 
at “Public/private networks with group membership” to see what info I believe 
we need for each use case?  And then if your use case is missing something, 
could you add it (or leave a note explaining why the info is 
unnecessary/unknown)?

Thanks,
Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] data-source renovation

2014-07-29 Thread Tim Hinrichs
Hi all,

As I mentioned in a previous IRC, when writing our first few policies I had 
trouble using the tables we currently use to represent external data sources 
like Nova/Neutron.  

The main problem is that wide tables (those with many columns) are hard to use. 
 (a) it is hard to remember what all the columns are, (b) it is easy to 
mistakenly use the same variable in two different tables in the body of the 
rule, i.e. to create an accidental join, (c) changes to the datasource drivers 
can require tedious/error-prone modifications to policy.

I see several options.  Once we choose something, I’ll write up a spec and 
include the other options as alternatives.


1) Add a preprocessor to the policy engine that makes it easier to deal with 
large tables via named-argument references.

Instead of writing a rule like

p(port_id, name) :-
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts, 
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif, 
device_owner, mac_address, fixed_ips, router_id, binding_host)

we would write

p(id, nme) :-
neutron:ports(port_id=id, name=nme)

The preprocessor would fill in all the missing variables and hand the original 
rule off to the Datalog engine.

Pros: (i) leveraging vanilla database technology under the hood
  (ii) policy is robust to changes in the fields of the original data b/c 
the Congress data model is different than the Nova/Neutron data models
Cons: (i) we will need to invert the preprocessor when showing 
rules/traces/etc. to the user
  (ii) a layer of translation makes debugging difficult

2) Be disciplined about writing narrow tables and write 
tutorials/recommendations demonstrating how.

Instead of a table like...
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts, 
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif, 
device_owner, mac_address, fixed_ips, router_id, binding_host)

we would have many tables...
neutron:ports(port_id)
neutron:ports.addr_pairs(port_id, addr_pairs)
neutron:ports.security_groups(port_id, security_groups)
neutron:ports.extra_dhcp_opts(port_id, extra_dhcp_opts)
neutron:ports.name(port_id, name)
...

People writing policy would write rules such as ...

p(x) :- neutron:ports.name(port, name), ...

[Here, the period e.g. in ports.name is not an operator--just a convenient way 
to spell the tablename.]

To do this, Congress would need to know which columns in a table are sufficient 
to uniquely identify a row, which in most cases is just the ID.

Pros: (i) this requires only changes in the datasource drivers; everything else 
remains the same
  (ii) still leveraging database technology under the hood
  (iii) policy is robust to changes in fields of original data
Cons: (i) datasource driver can force policy writer to use wide tables
  (ii) this data model is much different than the original data models
  (iii) we need primary-key information about tables

3) Enhance the Congress policy language to handle objects natively.

Instead of writing a rule like the following ...

p(port_id, name, group) :-
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts, 
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif, 
device_owner, mac_address, fixed_ips, router_id, binding_host),
neutron:ports.security_groups(security_group, group)

we would write a rule such as
p(port_id, name) :-
neutron:ports(port),
port.name(name),
port.id(port_id),
port.security_groups(group)

The big difference here is that the period (.) is an operator in the language, 
just as in C++/Java.

Pros:
(i) The data model we use in Congress is almost exactly the same as the data 
model we use in Neutron/Nova.

(ii) Policy is robust to changes in the Neutron/Nova data model as long as 
those changes only ADD fields.

(iii) Programmers may be slightly more comfortable with this language.

Cons:

(i) The obvious implementation (changing the engine to implement the (.) 
operator directly is quite a change from traditional database technology.  At 
this point, that seems risky.

(ii) It is unclear how to implement this via a preprocessor (thereby leveraging 
database technology).  The key problem I see is that we would need to translate 
port.name(...) into something like option (2) above.  The difficulty is that 
TABLE could sometimes be a port, sometimes be a network, sometimes be a subnet, 
etc.

(iii) Requires some extra syntactic restrictions to ensure we don't lose 
decidability.

(iv) Because the Congress and Nova/Neutron models are the same, changes to the 
Nova/Neutron model can require rewriting policy.



Thoughts?
Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Integration Demo

2014-07-31 Thread Tim Hinrichs
Hi Madhu,

Sounds like you’re doing some good stuff!  We’d love to have you involved.

Here’s the status of the items you mentioned.

- We’re about to merge a keystone/devstack integration.  
- We have an HTTP API merged. 
- arosen was working on a python-client; not sure how far along that is.
- As you mentioned, the Tetris group, who just joined, also has plans for 
optimization.  No work has been done in this area yet.
- kudva is working on a ceilometer integration.
- Hopefully soon we’ll have an end-to-end example at least pushed to review, 
along with docs.

I’d suggest touching base with arosen discuss the python-client, since you’ve 
already made progress on that.

The other thing I’d suggest is touching base with gokul, one of the Tetris 
members, and discuss optimization.

I’ll let you know once I’ve put some basic docs together along with a working 
example so you can start getting a feel for the end-to-end flow.

Tim





On Jul 30, 2014, at 10:12 PM, Madhu Mohan Nelemane snmoha...@gmail.com wrote:

 Hi All,
 
 I have been working on integrating developing a policy demo using Congress. I 
 have done following activities in this direction
 - Integrated Congress as a service with keystone 
 - Developed a trivial Python-Congressclient with support for basic commands 
 through command line
 - Integrated congress and python-congressclient into devstack
 
 To verify the integration, I am using the example policy files (VM Grouping 
 policy)  as input and trying to see the policies being written to the 
 Congress tables and the Data source tables.
 
 Could some one help me out understand what are the other missing parts to get 
 this integration working ? What needs to be implemented from the Congress 
 server side ?
 
 I am particularly interested in Contributing to all areas of Congress 
 including policy definition, enforcement and also have special interests in 
 run-time optimization, high-availability use-cases.
 
 Any guidance would be of utmost help.
 
 Thanks and Regards,
 Madhu Mohan
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [NFV] - follow up on scheduling discussion

2014-06-10 Thread Tim Hinrichs
Hi all,

I see that many of the use cases require information from different OS 
components, e.g. networking, compute, and storage.  One thing to think about is 
where those constraints are written/stored and how the data the constraints 
depend on is pulled together.  The Congress project might be helpful here, and 
I’m happy to help explore options.  Let me know if you’re interested.

https://wiki.openstack.org/wiki/Congress

Tim  



On Jun 4, 2014, at 11:25 AM, ramki Krishnan r...@brocade.com wrote:

 All,
 
 Thanks for the interest in the NFV scheduling topic. Please find a proposal 
 on Smart Scheduler (Solver Scheduler) enhancements for NFV: Use Cases, 
 Constraints etc.. 
 https://urldefense.proofpoint.com/v1/url?u=https://docs.google.com/document/d/1k60BQXOMkZS0SIxpFOppGgYp416uXcJVkAFep3Oeju8/edit%23heading%3Dh.wlbclagujw8ck=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=vTulCeloS8Hc59%2FeAOd32Ri4eqbNqVE%2FeMgNRzGZnz4%3D%0As=836991d6daab66b519de3b670db8af001144ddb20e636665b395597aa118538f
 
 Based on this proposal, we are planning to enhance the existing 
 solver-scheduler blueprint 
 https://urldefense.proofpoint.com/v1/url?u=https://blueprints.launchpad.net/nova/%2Bspec/solver-schedulerk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=vTulCeloS8Hc59%2FeAOd32Ri4eqbNqVE%2FeMgNRzGZnz4%3D%0As=d1883d82f8d09081b35986987b5f2f9f1d7731f16b2a5251fdbf26c1b26b294d.
 
 Would love to hear your comments and thoughts. Would be glad to arrange a 
 conference call if needed.
 
 Thanks,
 Ramki
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=vTulCeloS8Hc59%2FeAOd32Ri4eqbNqVE%2FeMgNRzGZnz4%3D%0As=7df001977efa968a09f3fae30b16ae35f4278a7bc82fcb3cdbbc9639cc505503


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][policy][keystone] Better Policy Model and Representing Capabilites

2014-10-14 Thread Tim Hinrichs
First, some truth in advertising: I work on Congress (policy as a service), so 
I’ve mostly given thought to this problem in that context.

1) I agree with the discussion below about creating a token that encodes all 
the permitted actions for the user.  The cons seem substantial.  

(i) The token will get stale, requiring us to either revoke it when 
policy/roles change or to live with incorrect access control enforcement until 
the token expires.  

(ii) The token could become large, complex, or both.  Suppose the policy is 
granular enough to put restrictions on the arguments a user is permitted to 
provide to an action.  The token might end up encoding a significant portion of 
the policy itself.  Checking if the token permits a given action could be 
similar computationally to checking the original policy.json file.  


2) I like the idea of an out-of-band service that caches a copy of all the 
policy.jsons and allows users to interrogate/edit them.  I’ve definitely talked 
to operators who would like this kind of thing.  This would be a low-risk, 
low-friction solution to the problem because nothing about OpenStack today 
would need to change.  We’d just add an extra service and tell people to use 
it—sort of a new UI for the policy.json files.  And we could add interesting 
functionality, e.g. hypothetical queries such as “if I were to add role X, what 
changes would that make in my rights?

Perhaps some more context about why users want to know all of the actions they 
are permitted to execute might help.

Tim


 
On Oct 14, 2014, at 1:56 AM, David Chadwick d.w.chadw...@kent.ac.uk wrote:

 
 
 On 14/10/2014 01:25, Nathan Kinder wrote:
 
 
 On 10/13/2014 01:17 PM, Morgan Fainberg wrote:
 Description of the problem: Without attempting an action on an
 endpoint with a current scoped token, it is impossible to know what
 actions are available to a user.
 
 
 This is not unusual in the physical world. If you think about all the
 authz tokens you carry around in your pocket (as plastic cards), very
 few of them (if any) list what you are entitled to do with them. This
 gives the issuers and SPs flexibility to dynamically change your
 accesses rights without changing your authorisation. What you can do, in
 general terms, may be written in policy documents that you can consult
 if you wish. So you may wish to introduce a service that is equivalent
 to this (i.e. user may optionally consult some policy advice service).
 
 If you introduce a service to allow a user to dynamically determine his
 access rights (absolutely), you have to decide what to do about the
 dynamics of this service compared to the lifetime of the keystone token,
 as the rights may change more quickly than the token's lifetime.
 
 
 Horizon makes some attempts to solve this issue by sourcing all of
 the policy files from all of the services to determine what a user
 can accomplish with a given role. This is highly inefficient as it
 requires processing the various policy.json files for each request
 in multiple places and presents a mechanism that is not really
 scalable to understand what a user can do with the current
 authorization. Horizon may not be the only service that (in the
 long term) would want to know what actions a token can take.
 
 This is also extremely useful for being able to actually support
 more restricted tokens as well.  If I as an end user want to request
 a token that only has the roles required to perform a particular
 action, I'm going to need to have a way of knowing what those roles
 are.  I think that is one of the main things missing to allow the
 role-filtered tokens option that I wrote up after the last Summit
 to be a viable approach:
 
 https://urldefense.proofpoint.com/v1/url?u=https://blog-nkinder.rhcloud.com/?p%3D101k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=XcBszEjYqiYgkoy9iUk3baKeyYoE%2Bb20k6zm3jIXGAs%3D%0As=ff78352cef9982b47c9e6cca97aa001f38b13837387332a38b56ce30e1394b87
 
 
 I would like to start a discussion on how we should improve our
 policy implementation (OpenStack wide) to help make it easier to
 know what is possible with a current authorization context
 (Keystone token). The key feature should be that whatever the
 implementation is, it doesn’t require another round-trip to a third
 party service to “enforce” the policy which avoids another scaling
 point like UUID Keystone token validation.
 
 Presumably this does not rule out the user, at his option, calling
 another service to ask for advice what can I do with this token,
 bearing in mind that the response will be advice and not a definite
 answer (since the PDP will always be the one to provide the definitive
 answer).
 
 
 
 
 Here are a couple of ideas that we’ve discussed over the last few
 development cycles (and none of this changes the requirements to
 manage scope of authorization, e.g. project, domain, trust, ...):
 
 1. Keystone is the holder of all policy files. Each 

Re: [openstack-dev] [all][policy][keystone] Better Policy Model and Representing Capabilites

2014-10-14 Thread Tim Hinrichs
That was really helpful background.  Thanks!

I’d be happy to look into using Congress to implement what we’ve discussed: 
caching policy.json files, updating them periodically, and answering queries 
about the roles required to be granted access to a certain kind of action.  I 
think we have the right algorithms sitting around.

Let me know if that would help or if you’d prefer a different approach.

Tim



On Oct 14, 2014, at 10:31 AM, Morgan Fainberg 
morgan.fainb...@gmail.commailto:morgan.fainb...@gmail.com wrote:



2) What role(s) do I need to perform a particular action?

For me, the second question is more interesting.  A user likely already
has an idea of a task that they want to perform.  With question number
1, what do I do as a user if the response says that I'm not allowed to
perform the task I'm trying to accomplish?  The answer really doesn't
give me a way to move forward and perform my task.

With question 2, I'm able to find out what exact roles are needed to
perform a specific action.  With this information, I could request a
Keystone token with a subset of my roles that is authorized to perform
the task while leaving out roles that might have a higher level of
authorization.  For instance, why should I need to send a token with the
'admin' role to Nova just to launch an instance if '_member_' is all
that's required?

Another real use case is determining what roles are needed when creating
a trust in Keystone.  If I want to use a trust to allow a service like
Heat or Neutron's LBaaS to perform an action on my behalf, I want to
minimize the authorization that I'm delegating to those services.
Keystone trusts already have the ability to explicitly define the roles
that will be present in the issues trust tokens, but I have no way of
knowing what roles are required to perform a particular action without
consulting the policy

This sums up the large(er) part of or starting this conversation.


-NGK


 Tim



 On Oct 14, 2014, at 1:56 AM, David Chadwick 
 d.w.chadw...@kent.ac.ukjavascript:; wrote:



 On 14/10/2014 01:25, Nathan Kinder wrote:


 On 10/13/2014 01:17 PM, Morgan Fainberg wrote:
 Description of the problem: Without attempting an action on an
 endpoint with a current scoped token, it is impossible to know what
 actions are available to a user.


 This is not unusual in the physical world. If you think about all the
 authz tokens you carry around in your pocket (as plastic cards), very
 few of them (if any) list what you are entitled to do with them. This
 gives the issuers and SPs flexibility to dynamically change your
 accesses rights without changing your authorisation. What you can do, in
 general terms, may be written in policy documents that you can consult
 if you wish. So you may wish to introduce a service that is equivalent
 to this (i.e. user may optionally consult some policy advice service).

 If you introduce a service to allow a user to dynamically determine his
 access rights (absolutely), you have to decide what to do about the
 dynamics of this service compared to the lifetime of the keystone token,
 as the rights may change more quickly than the token's lifetime.


 Horizon makes some attempts to solve this issue by sourcing all of
 the policy files from all of the services to determine what a user
 can accomplish with a given role. This is highly inefficient as it
 requires processing the various policy.json files for each request
 in multiple places and presents a mechanism that is not really
 scalable to understand what a user can do with the current
 authorization. Horizon may not be the only service that (in the
 long term) would want to know what actions a token can take.

 This is also extremely useful for being able to actually support
 more restricted tokens as well.  If I as an end user want to request
 a token that only has the roles required to perform a particular
 action, I'm going to need to have a way of knowing what those roles
 are.  I think that is one of the main things missing to allow the
 role-filtered tokens option that I wrote up after the last Summit
 to be a viable approach:

 https://urldefense.proofpoint.com/v1/url?u=https://blog-nkinder.rhcloud.com/?p%3D101k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0Am=XcBszEjYqiYgkoy9iUk3baKeyYoE%2Bb20k6zm3jIXGAs%3D%0As=ff78352cef9982b47c9e6cca97aa001f38b13837387332a38b56ce30e1394b87


 I would like to start a discussion on how we should improve our
 policy implementation (OpenStack wide) to help make it easier to
 know what is possible with a current authorization context
 (Keystone token). The key feature should be that whatever the
 implementation is, it doesn’t require another round-trip to a third
 party service to “enforce” the policy which avoids another scaling
 point like UUID Keystone token validation.

 Presumably this does not rule out the user, at his option, calling
 another service to ask for advice what can I do with this token,
 bearing 

Re: [openstack-dev] [congress] kilo design session

2014-10-23 Thread Tim Hinrichs
Works for me. 

Tim

P. S. Pardon the brevity. Sent from my mobile. 

 On Oct 22, 2014, at 5:01 PM, Sean Roberts seanrobert...@gmail.com wrote:
 
 We are scheduled for Monday, 03 Nov, 14:30 - 16:00. I have a conflict with 
 the “Meet the Influencers” talk that runs from 14:30-18:30, plus the GBP 
 session is on Tuesday, 04 Nov, 12:05-12:45. I was thinking we would want to 
 co-located the Congress and GBP talks as much as possible.
 
 The BOSH team has the Tuesday, 04 Nov, 16:40-18:10 slot and wants to switch. 
 
 Does this switch work for everyone?
 
 Maybe we can get some space in one of the pods or cross-project workshops on 
 Tuesday between the GBP and the potential Congress session to make it even 
 more better.
 
 ~sean
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Keystone] New Policy Administration Service

2014-11-18 Thread Tim Hinrichs
Based on the thread entitled [all][policy][keystone] Better Policy Model and 
Representing Capabilites from October 20, I wrote some code to pull a 
policy.json file into Congress and figure out what roles are necessary to give 
access to a specific API call.

So if bundling this kind of functionality into Congress is a reasonable way 
forward, it seems doable technically.  We’re happy to help in any case, so let 
us know!

Tim


-- Forwarded message --
From: Ioram Schechtman Sette i...@cin.ufpe.brmailto:i...@cin.ufpe.br
Date: Tue, Nov 18, 2014 at 5:52 AM
Subject: [openstack-dev] [Keystone] New Policy Administration Service
To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org


Hi all,

In Paris, on the last day, we listed the new features that we would like to see 
in the next release of Keystone.
The top 3 were chosen as high priority.

Further down the list was a policy administration service that will collect 
policies from all the Openstack services and allow the Keystone administrator 
to ask the question what role do I need to assign to a user to give access to 
these services? and will allow users to ask the question what can I access 
with my roles?.

We have now started to design and build this service. An important design 
decision is should this service be integrated with Keystone or be a separated 
standalone Openstack service? What does the Keystone group think?

If policy administration should be a separate service, what is the process to 
register blueprints, apis and code reviews?

Regards,
Ioram and David

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] Reactive enforcement specs

2014-11-20 Thread Tim Hinrichs
Hi all,

Recently there’s been quite a bit of interest in adding reactive enforcement to 
Congress: the ability to write policies that tell Congress to execute actions 
to correct policy violations.  We’re planning to add this feature in the next 
release.  I wrote a few specs that split this work into several bite-sized 
pieces (one was accidentally merged prematurely—it’s still up for discussion).

Let’s discuss these over Gerrit (the usual spec process).  We’re trying to 
finalize these specs by the middle of next week (a little behind the usual 
OpenStack schedule).  For those of you who haven’t left comments via Gerrit, 
you need to ...

1) log in to Gerrit using your Launchpad ID,
2) leave comments on specific lines in individual files by double-clicking the 
line you’d like to comment on,
3) click the Review button on the initial page
4) click the Publish Comments button.

Add triggers to policy engine (a programmatic interface useful for implementing 
reactive enforcement)
https://review.openstack.org/#/c/130010/

Add modal operators to policy language (how we might express reactive 
enforcement policies within Datalog)
https://review.openstack.org/#/c/134376/

Action-execution interface (how we might modify data-source drivers so they can 
execute actions)
https://review.openstack.org/#/c/134417/

Explicit reactive enforcement (pulling all the pieces together)
https://review.openstack.org/#/c/134418/


There are a number of additional specs generated since the summit.  Feel free 
to chime in on those too.
https://review.openstack.org/#/q/status:open+project:stackforge/congress-specs,n,z

Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [policy] [congress] Protocol for Congress -- Enactor

2014-11-20 Thread Tim Hinrichs
Thanks for the summary Greg—that was great!  Here’s my take.

It would be great if all the services Congress interacts with implemented the 
same protocol and used the same policy/data language.  It is worth our time to 
figure out what that protocol and language should be.

But we should not forget that there will always be legacy services that people 
are unwilling or unable to change that don’t speak that protocol/language. And 
right now no services speak that protocol/language (since it doesn’t exist).  
So it’s useful today and in the future to have an adapter/wrapper framework 
that enables Congress to  interact with other protocols and languages.

That means we need to push on 2 fronts: (i) designing the ideal 
protocol/language and (ii) designing the adapter framework.  I’ve been focused 
on (ii) since it’s absolutely necessary today, but if anyone would like to 
spearhead (i) I’d be happy to help.

Tim


On Nov 1, 2014, at 11:13 AM, Gregory Lebovitz 
gregory.i...@gmail.commailto:gregory.i...@gmail.com wrote:


Summary from IRC chat 10/14/2014 on weekly meeting [1] [2]

Topic:  Declarative Language for Congress — Enactor/Enforcer

Question: Shall we specify a declarative language for communicating policy 
configured in Congress to enactors / enforcement systems

Hypothesis (derived at conclusion of discussion):
 - Specify declarative protocol and framework for describing policy with 
extensible attributes/value fields described in a base ontology, with 
additional affinity ontologies, is what is needed earlier than later, to be 
able to achieve it as an end-state, before too many Enactors dive into one-offs.
 - We could achieve that specification once we know the right structure


Discussion:

  *   Given the following framework:
 *   Elements:
*   Congress - The policy description point, a place where:
   *   (a) policy inputs are collected
   *   (b) collected policy inputs are integrated
   *   (c) policy is defined
   *   (d) declares policy intent to enforcing / enacting systems
   *   (e) observes state of environment, noting policy violations
*   Feeders - provides policy inputs to Congress
*   Enactors / Enforcers - receives policy declarations from Congress 
and enacts / enforces the policy according to its capabilities
   *   E.g. Nova for VM placement, Neutron for interface connectivity, 
FWaaS for access control, etc.

What will the protocol be for the Congress — Enactors / Enforcers?


thinrichs:  we’ve we've been assuming that Congress will leverage whatever the 
Enactors (policy engines) and Feeders (and more generally datacenter services) 
that exist are using. For basic datacenter services, we had planned on teaching 
Congress what their API is and what it does. So there's no new protocol 
there—we'd just use HTTP or whatever the service expects. For Enactors, there 
are 2 pieces: (1) what policy does Congress push and (2) what protocol does it 
use to do that? We don't know the answer to (1) yet.  (2) is less important, I 
think. For (2) we could use opflex, for example, or create a new one. (1) is 
hard because the Enactors likely have different languages that they understand. 
I’m not aware of anyone thinking about (2). I’m not thinking about (2) b/c I 
don't know the answer to (1). The *really* hard thing to understand IMO is how 
these Enactors should cooperate (in terms of the information they exchange and 
the functionality they provide).  The bits they use to wrap the messages they 
send while cooperating is a lower-level question.


jasonsb  glebo: feel the need to clarify (2)


glebo: if we come out strongly with a framework spec that identifies a protocol 
for (2), and make it clear that Congress participants, including several data 
center Feeders and Enactors, are in consensus, then the other Feeders  
Enactors will line up, in order to be useful in the modern deployments. Either 
that, or they will remain isolated from the new environment, or their customers 
will have to create custom connectors to the new environment. It seems that we 
have 2 options. (a) Congress learns any language spoken by Feeders and 
Enactors, or (b) specifies a single protocol for Congress — Enactors policy 
declarations, including a highly adaptable public registry(ies) for defining 
the meaning of content blobs in those messages. For (a) Congress would get VERY 
bloated with an abstraction layer, modules, semantics and state for each 
different language it needed to speak. And there would be 10s of these 
languages. For (b), there would be one way to structure messages that were 
constructed of blobs in (e.g.) some sort of Type/Length/Value (TLV) method, 
where the Types and Values were specified in some Internet registry.


jasonsb: Could we attack this from the opposite direction? E.g. if Congress 
wanted to provide an operational dashboard to show if things are in compliance, 
it would be better served 

[openstack-dev] [Congress] Re: Placement and Scheduling via Policy

2014-12-16 Thread Tim Hinrichs
[Adding openstack-dev to this thread.  For those of you just joining… We 
started kicking around ideas for how we might integrate a special-purpose VM 
placement engine into Congress.]

Kudva: responses inline.


On Dec 16, 2014, at 6:25 AM, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com wrote:

Hi,

I am very interested in this.

So, it looks like there are two parts to this:
1. Policy analysis when there are a significant mix of logical and builtin 
predicates (i.e.,
runtime should identify a solution space when there are arithmetic operators). 
This will
require linear programming/ILP type solvers.  There might be a need to have a 
function
in runtime.py that specifically deals with this (Tim?)


I think it’s right that we expect there to be a mix of builtins and standard 
predicates.  But what we’re considering here is having the linear solver be 
treated as if it were a domain-specific policy engine.  So that solver wouldn’t 
be embedded into the runtime.py necessarily.  Rather, we’d delegate part of the 
policy to that domain-specific policy engine.

2. Enforcement. That is with a large number of constraints in place for 
placement and
scheduling, how does the policy engine communicate and enforce the placement
constraints to nova scheduler.


I would imagine that we could delegate either enforcement or monitoring or 
both.  Eventually we want enforcement here, but monitoring could be useful too.

And yes you’re asking the right questions.  I was trying to break the problem 
down into pieces in my bullet (1) below.  But I think there is significant 
overlap in the questions we need to answer whether we’re delegating monitoring 
or enforcement.

Both of these require some form of mathematical analysis.

Would be happy and interested to discuss more on these lines.


Maybe take a look at how I tried to breakdown the problem into separate 
questions in bullet (1) below and see if that makes sense.

Tim

Prabhakar






From:Tim Hinrichs thinri...@vmware.commailto:thinri...@vmware.com
To:ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com
Cc:Ramki Krishnan (r...@brocade.commailto:r...@brocade.com) 
r...@brocade.commailto:r...@brocade.com, Gokul B 
Kandiraju/Watson/IBM@IBMUS, Prabhakar Kudva/Watson/IBM@IBMUS
Date:12/15/2014 12:09 PM
Subject:Re: Placement and Scheduling via Policy




[Adding Prabhakar and Gokul, in case they are interested.]

1) Ruby, thinking about the solver as taking 1 matrix of [vm, server] and 
returning another matrix helps me understand what we’re talking about—thanks.  
I think you’re right that once we move from placement to optimization problems 
in general we’ll need to figure out how to deal with actions.  But if it’s a 
placement-specific policy engine, then we can build VM-migration into it.

It seems to me that the only part left is figuring out how to take an arbitrary 
policy, carve off the placement-relevant portion, and create the inputs the 
solver needs to generate that new matrix.  Some thoughts...

- My gut tells me that the placement-solver should basically say “I enforce 
policies having to do with the schema nova:location.”  This way the Congress 
policy engine knows to give it policies relevant to nova:location (placement).  
If we do that, I believe we can carve off the right sub theory.

- That leaves taking a Datalog policy where we know nova:location is important 
and converting it to the input language required by a linear solver.  We need 
to remember that the Datalog rules may reference tables from other services 
like Neutron, Ceilometer, etc.  I think the key will be figuring out what class 
of policies we can actually do that for reliably.  Cool—a concrete question.


2) We can definitely wait until January on this.  I’ll be out of touch starting 
Friday too; it seems we all get back early January, which seems like the right 
time to resume our discussions.  We have some concrete questions to answer, 
which was what I was hoping to accomplish before we all went on holiday.

Happy Holidays!
Tim


On Dec 15, 2014, at 5:53 AM, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com wrote:

Hi Tim

“Questions:
1) Is there any more data the solver needs?  Seems like it needs something 
about CPU-load for each VM.
2) Which solver should we be using?  What does the linear program that we feed 
it look like?  How do we translate the results of the linear solver into a 
collection of ‘migrate_VM’ API calls?”



  Question (2) seems to me the first to address, in particular:
 “how to prepare the input (variables, constraints, goal) and invoke the 
solver”
=  We need rules that represent constraints to give the solver (e.g. a 
technical constraint that a VM should not be assigned to more than one server 
or that more than maximum

Re: [openstack-dev] [Congress] Re: Placement and Scheduling via Policy

2014-12-18 Thread Tim Hinrichs
Hi all,

Responses inline.

On Dec 16, 2014, at 10:57 PM, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com wrote:

Hi Tim  All

@Tim: I did not reply to openstack-dev. Do you think we could have an openstack 
list specific for “congress” to which anybody may subscribe?

Sending to openstack-dev is the right thing, as long as we put [Congress] in 
the subject.  Everyone I know sets up filters on openstack-dev so they only get 
the mail they care about.  I think you’re the only one in the group who isn’t 
subscribed to that list.



1) Enforcement:
   By this we mean “how will the actions computed by the policy engine 
be executed by the concerned OpenStack functional module”.


  In this case, it is better to first work this out for a “simpler” case, 
e.g. your running example concerning the network/groups.
Note: some actions concern only some data base (e.g. insert the 
user within some group).



2)  From Prabhakar’s mail

“Enforcement. That is with a large number of constraints in place for placement 
and
scheduling, how does the policy engine communicate and enforce the placement
constraints to nova scheduler. “

Nova scheduler (current): It assigns VMs to servers based on the 
policy set by the administrator (through filters and host aggregates).

  The administrator also configures a scheduling heuristic (implemented as a 
driver), for example “round-robin” driver.
 Then the computed assignment 
is sent back to the requestor (API server) that interacts with nova-compute to 
provision the VM.
 The current nova-scheduler has 
another function: It updates the allocation status of each compute node on the 
DB (through another indirection called nova-conductor)

So it is correct to re-interpret your statement as follows:

-   What is the entity with which the policy engine interacts for either 
proactive or reactive placement management?

-   How will the output from the policy engine (for example the placement 
matrix) be communicated back?

oProactive: this gives the mapping of VM to host

oReactive: this gives the new mapping of running VMs to hosts

-   How starting from the placement matrix, the correct migration plan will 
be executed? (for reactive case)



3) Currently openstack does not have “automated management of reactive 
placement”:  Hence if the policy engine is used for reactive placement, then 
there is a need for another “orchestrator” that can interpret the new proposed 
placement configuration (mapping of VM to servers) and execute the 
reconfiguration workflow.


4) So with a policy-based “placement engine” that is integrated with 
external solvers, then this engine will replace nova-scheduler?

Could we converge on this?



The notes from Yathiraj say that there is already a policy-based Nova scheduler 
we can use.  I suggest we look into that.  It could potentially simplify our 
problem to the point where we need only figure out how to convert a fragment of 
the Congress policy language into their policy language.  But those of you who 
are experts in placement will know better.

   
https://github.com/stackforge/nova-solver-schedulerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dschedulerd=AAMGaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0Ms=mdMcHh7nMTJv8PmY0i8NpQXP9_gpUpI3gxEec6zyt7Ae=

Tim


Regards
Ruby

De : Tim Hinrichs [mailto:thinri...@vmware.com]
Envoyé : mardi 16 décembre 2014 19:25
À : Prabhakar Kudva
Cc : KRISHNASWAMY Ruby IMT/OLPS; Ramki Krishnan 
(r...@brocade.commailto:r...@brocade.com); Gokul B Kandiraju; openstack-dev
Objet : [Congress] Re: Placement and Scheduling via Policy

[Adding openstack-dev to this thread.  For those of you just joining… We 
started kicking around ideas for how we might integrate a special-purpose VM 
placement engine into Congress.]

Kudva: responses inline.


On Dec 16, 2014, at 6:25 AM, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com wrote:


Hi,

I am very interested in this.

So, it looks like there are two parts to this:
1. Policy analysis when there are a significant mix of logical and builtin 
predicates (i.e.,
runtime should identify a solution space when there are arithmetic operators). 
This will
require linear programming/ILP type solvers.  There might be a need to have a 
function
in runtime.py that specifically deals with this (Tim?)

I think it’s right that we expect there to be a mix of builtins and standard 
predicates.  But what we’re considering here is having the linear solver be 
treated as if it were a domain-specific policy engine.  So that solver wouldn’t 
be embedded into the runtime.py necessarily.  Rather, we’d

Re: [openstack-dev] [Congress] Re: Placement and Scheduling via Policy

2014-12-18 Thread Tim Hinrichs
Hi Yathi,

Thanks for the reminder about the nova solver scheduler.  It’s definitely a 
cool idea to look at integrating the two systems!

Ramki is definitely involved in this discussion.  We thought placement was a 
good first example of a broad class of problems that a linear solver could help 
address, esp. in the NFV context.  I like the idea of integrating Congress and 
the Nova solver scheduler and then generalizing what we learned to handle other 
kinds of optimization problems.  So that’s what I’m thinking long term.

Tim





On Dec 16, 2014, at 11:28 AM, Yathiraj Udupi (yudupi) 
yud...@cisco.commailto:yud...@cisco.com wrote:

To add to what I mentioned below… We from the Solver Scheduler team are a small 
team here at Cisco, trying to drive this project and slowly adding more complex 
use cases for scheduling and policy–driven placements.We would really love 
to have some real contributions from everyone in the community and build this 
the right way.
If it may interest – some interesting scheduler use cases are here based on one 
of our community meetings in IRC - 
https://etherpad.openstack.org/p/SchedulerUseCases  This could apply to 
Congress driving some of this too.

I am leading the effort for the  Solver Scheduler project ( 
https://github.com/stackforge/nova-solver-schedulerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dschedulerd=AAMGaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0Ms=mdMcHh7nMTJv8PmY0i8NpQXP9_gpUpI3gxEec6zyt7Ae=
 ) , and if any of you are willing to contribute code, API, benchmarks, and 
also work on integration, my team and I can help you guide through this.   We 
would be following the same processes under Stackforge at the moment.

Thanks,
Yathi.





On 12/16/14, 11:14 AM, Yathiraj Udupi (yudupi) 
yud...@cisco.commailto:yud...@cisco.com wrote:

Tim,

I read the conversation thread below and this got me interested as it relates 
to our discussion we had in the Policy Summit (mid cycle meet up) held in Palo 
Alto a few months ago.

This relates to our project – Nova Solver Scheduler, which I had talked about 
at the Policy summit.   Please see this - 
https://github.com/stackforge/nova-solver-schedulerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dschedulerd=AAMGaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0Ms=mdMcHh7nMTJv8PmY0i8NpQXP9_gpUpI3gxEec6zyt7Ae=

We already have a working constraints-based solver framework/engine that 
handles Nova placement, and we are currently active in Stackforge, and aim to 
get this integrated into the Gantt project 
(https://blueprints.launchpad.net/nova/+spec/solver-scheduler), based on our 
discussions in the Nova scheduler sub group.

When I saw discussions around using Linear programming (LP) solvers, PULP, etc, 
 I thought of pitching in here to say, we already have demonstrated integrating 
a LP based solver for Nova compute placements.   Please see: 
https://www.youtube.com/watch?v=7QzDbhkk-BI#t=942https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3D7QzDbhkk-2DBI-23t-3D942d=AAMGaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0Ms=EH2bJNaqNXJoAXzDbMtWkwQaJDUj-IC9QDgie_xdA8Ee=
 for a demo of this (from our talk at the Atlanta Openstack summit).
 Based on this email thread,  I believe Ramki, one of our early collaborators 
is driving a similar solution in the NFV ETSI research group.  Glad to know our 
Solver scheduler project is getting interest now.

As part of Congress integration,  at the policy summit, I had suggested, we can 
try to translate a Congress policy into our Solver Scheduler’s constraints,  
and use this to enforce Nova placement policies.
We can already demonstrate policy-driven nova placements using our pluggable 
constraints model.  So it should be easy to integrate with Congress.

The Nova solver scheduler team would be glad to help with any efforts wrt to 
trying out a Congress integration for Nova placements.

Thanks,
Yathi.



On 12/16/14, 10:24 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:

[Adding openstack-dev to this thread.  For those of you just joining… We 
started kicking around ideas for how we might integrate a special-purpose VM 
placement engine into Congress.]

Kudva: responses inline.


On Dec 16, 2014, at 6:25 AM, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com wrote:

Hi,

I am very interested in this.

So, it looks like there are two parts to this:
1. Policy analysis when there are a significant mix of logical and builtin 
predicates (i.e.,
runtime should identify a solution space when there are arithmetic operators). 
This will
require linear programming/ILP type solvers

Re: [openstack-dev] [Congress] simulate examples

2014-12-23 Thread Tim Hinrichs
Here's a description.  We need to get this added to the docs.


Below is a full description of how you might utilize the Action-centric version 
of simulate.  The idea is that if you describe the effects that an 
action/API-call will have on the basic tables of nova/neutron/etc. (below 
called an Action Description policy) then you can ask Congress to simulate the 
execution of that action and answer a query in the resulting state.  The only 
downside to the action-centric application of simulate is writing the Action 
Policy for all of the actions you care about.

The other way to utilize simulate is to give it the changes in 
nova/neutron/etc. directly that you’d like to make.  That is, instead of an 
action, you’ll tell simulate what rows should be inserted and which ones should 
be deleted.  An insertion is denoted with a plus (+) and deletion is denoted 
with a minus (-).

For example, to compute all the errors after

  1.  inserting a row into the nova:servers table with ID uuid1, 2TB of disk, 
and 10GB of memory (this isn’t the actual schema BTW) and
  2.  deleting the row from neutron:security_groups with the ID “uuid2” and 
name “alice_default_group” (again not the real schema),

you’d write something like the following.

openstack congress policy simulate classification 'error(x)’ 
‘nova:servers+(“uuid1”, “2TB”, “10 GB”) neutron:security_groups-(“uuid2”, 
“alice_default_group”)' action

But I’d suggest reading the following to see some of the options.

=
1. CREATE ACTION DESCRIPTION POLICY
=

Suppose the table 'p' is a collection of key-value pairs:  p(key, value).

Suppose we have a single action 'set(key, newvalue)’ that changes the existing 
value of 'key' to 'newvalue' or sets the value of 'key' to 'newvalue' if 'key' 
was not already assigned.  We can describe the effects of ‘set’ using the 
following 3 Datalog rules.

p+(x,y) :- set(x,y)
p-(x,oldy) :- set(x,y), p(x,oldy)
action(set)

The first thing we do is add each of these 3 rules to the policy named 'action'.

$ openstack congress policy rule create action 'p+(x,y) :- set(x,y)'
$ openstack congress policy rule create action 'p-(x,oldy) :- set(x,y), 
p(x,oldy)'
$ openstack congress policy rule create action 'action(set)'


=
2. ADD SOME KEY/VALUE PAIRS FOR TESTING
=

Here’s we’ll populate the ‘classification’ policy with a few key/value pairs.

$ openstack congress policy rule create classification 'p(101, 0)'
$ openstack congress policy rule create classification 'p(202, abc)'
$ openstack congress policy rule create classification 'p(302, 9)'


==
3. DEFINE POLICY
==

There's an error if a key's value is 9.

$ openstack congress policy rule create classification 'error(x) :- p(x, 9)'


===
4. RUN SIMULATION QUERIES
===

Each of the following is an example of a simulation query you might want to run.

a) Simulate changing the value of key 101 to 5 and query the contents of p.

$ openstack congress policy simulate classification 'p(x,y)' 'set(101, 5)' 
action
p(101, 5)
p(202, abc)
p(302, 9)


b) Simulate changing the value of key 101 to 5 and query the error table

$ openstack congress policy simulate classification 'error(x)' 'set(101, 5)' 
action
error(302)


c) Simulate changing the value of key 101 to 9 and query the error table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9)' 
action
error(302)
error(101)


d) Simulate changing the value of key 101 to 9 and query the *change* in the 
error table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9)' 
action --delta
error+(101)


e) Simulate changing 101:9, 202:9, 302:1 and query the *change* in the error 
table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9) 
set(202, 9) set(302, 1)' action --delta
error+(202)
error+(101)
error-(302)


f) Simulate changing 101:9, 202:9, 302:1, and finally 101:15 (in that order).  
Then query the *change* in the error table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9) 
set(202, 9) set(302, 1) set(101, 15)' action --delta
error+(202)
error-(302)


g) Simulate changing 101:9 and query the *change* in the error table, while 
asking for a debug trace of the computation.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9)' 
action --delta --trace

error+(101)
RT: ** Simulate: Querying error(x)
Clas  : Call: error(x)
Clas  : | Call: p(x, 9)
Clas  : | Exit: p(302, 9)
Clas  : Exit: error(302)
Clas  : Redo: error(302)
Clas  : | Redo: p(302, 9)
Clas  : | Fail: p(x, 9)
Clas  : Fail: error(x)
Clas  : Found answer [error(302)]
RT: Original result of error(x) is [error(302)]
RT: ** Simulate: Applying sequence [set(101, 9)]
Action: Call: action(x)
...

Tim





[openstack-dev] [Congress][Delegation] Google doc for working notes

2015-02-11 Thread Tim Hinrichs
Hi all,

A (growing) group of folks are interested in working on the problem of 
delegating policy from Congress to domain-specific policy engines.  We started 
looking at an NFV use case: migrating VMs to reduce energy consumption.  In 
particular we’re looking into building a VM-placement policy engine built on 
top of a linear programming solver.  Here’s a doc with some working notes where 
we’re trying to figure out how to do the translation from Congress policy to 
the linear program.

https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit?usp=sharing

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [congress][Policy][Copper]Collaboration between OpenStack Congress and OPNFV Copper

2015-02-11 Thread Tim Hinrichs
Hi Zhipeng,

We’d be happy to meet.  Sounds like fun!

I don’t know of anyone on the Congress team who is planning to attend the LF 
collaboration summit.  But we might be able to send a couple of people if it’s 
the only real chance to have a face-to-face.  Otherwise, there are a bunch of 
us in and around Palo Alto.  And of course, phone/google hangout/irc are fine 
options as well.

Tim



On Feb 11, 2015, at 8:59 AM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:

Hi Congress Team,

As you might already knew, we had a project in OPNFV covering deployment policy 
called 
Copperhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copperd=AwMFaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=R1ER1wU47Knv6PaOiamDwCm76pwx5uuE47mpn_03mzYs=S7VfJALm_Pmzb2S-o3NUlcNzLAy9yYceGzcyKX3CA-we=,
 in which we identify Congress as one of the upstream projects that we need to 
put our requirement to. Our team has been working on setting up a simple 
openstack environment with congress integrated that could demo simple use cases 
for policy deployment.

Would it possible for you guys and our team to find a time do an 
Copper/Congress interlock meeting, during which Congress Team could introduce 
how to best integrate congress with vanilla openstack? Will some of you 
attend LF Collaboration Summit?

Thanks a lot :)

--
Zhipeng (Howard) Huang

Standard Engineer
IT Standard  Patent/IT Prooduct Line
Huawei Technologies Co,. Ltd
Email: huangzhip...@huawei.commailto:huangzhip...@huawei.com
Office: Huawei Industrial Base, Longgang, Shenzhen

(Previous)
Research Assistant
Mobile Ad-Hoc Network Lab, Calit2
University of California, Irvine
Email: zhipe...@uci.edumailto:zhipe...@uci.edu
Office: Calit2 Building Room 2402

OpenStack, OpenDaylight, OpenCompute affcienado
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

2015-02-12 Thread Tim Hinrichs
Hi Debo and Yathiraj,

I took a third look at the solver-scheduler docs and code with your comments in 
mind.  A few things jumped out.

1)  Choice of LP solver.

I see solver-scheduler uses Pulp, which was on the Congress short list as well. 
 So we’re highly aligned on the choice of underlying solver.

2) User control over VM-placement.

To choose the criteria for VM-placement, the solver-scheduler user picks from a 
list of predefined options, e.g. ActiveHostConstraint, 
MaxRamAllocationPerHostConstraint.

We’re investigating a slightly different approach, where the user defines the 
criteria for VM-placement by writing any policy they like in Datalog.  Under 
the hood we then convert that Datalog to an LP problem.  From the developer’s 
perspective, with the Congress approach we don’t attempt to anticipate the 
different policies the user might want and write code for each policy; instead, 
we as developers write a translator from Datalog to LP.  From the user’s 
perspective, the difference is that if the option they want isn’t on the 
solver-scheduler's list, they’re out of luck or need to write the code 
themselves.  But with the Congress approach, they can write any VM-placement 
policy they like.

What I’d like to see is the best of both worlds.  Users write Datalog policies 
describing whatever VM-placement policy they want.  If the policy they’ve 
written is on the solver-scheduler’s list of options, we use the hard-coded 
implementation, but if the policy isn’t on that list we translate directly to 
LP.  This approach gives us the ability to write custom code to handle common 
cases while at the same time letting users write whatever policy they like.

3) API and architecture.

Today the solver-scheduler's VM-placement policy is defined at config-time 
(i.e. not run-time).  Am I correct that this limitation is only because there’s 
no API call to set the solver-scheduler’s policy?  Or is there some other 
reason the policy is set at config-time?

Congress policies change at runtime, so we’ll definitely need a VM-placement 
engine whose policy can be changed at run-time as well.

If we focus on just migration (and not provisioning), we can build a 
VM-placement engine that sits outside of Nova that has an API call that allows 
us to set policy at runtime.  We can also set up that engine to get data 
updates that influence the policy.  We were planning on creating this kind of 
VM-placement engine within Congress as a node on the DSE (our message bus).  
This is convenient because all nodes on the DSE run in their own thread, any 
node on the DSE can subscribe to any data from any other node (e.g. 
ceilometer’s data), and the algorithms for translating Datalog to LP look to be 
quite similar to the algorithms we’re using in our domain-agnostic policy 
engine.

Tim


On Feb 11, 2015, at 4:50 PM, Debojyoti Dutta 
ddu...@gmail.commailto:ddu...@gmail.com wrote:

Hi Tim: moving our thread to the mailer. Excited to collaborate!



From: Debo~ Dutta dedu...@cisco.commailto:dedu...@cisco.com
Date: Wednesday, February 11, 2015 at 4:48 PM
To: Tim Hinrichs thinri...@vmware.commailto:thinri...@vmware.com
Cc: Yathiraj Udupi (yudupi) yud...@cisco.commailto:yud...@cisco.com, 
Gokul B Kandiraju go...@us.ibm.commailto:go...@us.ibm.com, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com, 
dilik...@in.ibm.commailto:dilik...@in.ibm.com 
dilik...@in.ibm.commailto:dilik...@in.ibm.com, Norival Figueira 
nfigu...@brocade.commailto:nfigu...@brocade.com, Ramki Krishnan 
r...@brocade.commailto:r...@brocade.com, Xinyuan Huang (xinyuahu) 
xinyu...@cisco.commailto:xinyu...@cisco.com, Rishabh Jain -X (rishabja - 
AAP3 INC at Cisco) risha...@cisco.commailto:risha...@cisco.com
Subject: Re: Nova solver scheduler and Congress

Hi Tim

To address your particular questions:

  1.  translate some policy language into constraints for the LP/CVP and we had 
left that to congress hoping to integrate when the policy efforts in openstack 
were ready (our initial effort was pre congress)
  2.  For migrations: we are currently doing that – its about incremental 
constraints into the same solver. Hence its a small deal ….

Joining forces is a terrific idea. Would love to join the IRC call and see how 
we can build cool stuff in the community together. I hope we don’t have to 
replicate the vm placement engine while the work that was done in the community 
does something very similar (and be adapted)

debo

From: Tim Hinrichs thinri...@vmware.commailto:thinri...@vmware.com
Date: Wednesday, February 11, 2015 at 4:43 PM
To: Debo~ Dutta dedu...@cisco.commailto:dedu...@cisco.com
Cc: Yathiraj Udupi (yudupi) yud...@cisco.commailto:yud...@cisco.com, 
Gokul B Kandiraju go...@us.ibm.commailto:go...@us.ibm.com, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw

Re: [openstack-dev] [congress][Policy][Copper]Collaboration between OpenStack Congress and OPNFV Copper

2015-02-12 Thread Tim Hinrichs
Bryan and Zhipeng,

Sean Roberts (CCed) is planning to be in Santa Rosa.   Sean’s definitely there 
on Wed.  Less clear about Thu/Fri.

I don’t know if I’ll make the trip yet, but I’m guessing Wed early afternoon if 
I can.

Tim

On Feb 11, 2015, at 9:04 PM, SULLIVAN, BRYAN L 
bs3...@att.commailto:bs3...@att.com wrote:

Hi Tim,

It would be great to meet with members of the Congress project if possible at 
our meetup in Santa Rosa. I plan by then to have a basic understanding of 
Congress and some test driver apps / use cases to demo at the meetup. The goal 
is to assess the current state of Congress support for the use cases on the 
OPNFV wiki: 
https://wiki.opnfv.org/copper/use_caseshttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copper_use-5Fcasesd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=d4pb7BHqqZMj3oOoJBwixcr4VsTM0B4JwHe_JHRQ_VUe=

I would be doing the same with ODL but I’m not as far on getting ready with it. 
So the opportunity to discuss the use cases under Copper and the other 
policy-related projects
(fault 
managementhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_doctord=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=Wq56oTQYc1glpCeJ6wfL60x0AdyAphZeL55R7Kc7TvUe=,
 resource 
managementhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_promised=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=69Ak90Xh9biVNpWyCeLW8_7I0CoX0WrcDuFwlHQmM30e=,
 resource 
schedulerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_requirements-5Fprojects_resource-5Fschedulerd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=haq_oYTeYW7TkZp-eJrCx33KJjCg_tQlWTwiH_4OO9Ie=)
 with Congress experts would be great.

Once we understand the gaps in what we are trying to build in OPNFV, the goal 
for our first OPNFV release is to create blueprints for new work in Congress. 
We might also just find some bugs and get directly involved in Congress to 
address them, or start a collaborative development project in OPNFV for that. 
TBD

Thanks,
Bryan Sullivan | Service Standards | ATT

From: Tim Hinrichs [mailto:thinri...@vmware.com]
Sent: Wednesday, February 11, 2015 10:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: SULLIVAN, BRYAN L; HU, BIN; Rodriguez, Iben; Howard Huang
Subject: Re: [openstack-dev] [congress][Policy][Copper]Collaboration between 
OpenStack Congress and OPNFV Copper

Hi Zhipeng,

We’d be happy to meet.  Sounds like fun!

I don’t know of anyone on the Congress team who is planning to attend the LF 
collaboration summit.  But we might be able to send a couple of people if it’s 
the only real chance to have a face-to-face.  Otherwise, there are a bunch of 
us in and around Palo Alto.  And of course, phone/google hangout/irc are fine 
options as well.

Tim



On Feb 11, 2015, at 8:59 AM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:

Hi Congress Team,

As you might already knew, we had a project in OPNFV covering deployment policy 
called 
Copperhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copperd=AwMFaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=R1ER1wU47Knv6PaOiamDwCm76pwx5uuE47mpn_03mzYs=S7VfJALm_Pmzb2S-o3NUlcNzLAy9yYceGzcyKX3CA-we=,
 in which we identify Congress as one of the upstream projects that we need to 
put our requirement to. Our team has been working on setting up a simple 
openstack environment with congress integrated that could demo simple use cases 
for policy deployment.

Would it possible for you guys and our team to find a time do an 
Copper/Congress interlock meeting, during which Congress Team could introduce 
how to best integrate congress with vanilla openstack? Will some of you 
attend LF Collaboration Summit?

Thanks a lot :)

--
Zhipeng (Howard) Huang

Standard Engineer
IT Standard  Patent/IT Prooduct Line
Huawei Technologies Co,. Ltd
Email: huangzhip...@huawei.commailto:huangzhip...@huawei.com
Office: Huawei Industrial Base, Longgang, Shenzhen

(Previous)
Research Assistant
Mobile Ad-Hoc Network Lab, Calit2
University of California, Irvine
Email: zhipe...@uci.edumailto:zhipe...@uci.edu
Office: Calit2 Building Room 2402

OpenStack, OpenDaylight, OpenCompute affcienado
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

2015-02-17 Thread Tim Hinrichs
Hi Ruby,

I was envisioning the VM-placement engine choosing the hard-coded 
implementation or converting to LP.

There are 2 places that logic could go: in the VM-placement engine wrapper that 
runs on the DSE message bus, or within the VM-placement engine itself (assuming 
the two are different).  I think we’re still trying to figure out which one 
(and the initial PoC and the long-term solution may be different).

Right now I’m thinking that the VM-placement engine wrapper running on the DSE 
bus should subscribe to whatever data it needs.

Tim

On Feb 16, 2015, at 9:05 AM, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com wrote:

Hi Tim

What I’d like to see is the best of both worlds.  Users write Datalog policies 
describing whatever VM-placement policy they want.  If the policy they’ve 
written is on the solver-scheduler’s list of options, we use the hard-coded 
implementation, but if the policy isn’t on that list we translate directly to 
LP.


ð  How (calling the hard-coded implementation) ?

o   Through the message bus?

ð  Is it Congress that will send out the data or should each implementation (of 
a policy) read it in directly?

Ruby

De : Tim Hinrichs [mailto:thinri...@vmware.com]
Envoyé : jeudi 12 février 2015 19:03
À : OpenStack Development Mailing List (not for usage questions)
Objet : Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

Hi Debo and Yathiraj,

I took a third look at the solver-scheduler docs and code with your comments in 
mind.  A few things jumped out.

1)  Choice of LP solver.

I see solver-scheduler uses Pulp, which was on the Congress short list as well. 
 So we’re highly aligned on the choice of underlying solver.

2) User control over VM-placement.

To choose the criteria for VM-placement, the solver-scheduler user picks from a 
list of predefined options, e.g. ActiveHostConstraint, 
MaxRamAllocationPerHostConstraint.

We’re investigating a slightly different approach, where the user defines the 
criteria for VM-placement by writing any policy they like in Datalog.  Under 
the hood we then convert that Datalog to an LP problem.  From the developer’s 
perspective, with the Congress approach we don’t attempt to anticipate the 
different policies the user might want and write code for each policy; instead, 
we as developers write a translator from Datalog to LP.  From the user’s 
perspective, the difference is that if the option they want isn’t on the 
solver-scheduler's list, they’re out of luck or need to write the code 
themselves.  But with the Congress approach, they can write any VM-placement 
policy they like.

What I’d like to see is the best of both worlds.  Users write Datalog policies 
describing whatever VM-placement policy they want.  If the policy they’ve 
written is on the solver-scheduler’s list of options, we use the hard-coded 
implementation, but if the policy isn’t on that list we translate directly to 
LP.  This approach gives us the ability to write custom code to handle common 
cases while at the same time letting users write whatever policy they like.

3) API and architecture.

Today the solver-scheduler's VM-placement policy is defined at config-time 
(i.e. not run-time).  Am I correct that this limitation is only because there’s 
no API call to set the solver-scheduler’s policy?  Or is there some other 
reason the policy is set at config-time?

Congress policies change at runtime, so we’ll definitely need a VM-placement 
engine whose policy can be changed at run-time as well.

If we focus on just migration (and not provisioning), we can build a 
VM-placement engine that sits outside of Nova that has an API call that allows 
us to set policy at runtime.  We can also set up that engine to get data 
updates that influence the policy.  We were planning on creating this kind of 
VM-placement engine within Congress as a node on the DSE (our message bus).  
This is convenient because all nodes on the DSE run in their own thread, any 
node on the DSE can subscribe to any data from any other node (e.g. 
ceilometer’s data), and the algorithms for translating Datalog to LP look to be 
quite similar to the algorithms we’re using in our domain-agnostic policy 
engine.

Tim


On Feb 11, 2015, at 4:50 PM, Debojyoti Dutta 
ddu...@gmail.commailto:ddu...@gmail.com wrote:


Hi Tim: moving our thread to the mailer. Excited to collaborate!



From: Debo~ Dutta dedu...@cisco.commailto:dedu...@cisco.com
Date: Wednesday, February 11, 2015 at 4:48 PM
To: Tim Hinrichs thinri...@vmware.commailto:thinri...@vmware.com
Cc: Yathiraj Udupi (yudupi) yud...@cisco.commailto:yud...@cisco.com, 
Gokul B Kandiraju go...@us.ibm.commailto:go...@us.ibm.com, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com, 
dilik...@in.ibm.commailto:dilik

Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

2015-02-13 Thread Tim Hinrichs
 additional constraints at runtime (PulP model or any other 
models we can use and support).  It will just take in any external policy (say 
Datalog in Congress example), and it can easily add those set of resulting 
translated constraints via this custom constraint builder class.  This is 
something we can definitely add value to solver scheduler by implementing and 
adding here.

This sounds promising.  What would that custom constraint class look like?  
What language would it support, e.g. would Congress send over a PulP LP?



3) API and architecture.

Today the solver-scheduler's VM-placement policy is defined at config-time 
(i.e. not run-time).  Am I correct that this limitation is only because there’s 
no API call to set the solver-scheduler’s policy?  Or is there some other 
reason the policy is set at config-time?

Congress policies change at runtime, so we’ll definitely need a VM-placement 
engine whose policy can be changed at run-time as well.

YATHI -  We have working code to set VM placement policies at run-time to 
dynamically select the constraint or cost classes to use.   It is yet to 
upstreamed to solver scheduler stackforge repo, but will be soon.  But yeah I 
agree with you, this is definitely needed for any policy-driven VM placement 
engine, as the policies are dynamic. Short answer, yes solver scheduler has 
abilities to support this.

Good to know.  Let us know when that gets committed so we can take a look.



If we focus on just migration (and not provisioning), we can build a 
VM-placement engine that sits outside of Nova that has an API call that allows 
us to set policy at runtime.  We can also set up that engine to get data 
updates that influence the policy.  We were planning on creating this kind of 
VM-placement engine within Congress as a node on the DSE (our message bus).  
This is convenient because all nodes on the DSE run in their own thread, any 
node on the DSE can subscribe to any data from any other node (e.g. 
ceilometer’s data), and the algorithms for translating Datalog to LP look to be 
quite similar to the algorithms we’re using in our domain-agnostic policy 
engine.

YATHI – The entire scheduling community in Nova is planning on an external 
scheduler (Gantt), and we are pitching solver scheduler also as a stand-alone 
placement engine a scheduler as a service.  Nova integration is just to ensure 
it fits within the Nova workflow.   I am not quite familiar with the DSE 
architecture yet,  but the simple idea we have is, Congress policies, as part 
of the enforcement workflow, should set the VM placement constraints, and feed 
any additional data to be used for scheduling/placement decisions, which will 
be consumed dynamically by the Solver Scheduler, and after the delegation, the 
Solver scheduler module will calculate the placement decisions, and complete 
the VM initial placement or call the VM migration APIs and enable the required 
migrations.

So the solver-scheduler would take an LP program as input (using some 
agreed-upon decision variables) and handle provisioning or migration as needed. 
 The Congress side would take care of translating Datalog to that LP problem.  
Right?

Tim




Thanks,
Yathi.


On 2/12/15, 10:02 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:

Hi Debo and Yathiraj,

I took a third look at the solver-scheduler docs and code with your comments in 
mind.  A few things jumped out.





2) User control over VM-placement.

Tim


On Feb 11, 2015, at 4:50 PM, Debojyoti Dutta 
ddu...@gmail.commailto:ddu...@gmail.com wrote:

Hi Tim: moving our thread to the mailer. Excited to collaborate!



From: Debo~ Dutta dedu...@cisco.commailto:dedu...@cisco.com
Date: Wednesday, February 11, 2015 at 4:48 PM
To: Tim Hinrichs thinri...@vmware.commailto:thinri...@vmware.com
Cc: Yathiraj Udupi (yudupi) yud...@cisco.commailto:yud...@cisco.com, 
Gokul B Kandiraju go...@us.ibm.commailto:go...@us.ibm.com, Prabhakar Kudva 
ku...@us.ibm.commailto:ku...@us.ibm.com, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com, 
dilik...@in.ibm.commailto:dilik...@in.ibm.com 
dilik...@in.ibm.commailto:dilik...@in.ibm.com, Norival Figueira 
nfigu...@brocade.commailto:nfigu...@brocade.com, Ramki Krishnan 
r...@brocade.commailto:r...@brocade.com, Xinyuan Huang (xinyuahu) 
xinyu...@cisco.commailto:xinyu...@cisco.com, Rishabh Jain -X (rishabja - 
AAP3 INC at Cisco) risha...@cisco.commailto:risha...@cisco.com
Subject: Re: Nova solver scheduler and Congress

Hi Tim

To address your particular questions:

  1.  translate some policy language into constraints for the LP/CVP and we had 
left that to congress hoping to integrate when the policy efforts in openstack 
were ready (our initial effort was pre congress)
  2.  For migrations: we are currently doing that – its about incremental 
constraints into the same solver. Hence its a small deal ….

Joining forces is a terrific idea. Would

Re: [openstack-dev] [congress][Policy][Copper]Collaboration between OpenStack Congress and OPNFV Copper

2015-02-13 Thread Tim Hinrichs
Hi Zhipeng,

Sure we can talk online Mon/Tue.  If you come up with some times the Copper 
team is available, I’ll do the same for a few of the Congress team.  We’re 
usually available 9-4p Pacific, and for me Monday looks better than Tuesday.

If we schedule an hour meeting in Santa Rosa for Wed at say 1:30 or 2p, I’ll do 
my best to make it there.  Even if I can’t make it, you’ll always have Sean to 
talk with.

Tim




On Feb 12, 2015, at 6:49 PM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:

THX Tim!

I think It'd be great if we could have some online discussion ahead of F2F LFC 
summit.We could have the crash course early next week (Monday or Tuesday), and 
then Bryan could discuss with Sean in detail when they met, with specific 
questions.

Would this be ok for everyone ?

On Fri, Feb 13, 2015 at 7:21 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:
Bryan and Zhipeng,

Sean Roberts (CCed) is planning to be in Santa Rosa.   Sean’s definitely there 
on Wed.  Less clear about Thu/Fri.

I don’t know if I’ll make the trip yet, but I’m guessing Wed early afternoon if 
I can.

Tim


On Feb 11, 2015, at 9:04 PM, SULLIVAN, BRYAN L 
bs3...@att.commailto:bs3...@att.com wrote:

Hi Tim,

It would be great to meet with members of the Congress project if possible at 
our meetup in Santa Rosa. I plan by then to have a basic understanding of 
Congress and some test driver apps / use cases to demo at the meetup. The goal 
is to assess the current state of Congress support for the use cases on the 
OPNFV wiki: 
https://wiki.opnfv.org/copper/use_caseshttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copper_use-5Fcasesd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=d4pb7BHqqZMj3oOoJBwixcr4VsTM0B4JwHe_JHRQ_VUe=

I would be doing the same with ODL but I’m not as far on getting ready with it. 
So the opportunity to discuss the use cases under Copper and the other 
policy-related projects
(fault 
managementhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_doctord=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=Wq56oTQYc1glpCeJ6wfL60x0AdyAphZeL55R7Kc7TvUe=,
 resource 
managementhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_promised=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=69Ak90Xh9biVNpWyCeLW8_7I0CoX0WrcDuFwlHQmM30e=,
 resource 
schedulerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_requirements-5Fprojects_resource-5Fschedulerd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=haq_oYTeYW7TkZp-eJrCx33KJjCg_tQlWTwiH_4OO9Ie=)
 with Congress experts would be great.

Once we understand the gaps in what we are trying to build in OPNFV, the goal 
for our first OPNFV release is to create blueprints for new work in Congress. 
We might also just find some bugs and get directly involved in Congress to 
address them, or start a collaborative development project in OPNFV for that. 
TBD

Thanks,
Bryan Sullivan | Service Standards | ATT

From: Tim Hinrichs [mailto:thinri...@vmware.com]
Sent: Wednesday, February 11, 2015 10:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: SULLIVAN, BRYAN L; HU, BIN; Rodriguez, Iben; Howard Huang
Subject: Re: [openstack-dev] [congress][Policy][Copper]Collaboration between 
OpenStack Congress and OPNFV Copper

Hi Zhipeng,

We’d be happy to meet.  Sounds like fun!

I don’t know of anyone on the Congress team who is planning to attend the LF 
collaboration summit.  But we might be able to send a couple of people if it’s 
the only real chance to have a face-to-face.  Otherwise, there are a bunch of 
us in and around Palo Alto.  And of course, phone/google hangout/irc are fine 
options as well.

Tim



On Feb 11, 2015, at 8:59 AM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:

Hi Congress Team,

As you might already knew, we had a project in OPNFV covering deployment policy 
called 
Copperhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copperd=AwMFaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=R1ER1wU47Knv6PaOiamDwCm76pwx5uuE47mpn_03mzYs=S7VfJALm_Pmzb2S-o3NUlcNzLAy9yYceGzcyKX3CA-we=,
 in which we identify Congress as one of the upstream projects that we need to 
put our requirement to. Our team has been working on setting up a simple 
openstack environment with congress integrated that could demo simple use cases 
for policy deployment.

Would it possible for you guys and our team to find a time do an 
Copper/Congress interlock meeting, during which Congress Team could introduce 
how

Re: [openstack-dev] [congress][Policy][Copper]Collaboration between OpenStack Congress and OPNFV Copper

2015-02-15 Thread Tim Hinrichs
I just realized Monday is a holiday. What about 8a, 10a, 2-4p Tuesday?

I'm happy to try out gotomeeting.  Looks like I don't need an account, right?

Tim

P. S. Pardon the brevity. Sent from my mobile.

On Feb 13, 2015, at 4:25 PM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:


Hi Tim,

Monday 9am PST should be ok for me, it is better if u guys could do 4 - 5 pm, 
but let's settle on 9 am for now, see if Bryan and others would be ok. 
Regarding meeting tools, in opnfv we use GoToMeeting a lot, would u guys be ok 
with that? Or should we have a Google Hangout?

On Feb 14, 2015 8:04 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:
Hi Zhipeng,

Sure we can talk online Mon/Tue.  If you come up with some times the Copper 
team is available, I’ll do the same for a few of the Congress team.  We’re 
usually available 9-4p Pacific, and for me Monday looks better than Tuesday.

If we schedule an hour meeting in Santa Rosa for Wed at say 1:30 or 2p, I’ll do 
my best to make it there.  Even if I can’t make it, you’ll always have Sean to 
talk with.

Tim




On Feb 12, 2015, at 6:49 PM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:

THX Tim!

I think It'd be great if we could have some online discussion ahead of F2F LFC 
summit.We could have the crash course early next week (Monday or Tuesday), and 
then Bryan could discuss with Sean in detail when they met, with specific 
questions.

Would this be ok for everyone ?

On Fri, Feb 13, 2015 at 7:21 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:
Bryan and Zhipeng,

Sean Roberts (CCed) is planning to be in Santa Rosa.   Sean’s definitely there 
on Wed.  Less clear about Thu/Fri.

I don’t know if I’ll make the trip yet, but I’m guessing Wed early afternoon if 
I can.

Tim


On Feb 11, 2015, at 9:04 PM, SULLIVAN, BRYAN L 
bs3...@att.commailto:bs3...@att.com wrote:

Hi Tim,

It would be great to meet with members of the Congress project if possible at 
our meetup in Santa Rosa. I plan by then to have a basic understanding of 
Congress and some test driver apps / use cases to demo at the meetup. The goal 
is to assess the current state of Congress support for the use cases on the 
OPNFV wiki: 
https://wiki.opnfv.org/copper/use_caseshttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copper_use-5Fcasesd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=d4pb7BHqqZMj3oOoJBwixcr4VsTM0B4JwHe_JHRQ_VUe=

I would be doing the same with ODL but I’m not as far on getting ready with it. 
So the opportunity to discuss the use cases under Copper and the other 
policy-related projects
(fault 
managementhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_doctord=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=Wq56oTQYc1glpCeJ6wfL60x0AdyAphZeL55R7Kc7TvUe=,
 resource 
managementhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_promised=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=69Ak90Xh9biVNpWyCeLW8_7I0CoX0WrcDuFwlHQmM30e=,
 resource 
schedulerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_requirements-5Fprojects_resource-5Fschedulerd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=79iOYd5evGtBk2y36AKWDlDGaxiAbtt-Aago3I-8XcUs=haq_oYTeYW7TkZp-eJrCx33KJjCg_tQlWTwiH_4OO9Ie=)
 with Congress experts would be great.

Once we understand the gaps in what we are trying to build in OPNFV, the goal 
for our first OPNFV release is to create blueprints for new work in Congress. 
We might also just find some bugs and get directly involved in Congress to 
address them, or start a collaborative development project in OPNFV for that. 
TBD

Thanks,
Bryan Sullivan | Service Standards | ATT

From: Tim Hinrichs [mailto:thinri...@vmware.com]
Sent: Wednesday, February 11, 2015 10:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: SULLIVAN, BRYAN L; HU, BIN; Rodriguez, Iben; Howard Huang
Subject: Re: [openstack-dev] [congress][Policy][Copper]Collaboration between 
OpenStack Congress and OPNFV Copper

Hi Zhipeng,

We’d be happy to meet.  Sounds like fun!

I don’t know of anyone on the Congress team who is planning to attend the LF 
collaboration summit.  But we might be able to send a couple of people if it’s 
the only real chance to have a face-to-face.  Otherwise, there are a bunch of 
us in and around Palo Alto.  And of course, phone/google hangout/irc are fine 
options as well.

Tim



On Feb 11, 2015, at 8:59 AM, Zhipeng Huang 
zhipengh...@gmail.commailto:zhipengh...@gmail.com wrote:

Hi Congress Team,

As you might already knew, we had a project in OPNFV covering deployment policy

Re: [openstack-dev] [Congress] [Delegation] Meeting scheduling

2015-03-18 Thread Tim Hinrichs
I responded in the gdoc.  Here’s a copy.

One of my goals for delegation is to avoid asking people to write policy 
statements specific to any particular domain-specific solver.  People ought to 
encode policy however they like, and the system ought to figure out how best to 
enforce that policy  (delegation being one option).

Assuming that's a reasonable goal, I see two options for delegation to  
solverScheduler

(1) SolverScheduler exposes a custom constraint class.  Congress generates the 
LP program from the Datalog, similar to what is described in this doc, and 
gives that LP program as custom constraints to the  SolverScheduler.  
SolverScheduler is then responsible for enforcing that policy both during 
provisioning of new servers and for monitoring/migrating servers once 
provisioning is finished.

(2) The Congress adapter for SolverScheduler understands the semantics of 
MemoryCapacityConstraint, identifies when the user has asked for that 
constraint, and replaces that part of the LP program with the 
MemoryCapacityConstraint.

We probably want a combination of (1) and (2) so that we handle any gaps in the 
pre-defined constraints that SolverScheduler has, while at the same time 
leveraging the pre-defined constraints when possible.

Tim


On Mar 17, 2015, at 6:09 PM, Yathiraj Udupi (yudupi) 
yud...@cisco.commailto:yud...@cisco.com wrote:

Hi Tim,

I posted this comment on the doc.  I am still pondering over a possibility of 
have a policy-driven scheduler workflow via the Solver Scheduler placement 
engine, which is also LP based like you describe in your doc.
I know in your initial meeting, you plan to go over your proposal of building a 
VM placement engine that subscribes to the Congress DSE,  I probably will 
understand the Congress workflows better and see how I could incorporate this 
proposal to talk to the Solver Scheduler to make the placement decisions.

The example you provide in the doc, is a very good scenario, where a VM 
placement engine should continuously monitor and trigger VM migrations.

I am also interested in the case of a policy-driven scheduling for the initial 
creation of VMs. This is where say people will call Nova APIs and create a new 
set of VMs. Here the scheduler workflow should address the constraints as 
imposed from the user's policies.

Say the simple policy is  Host's free RAM = 0.25 * Memory_Capacity
I would like the scheduler to use this policy as defined from Congress, and 
apply it during the scheduling as part of the Nova boot call.

I am really interested in and need help in coming up with a solution 
integrating Solver Scheduler, so say if I have an implementation of a 
MemoryCapacityConstraint, which takes a hint value free_memory_limit (0.25 
in this example),
could we have a policy in Datalog

placement_requirement(id) :-
nova:host(id),
solver_scheduler:applicable_constraints(id, [MemoryCapacityConstraint, ]),
applicable_metadata(id, {free_memory_limit: 0.25, })

This policy could be set and delegated by Congress to solver scheduler via the 
set_policy API. or the Solver Scheduler can query Congress via a get_policy 
API to get this policy, and incorporate it as part of the solver scheduler 
workflow ?

Does this sound doable ?

Thanks,
Yathi.



On 3/16/15, 11:05 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:

Hi all,

The feedback on the POC delegation proposal has been mostly positive.  Several 
people have asked for a meeting to discuss further.  Given time zone 
constraints, it will likely be 8a or 9a Pacific.  Let me know in the next 2 
days if you want to participate, and we will try to find a day that everyone 
can attend.

https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edithttps://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1ksDilJYXV-2D5AXWON8PLMedDKr9NpS8VbT0jIy-5FMIEtI_editd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=uiVXAFxgoK-F8a3oNLDykcTSmPGUMW2_kFB_BnUEBFgs=GWRQesGone_FbZRHXZmIcQd5MB20CHkriADqVNVnxiAe=

Thanks!
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] [Delegation] Meeting scheduling

2015-03-18 Thread Tim Hinrichs
Ruby,

The Custom constraint class was something Yathiraj mentioned a while back.  But 
yes the idea is that MemoryCapacityConstraint would be a special case of what 
we can express in the custom constraints.

Tim


On Mar 18, 2015, at 10:05 AM, 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com wrote:

Hello

o) custom constraint class
What did you mean by the “custom” constraint class?

  Did you mean we specify a “meta model” to specify constraints?  And then each 
“Policy” specifying a constraint  ( ) will lead to generation of the constraint 
in that meta-model.
Then the solver-scheduler could pick up the constraint?

This then will not require the “solver scheduler” to implement specific 
constraint classes such as “MemoryCapacityConstraint”.

We may have rules (not in sense of Datalog ☺ ) for name (e.g. variables or 
constants) generation?


Ruby

De : Tim Hinrichs [mailto:thinri...@vmware.com]
Envoyé : mercredi 18 mars 2015 16:34
À : OpenStack Development Mailing List (not for usage questions)
Objet : Re: [openstack-dev] [Congress] [Delegation] Meeting scheduling

I responded in the gdoc.  Here’s a copy.

One of my goals for delegation is to avoid asking people to write policy 
statements specific to any particular domain-specific solver.  People ought to 
encode policy however they like, and the system ought to figure out how best to 
enforce that policy  (delegation being one option).

Assuming that's a reasonable goal, I see two options for delegation to  
solverScheduler

(1) SolverScheduler exposes a custom constraint class.  Congress generates the 
LP program from the Datalog, similar to what is described in this doc, and 
gives that LP program as custom constraints to the  SolverScheduler.  
SolverScheduler is then responsible for enforcing that policy both during 
provisioning of new servers and for monitoring/migrating servers once 
provisioning is finished.

(2) The Congress adapter for SolverScheduler understands the semantics of 
MemoryCapacityConstraint, identifies when the user has asked for that 
constraint, and replaces that part of the LP program with the 
MemoryCapacityConstraint.

We probably want a combination of (1) and (2) so that we handle any gaps in the 
pre-defined constraints that SolverScheduler has, while at the same time 
leveraging the pre-defined constraints when possible.

Tim


On Mar 17, 2015, at 6:09 PM, Yathiraj Udupi (yudupi) 
yud...@cisco.commailto:yud...@cisco.com wrote:

Hi Tim,

I posted this comment on the doc.  I am still pondering over a possibility of 
have a policy-driven scheduler workflow via the Solver Scheduler placement 
engine, which is also LP based like you describe in your doc.
I know in your initial meeting, you plan to go over your proposal of building a 
VM placement engine that subscribes to the Congress DSE,  I probably will 
understand the Congress workflows better and see how I could incorporate this 
proposal to talk to the Solver Scheduler to make the placement decisions.

The example you provide in the doc, is a very good scenario, where a VM 
placement engine should continuously monitor and trigger VM migrations.

I am also interested in the case of a policy-driven scheduling for the initial 
creation of VMs. This is where say people will call Nova APIs and create a new 
set of VMs. Here the scheduler workflow should address the constraints as 
imposed from the user's policies.

Say the simple policy is  Host's free RAM = 0.25 * Memory_Capacity
I would like the scheduler to use this policy as defined from Congress, and 
apply it during the scheduling as part of the Nova boot call.

I am really interested in and need help in coming up with a solution 
integrating Solver Scheduler, so say if I have an implementation of a 
MemoryCapacityConstraint, which takes a hint value free_memory_limit (0.25 
in this example),
could we have a policy in Datalog

placement_requirement(id) :-
nova:host(id),
solver_scheduler:applicable_constraints(id, [MemoryCapacityConstraint, ]),
applicable_metadata(id, {free_memory_limit: 0.25, })

This policy could be set and delegated by Congress to solver scheduler via the 
set_policy API. or the Solver Scheduler can query Congress via a get_policy 
API to get this policy, and incorporate it as part of the solver scheduler 
workflow ?
Does this sound doable ?

Thanks,
Yathi.



On 3/16/15, 11:05 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:

Hi all,

The feedback on the POC delegation proposal has been mostly positive.  Several 
people have asked for a meeting to discuss further.  Given time zone 
constraints, it will likely be 8a or 9a Pacific.  Let me know in the next 2 
days if you want to participate, and we will try to find a day that everyone 
can attend.

https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edithttps://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1ksDilJYXV-2D5AXWON8PLMedDKr9NpS8VbT0jIy

[openstack-dev] [Congress] [Delegation] Meeting scheduling

2015-03-16 Thread Tim Hinrichs
Hi all,

The feedback on the POC delegation proposal has been mostly positive.  Several 
people have asked for a meeting to discuss further.  Given time zone 
constraints, it will likely be 8a or 9a Pacific.  Let me know in the next 2 
days if you want to participate, and we will try to find a day that everyone 
can attend.

https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit

Thanks!
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] [Delegation] Meeting scheduling

2015-03-20 Thread Tim Hinrichs
(Instead of having a copy of these emails in the gdoc, I resolved the comment 
in the gdoc and added a link to this email thread in the ML archives.)

Inline.

On Mar 20, 2015, at 8:34 AM, Yathiraj Udupi (yudupi) 
yud...@cisco.commailto:yud...@cisco.com wrote:

Hi Tim,

I think what you are saying is a reasonable goal in terms of high-level 
Congress policies not having to depend on the domain-specific solver / policy 
engines.  As long as there are Congress adapters to transform the user’s 
policies to something that domain-specific solver understands it will be fine.

With respect to delegation to solver scheduler, I agree that we need a 
combination of options (1) and (2).
For option (2), in order to leverage the pre-defined constraint classes in 
Solver Scheduler,  it is more like the Congress adapter notifying the Solver 
Scheduler to include that constraint as part of the current placement decision. 
  Also we have to take it to account that it is not just the user-defined 
Congress policies that define the placement choices,  the already existing 
infrastructure-specific constraints, in terms of Host capacities, or any other 
provider specific constraints will have to be included in the placement 
decision calculation by the solver.   The pre-configured scheduler/placement 
constraints will always have to be included.  But some additional policies from 
Congress can introduce additional constraints.

I’d love to hear more about this.  Where do these other 
policies/configurations/etc. live?  What do they look like?  One of the things 
we’ve always discussed is a model where we combine policies from different 
places to understand the *actual* policy that we need to act on.  If you could 
walk us through an example that shows different kinds of constraints and where 
they come from, that’d be a big help!


For option (1), new custom constraint class -  Solver Scheduler already has an 
interface BaseLinearConstraint - 
https://github.com/stackforge/nova-solver-scheduler/blob/master/nova/scheduler/solvers/linearconstraints/__init__.pyhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dscheduler_blob_master_nova_scheduler_solvers_linearconstraints_-5F-5Finit-5F-5F.pyd=AwMF-gc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=_MGkpdeOqH9x07MnF4pUGr69548scQd0-bA67jlWkHss=b0wV-mTuXGGQGYRUfvVshJDpNAT36A8bKVxcypWafgYe=
with methods -  get_coefficient_vectors, get_variable_vectors, and 
get_operations that will be invoked by the main solver class to feed in the 
variables, and the host metrics, along with some input parameters that used to 
get additional metadata that can be used to build matrices.  Eventually the 
main solver class builds the LP program by invoking all the classes.  So for 
the Congress delegation scenario, it will be along these lines, where the 
Congress adapter will have to populate these matrices based on the Datalog 
policy.   So as part of the solver scheduler’s workflow this special 
CongressConstraint class will have to call the Congress adapter with the 
variables already known, and get the necessary values.
For reference, an example implementation of this constraint class is here - 
https://github.com/stackforge/nova-solver-scheduler/blob/master/nova/scheduler/solvers/linearconstraints/max_disk_allocation_constraint.pyhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dscheduler_blob_master_nova_scheduler_solvers_linearconstraints_max-5Fdisk-5Fallocation-5Fconstraint.pyd=AwMF-gc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=_MGkpdeOqH9x07MnF4pUGr69548scQd0-bA67jlWkHss=kRmhPNGUt-v8xSY-tz1EsR_Ylcc4pr0gnuC9zRYkxnoe=


That sounds promising.  I’d like to do a compare/contrast of how it would look 
with and without the solver-scheduler interface.  I’ve never used an LP solver, 
so it’s hard for me to understand the benefits of the interface you have over 
using the raw LP solver interface.  Again, an example would help me.

Tim

Will need some more thoughts, but the approach seems reasonable.

Thanks,
Yathi.



On 3/18/15, 8:34 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:

I responded in the gdoc.  Here’s a copy.

One of my goals for delegation is to avoid asking people to write policy 
statements specific to any particular domain-specific solver.  People ought to 
encode policy however they like, and the system ought to figure out how best to 
enforce that policy  (delegation being one option).

Assuming that's a reasonable goal, I see two options for delegation to  
solverScheduler

(1) SolverScheduler exposes a custom constraint class.  Congress generates the 
LP program from the Datalog, similar to what is described in this doc, and 
gives that LP program as custom constraints to the  SolverScheduler.  
SolverScheduler is then responsible for enforcing that policy both during 
provisioning

Re: [openstack-dev] [keystone][congress][group-policy] Fetching policy from a remote source

2015-03-16 Thread Tim Hinrichs
Hi Adam,

For the most part we've been looking at Congress policy as complementary to 
Oslo policy, so we haven’t yet tried to incorporate Oslo policy into Congress 
(though I did some experiments with that a while back).   But looking forward, 
it would definitely be convenient if there were some way to fetch Oslo policy 
from each of the components.

More inline...

 On Mar 16, 2015, at 8:10 AM, Adam Young ayo...@redhat.com wrote:
 
 Oslo policy has been released as a stand alone library.  This is great, in 
 that the rules engine is relatively non-applicaition specific, and I assume 
 that all of the policy based project are planning to migrate over to using 
 the policy library instead of the incubated version.
 
 Part of the push toward a more dynamic policy mechanism is figuring out how 
 to fetch and cache the policy files from Keystone.  I suspect that the other 
 services have the same issue.

I thought each service would provide an API endpoint that would allow us to 
fetch their policy.json.  Why would we go through Keystone?

 
 1.  How long should a service hold on to the cached version of the policy 
 file?
 2.  How can we avoid the stampeding herd if Keystone pushes out a 
 notification change event?
 3.  How do we securely cache the file and still make it possible to debug.
 

I’m guessing performance won’t be a problem.  Policy.json is small, and there 
are likely few services listening to updates.  

Is the *content* of policy.json something sensitive that needs high security?  
Months ago there was a thread about building a tool for analyzing policy.json 
to tell people which groups they would need to execute a given API call.  
Wouldn’t that mean we’re probably not too concerned about people seeing the 
contents of policy.json?   I’m not saying we should broadcast policy.json to 
everyone who wants it, but it’s not clear to me we need to worry about 
protecting policy.json any more than administrator-level data returned by 
today’s API calls.

 The PKI tokens have a lot of these same issues, and have a one-off mechanism 
 for handling it.  We should probably look in to commonizing this function.
 
 There general mechanism should be fetch and cache but I think it should not 
 be tied to keystone token validation so much as capable of using it if 
 necessary.  I'm guessing that access to policy rules are typically controlled 
 by auth token validated services.  Is this correct?
 

If there were an API call for fetching policy, I would expect it to be 
protected just like any other API call.  And sure we’d provide whatever 
credentials are necessary to make that call.

Tim

 Maybe the right level of abstraction is a callback function for fetching the 
 file to be cached, with the default being something that uses 
 python-requests, and then an  auth plugin based alternative for those that 
 require Keystone tokens.
 
 Before I go off and write a spec, I'd like to know what the prior art is 
 here.  I'd also like to know if there oslo policy library is part of the 
 plans for the other groups that are doing policy based work?
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

2015-03-24 Thread Tim Hinrichs
Hi Bryan,

I wish we could claim Congress has most of the data from OpenStack APIs already 
available for policy, but that’s not the case today.

If it were me, I’d start with your use cases, figure out which services and API 
calls you’ll need to support those use cases, and then check if those APIs are 
already available in Congress.

Tim

On Mar 24, 2015, at 1:47 PM, Bryan Sullivan 
bls...@hotmail.commailto:bls...@hotmail.com wrote:

Thanks for clarifying, Tim (and Alex for the other response).

Since I have to understand the code anyway going to the drivers for current 
info is reasonable.

Can I derive from your statement One of Congress’s goals is to make any data 
available via an API call to policy-related that Congress already exposes (in 
Congress tables) most/all data available through OpenStack component APIs? That 
would save me digging through the OpenStack code as well.


From: thinri...@vmware.commailto:thinri...@vmware.com
To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tue, 24 Mar 2015 19:57:26 +
Subject: Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

Hi Bryan,

Inline.

On Mar 24, 2015, at 12:13 PM, Bryan Sullivan 
bls...@hotmail.commailto:bls...@hotmail.com wrote:

Hi, I have a question about where to find the complete set of currently 
supported data drivers (with table/row details) for Congress. The info at 
http://congress.readthedocs.org/en/latest/cloudservices.html#drivershttps://urldefense.proofpoint.com/v2/url?u=http-3A__congress.readthedocs.org_en_latest_cloudservices.html-23driversd=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=7K6Tq1uqgCY-gmJMedkayeCZgjUEWp7QD_HaJg6pHmYs=rEUZxD7aLCxwILxm2ugsR4Qr-4JXCdZ_-9BWcvG5F_ke=
 seems to cover only Nova and Neutron.  I'm sure that I can pull this info from 
the source at 
https://github.com/stackforge/congress/tree/master/congress/datasourceshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_congress_tree_master_congress_datasourcesd=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=7K6Tq1uqgCY-gmJMedkayeCZgjUEWp7QD_HaJg6pHmYs=XeNw_NiKGg3HrR5R6oVbN9EVV2RhB3CQ4waMWGvBAHMe=
 but I was looking for any documentation on the current state of that code's 
design. The existing blueprints (e.g. at 
https://github.com/stackforge/congress-specs)https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_congress-2Dspecs-29d=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=7K6Tq1uqgCY-gmJMedkayeCZgjUEWp7QD_HaJg6pHmYs=AjqWychcmYYsMd9oOGytLq3rXOqzhJGLbUTi2OuX2_0e=
 do not go into this detail AFAICT.


I’d definitely just do a directory listing on the source code.  Syncing the 
docs is something we do before a major release.  Things are just changing too 
quickly to do anything else.

Also, since I'm looking into how the current supported Congress tables will 
support use cases of the OPNFV Copper project [1], it will be also helpful to 
know where I would look for the set of potential policy-related data items 
inside OpenStack docs/code for the various components. If I find a gap against 
a use case, I would want to be able to specify related patches (e.g. where data 
source driver extensions should get the additional data) so I need to know 
where the data comes from (and what's available) inside OpenStack.


One of Congress’s goals is to make any data available via an API call to 
policy-related.  So anything the API calls of OpenStack returns are reasonable 
candidates for writing policy over.

Tim

[1] 
https://wiki.opnfv.org/copper/use_caseshttps://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_copper_use-5Fcasesd=AwMF-gc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=lssF60z2CQNbUwpN7SMHds5C8d33H4qy6lYURJ57ifMs=SWYETWsuJ04nF3g_lUDlGGi7SlwgIHjobgh9d5zGLA4e=

Thanks for your help!
Bryan





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__ 
OpenStack Development Mailing List (not for usage questions) Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

2015-03-24 Thread Tim Hinrichs
Hi Zhenzan,


I don't have the CLI in front of me, but check out the 'driver' commands.  The 
command you're looking for is something like the following.


$ openstack congress driver list



Tim



From: Zhou, Zhenzan zhenzan.z...@intel.com
Sent: Tuesday, March 24, 2015 7:39 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

Hi,

To check LOADED(or ENABLED) data source drivers for a running system, you can 
use congress cli. Maybe we could add a command to list SUPPORTED data source 
drivers?

zhenzan@zhenzan-openstack:~$ openstack congress datasource list
+--+---+-+--+-+
| id   | name  | enabled | type | config

  |
+--+---+-+--+-+
| 20a33403-e8d0-440a-b36f-0383bfcfd06f | cinder| True| None | 
{u'username': u'admin', u'tenant_name': u'admin', u'password': u'hidden', 
u'auth_url': u'http://10.239.47.118:5000/v2.0'} |
| 7326d165-9e03-4b9c-acf8-b94a7f0a013f | ironic| True| None | 
{u'username': u'admin', u'tenant_name': u'admin', u'password': u'hidden', 
u'auth_url': u'http://10.239.47.118:5000/v2.0'} |
| 80587cf8-ffbc-4c9a-b452-f884427b8cac | keystone  | True| None | 
{u'username': u'admin', u'tenant_name': u'admin', u'password': u'hidden', 
u'auth_url': u'http://10.239.47.118:5000/v2.0'} |
| 9a85d0f9-c869-41fe-b6d0-b784c1e84169 | nova  | True| None | 
{u'username': u'admin', u'tenant_name': u'admin', u'password': u'hidden', 
u'auth_url': u'http://10.239.47.118:5000/v2.0'} |
| df715798-7aa9-4fdc-8bfd-f37718bd4691 | neutronv2 | True| None | 
{u'username': u'admin', u'tenant_name': u'admin', u'password': u'hidden', 
u'auth_url': u'http://10.239.47.118:5000/v2.0'} |
| e819b6c4-3744-4c45-9623-ad26ead15485 | glancev2  | True| None | 
{u'username': u'admin', u'tenant_name': u'admin', u'password': u'hidden', 
u'auth_url': u'http://10.239.47.118:5000/v2.0'} |
+--+---+-+--+-+

zhenzan@zhenzan-openstack:~$ openstack congress datasource schema show cinder
+---++
| table | columns|
+---++
| services  | {'name': 'status', 'description': 'None'}, |
|   | {'name': 'binary', 'description': 'None'}, |
|   | {'name': 'zone', 'description': 'None'},   |
|   | {'name': 'state', 'description': 'None'},  |
|   | {'name': 'updated_at', 'description': 'None'}, |
|   | {'name': 'host', 'description': 'None'},   |
|   | {'name': 'disabled_reason', 'description': 'None'} |
|   ||
| snapshots | {'name': 'status', 'description': 'None'}, |
|   | {'name': 'created_at', 'description': 'None'}, |
|   | {'name': 'volume_id', 'description': 'None'},  |
|   | {'name': 'size', 'description': 'None'},   |
|   | {'name': 'id', 'description': 'None'}, |
|   | {'name': 'name', 'description': 'None'}|
|   ||
| volumes   | {'name': 'id', 'description': 'None'}, |
|   | {'name': 'size', 'description': 'None'},   |
|   | {'name': 'user_id', 'description': 'None'},|
|   | {'name': 'status', 'description': 'None'}, |
|   | {'name': 'description', 'description': 'None'},|
|   | {'name': 'name', 'description': 'None'},   |
|   | {'name': 'bootable', 'description': 'None'},   |
|   | {'name': 'created_at', 'description': 'None'}, |
|   | {'name': 'volume_type', 'description': 'None'} |
|   ||
+---++


BR
Zhou Zhenzan

From: Tim Hinrichs [mailto:thinri...@vmware.com]
Sent: Wednesday, March 25, 2015 6:07 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

Hi Bryan,

I wish we could claim Congress has most of the data

Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

2015-03-25 Thread Tim Hinrichs
Hi Dani,

We do have an interface in Horizon that you’ll get if you install via devstack 
(except that you can’t create rules quite yet—that’s under active development). 
If you install it standalone, you won’t get the Horizon interface, but you can 
still use the CLI or the restful API.

Let us know if you want help getting it setup.  And if you hit any bumps, we’d 
love to hear about them so we can smooth them out.

Tim


On Mar 25, 2015, at 3:28 AM, Daniel Comnea 
comnea.d...@gmail.commailto:comnea.d...@gmail.com wrote:

This project looks very interesting and i might give it a go by installing 
standalone. I have a question though: are there any plans to expose the current 
output as part of Horizon or any other UI ?

Thx,
Dani

On Wed, Mar 25, 2015 at 5:15 AM, Zhou, Zhenzan 
zhenzan.z...@intel.commailto:zhenzan.z...@intel.com wrote:
Just found it has been supported, e.g.

openstack  congress driver schema show ceilometer

So here is the way to check data source drivers:

For supported data source drivers:

1.   openstack  congress driver list

2.   openstack  congress driver schema show ds-name

For enabled data source drivers:

1.   openstack congress datasource list

2.   openstack congress datasource schema show ds-name

BR
Zhou Zhenzan

From: Zhou, Zhenzan 
[mailto:zhenzan.z...@intel.commailto:zhenzan.z...@intel.com]
Sent: Wednesday, March 25, 2015 12:24 PM

To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers


Hi,

The ‘driver list’ sub command could provide supported data source drivers, but 
we cannot show its schema if it’s NOT LOADED. So we still have to check the 
code for the schema. My point is that we could support show schemas for any 
supported data source drivers, even it’s not loaded.

zhenzan@zhenzan-openstack:~$ openstack congress driver list
++--+
| id | description  
|
++--+
| ceilometer | Datasource driver that interfaces with ceilometer.   
|
| plexxi | Datasource driver that interfaces with PlexxiCore.   
|
| swift  | Datasource driver that interfaces with swift.
|
| neutronv2  | Datasource driver that interfaces with OpenStack Networking 
aka Neutron. |
| nova   | Datasource driver that interfaces with OpenStack Compute aka 
nova.   |
| murano | Datasource driver that interfaces with murano
|
| keystone   | Datasource driver that interfaces with keystone. 
|
| cloudfoundryv2 | Datasource driver that interfaces with cloudfoundry  
|
| ironic | Datasource driver that interfaces with OpenStack bare metal 
aka ironic.  |
| cinder | Datasource driver that interfaces with OpenStack cinder. 
|
| vcenter| Datasource driver that interfaces with vcenter   
|
| glancev2   | Datasource driver that interfaces with OpenStack Images aka 
Glance.  |
++--+
zhenzan@zhenzan-openstack:~$ openstack congress datasource schema show 
ceilometer
ERROR: openstack Resource ceilometer not found (HTTP 404)


BR
Zhou Zhenzan
From: Pierre-Emmanuel Ettori [mailto:pe.ett...@gmail.com]
Sent: Wednesday, March 25, 2015 11:54 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

Hi Zhenzan,
Actually I believe the command that Tim is looking for is:
openstack congress datasource list

Please let us know if you are running into any issue.

Thanks
Pierre

On Tue, Mar 24, 2015 at 8:39 PM Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:

Hi Zhenzan,



I don't have the CLI in front of me, but check out the 'driver' commands.  The 
command you're looking for is something like the following.



$ openstack congress driver list



Tim




From: Zhou, Zhenzan zhenzan.z...@intel.commailto:zhenzan.z...@intel.com
Sent: Tuesday, March 24, 2015 7:39 PM

To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers
Hi,

To check LOADED(or ENABLED) data source drivers for a running system, you can 
use congress cli. Maybe we could add a command to list SUPPORTED data source 
drivers?

zhenzan@zhenzan-openstack:~$ openstack congress datasource list

Re: [openstack-dev] [Congress]How to add tempest tests for testing murano driver

2015-03-02 Thread Tim Hinrichs
Hi Hong,

Aaron started working on this, but we don’t have anything in place yet, as far 
as I know.  He’s a starting point.

https://review.openstack.org/#/c/157166/

Tim

On Feb 26, 2015, at 2:56 PM, Wong, Hong 
hong.w...@hp.commailto:hong.w...@hp.com wrote:

Hi Aaron,

I am new to congress and trying to write tempest tests for the newly added 
murano datasource driver.  Since the murano datasource tempest tests require 
both murano and python-congress clients as the dependencies.  I was told that I 
can't just simply add the requirements in the tempest/requirements.txt file as 
both packages are in not in the main branch, so CI will not be able to pick 
them up.  Do you know of any workaround ?

Thanks,
Hong
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress][Delegation] Initial workflow design

2015-03-02 Thread Tim Hinrichs


De : Tim Hinrichs [mailto:thinri...@vmware.com]
Envoyé : jeudi 26 février 2015 19:17
À : OpenStack Development Mailing List (not for usage questions)
Objet : Re: [openstack-dev] [Congress][Delegation] Initial workflow design

Inline.

From: ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Wednesday, February 25, 2015 at 8:53 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Congress][Delegation] Initial workflow design

Hi Tim, All,


1)  Step 3: The VM-placement engine is also a “datalog engine” . Right?

When policies are delegated:

when policies are inserted? When the VM-placement engine has already registered 
itself all policies are given to it?




“In our example, this would mean the domain-specific policy engine executes the 
following API call over the DSE”

ð  “domain-agnostic” ….



Done.


2)  Step 4:



Ok

But finally: if Congress will likely “delegate”



Not sure what you’re suggesting here.


3)  Step 5:  Compilation of subpolicy to LP in VM-placement engine



For the PoC, it is likely that the LP program ( in PuLP or some other ML) is 
*not* completely generated by compiler/translator.

ð  Right?

Where does the rest of the program originate?  I’m not saying the entire LP 
program is generated from the Datalog constraints; some of it is generated by 
the solver independent of the Datalog.  In the text, I gave the example of 
defining hMemUse[j].


 You also indicate that some category of constraints (“the LP solver 
doesn’t know what the relationship between assign[i][j], hMemUse[j], and 
vMemUse[i] actually is, so the VM-placement engine must also include 
constraints”) .
 These constraints must be “explicitly” written?  (e.g. max_ram_allocation 
etc that are constraints used in the solver-scheduler’s package).

The VM-placement engine does 2 things: (I) translates Datalog to LP and (ii) 
generates additional LP constraints.  (Both work items could leverage any 
constraints that are builtin to a specific solver, e.g. the solver-scheduler.  
The point is that there are 2 distinct, conceptual origins of the LP 
constraints: those that represent the Datalog and those that codify the domain.



 So what “parts” will be generated:
Cost function :
Constraint from Policy : memory usage  75%

 Then the rest should be “filled” up?

 Could we convene on an intermediary “modeling language”?
@Yathi: do you think we could use some thing like AMPL ? Is this 
proprietary?


A detail: the example “Y[host1] = hMemUse[host1]  0.75 * hMemCap[host1]”


ð  To be changed to a linear form (mi – Mi  0 then Yi = 1 else Yi = 0) so 
something like (mi – Mi)  100 yi

Each domain-specific solver can do whatever it wants, so it’s not clear to me 
what the value of choosing a modeling language actually is—unless we want to 
build a library of common functionality that makes the construction of 
domain-specific engine (wrappers) easier.  I’d prefer to spend our energy 
understanding whether the proposed workflow/interface works for a couple of 
different domain-specific policy engines OR to flush this one out and build it.



4)  Step 6: This is completely internal to the VM-placement engine (and we 
could say this is “transparent” to Congress)



We should allow configuration of a solver (this itself could be a policy ☺ )

How to invoke the solver API ?



The domain-specific placement engine could send out to DSE (action_handler: 
data)?



I had always envisioned the solver being just a library of code—not an entity 
that sits on the DSE itself.

3)   Step 7 : Perform the migrations (according to the assignments computed in 
the step 6)

 This part invokes OpenStack API (to perform migrations).
 We may suppose that there are services implementing “action handlers”?
 It can listen on the DSE and execute the action.


That interface is supposed to exist by the Kilo release.  I’ll check up on the 
progress.


5)  Nova tables to use
Policy warning(id) :-
nova:host(id, name, service, zone, memory_capacity),
legacy:special_zone(zone),
ceilometer:statistics(id, memory, avg, count, duration,
 durstart, durend, max, min, period,
perstart, perend, sum, unit),
avg  0.75 * memory_capacity



I believe that ceilometer gives usage of VMs and not hosts. The host table 
(ComputeNode table) should give current used capacity.



Good to know.


6)  One of the issues highlighted in OpenStack (scheduler) and also 
elsewhere (e.g. Omega scheduler by google) is :

Reading “host utilization” state from the data bases and DB (nova:host table) 
updates

Re: [openstack-dev] [Congress][Delegation] Initial workflow design

2015-03-02 Thread Tim Hinrichs
Inline.

On Feb 26, 2015, at 3:32 PM, Ramki Krishnan 
r...@brocade.commailto:r...@brocade.com wrote:

1)
Ruby: One of the issues highlighted in OpenStack (scheduler) and also elsewhere 
(e.g. Omega scheduler by google) is :

Reading “host utilization” state from the data bases and DB (nova:host table) 
updates and overhead of maintaining in-memory state uptodate.

ð  This is expensive and current nova-scheduler does face this issue (many 
blogs/discussions).

  While the first goal is a PoC, this will likely become a concern in terms 
of adoption.


Tim: So you’re saying we won’t have fresh enough data to make policy decisions? 
 If the data changes so frequently that we can’t get an accurate view, then I’m 
guessing we shouldn’t be migrating based on that data anyway. Could you point 
me to some of these discussions?

Ramki: We have to keep in mind that VM migration could be an expensive 
operation depending on the size of the VM and various other factors; such an 
operation cannot be performed frequently.

2)
From document: As soon as the subscription occurs, the DSE sends the 
VM-placement engine the current contents of those tables, and when these 
tables change, the DSE informs the VM-placement engine in the form of diffs 
(aka deltas or updates).

Ramki: Is the criteria for table change programmable? This would be useful to 
generate significant change events based on application needs.

Not as of now.  We’ve kicked around the idea of changing a subscription from an 
entire table to an arbitrary slice of a table (expressed via Datalog).  That 
functionality will be necessary for dealing with large datasources like 
Ceilometer.  But we don’t have the design fleshed out or the people to build it.

Tim



Thanks,
Ramki

From: Tim Hinrichs [mailto:thinri...@vmware.com]
Sent: Thursday, February 26, 2015 10:17 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Congress][Delegation] Initial workflow design

Inline.

From: ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com 
ruby.krishnasw...@orange.commailto:ruby.krishnasw...@orange.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Wednesday, February 25, 2015 at 8:53 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Congress][Delegation] Initial workflow design

Hi Tim, All,


1) Step 3: The VM-placement engine is also a “datalog engine” . Right?

When policies are delegated:

when policies are inserted? When the VM-placement engine has already registered 
itself all policies are given to it?




“In our example, this would mean the domain-specific policy engine executes the 
following API call over the DSE”

ð “domain-agnostic” ….



Done.


2) Step 4:



Ok

But finally: if Congress will likely “delegate”



Not sure what you’re suggesting here.


3) Step 5:  Compilation of subpolicy to LP in VM-placement engine



For the PoC, it is likely that the LP program ( in PuLP or some other ML) is 
*not* completely generated by compiler/translator.

ð Right?

Where does the rest of the program originate?  I’m not saying the entire LP 
program is generated from the Datalog constraints; some of it is generated by 
the solver independent of the Datalog.  In the text, I gave the example of 
defining hMemUse[j].


 You also indicate that some category of constraints (“the LP solver 
doesn’t know what the relationship between assign[i][j], hMemUse[j], and 
vMemUse[i] actually is, so the VM-placement engine must also include 
constraints”) .
 These constraints must be “explicitly” written?  (e.g. max_ram_allocation 
etc that are constraints used in the solver-scheduler’s package).

The VM-placement engine does 2 things: (I) translates Datalog to LP and (ii) 
generates additional LP constraints.  (Both work items could leverage any 
constraints that are builtin to a specific solver, e.g. the solver-scheduler.  
The point is that there are 2 distinct, conceptual origins of the LP 
constraints: those that represent the Datalog and those that codify the domain.



 So what “parts” will be generated:
Cost function :
Constraint from Policy : memory usage  75%

 Then the rest should be “filled” up?

 Could we convene on an intermediary “modeling language”?
@Yathi: do you think we could use some thing like AMPL ? Is this 
proprietary?


A detail: the example “Y[host1] = hMemUse[host1]  0.75 * hMemCap[host1]”


ð To be changed to a linear form (mi – Mi  0 then Yi = 1 else Yi = 0) so 
something like (mi – Mi)  100 yi

Each domain-specific solver can do whatever it wants, so it’s not clear to me 
what the value of choosing a modeling language actually is—unless we want to 
build a library of common

Re: [openstack-dev] [Congress][Delegation] Initial workflow design

2015-03-02 Thread Tim Hinrichs
Hi Ramki,

Good suggestion.  I added a paragraph at the top of the doc in the Overview 
section to explain what Delegation means and mentioned that some policies won’t 
be delegated.

Tim

On Feb 26, 2015, at 3:15 PM, Ramki Krishnan 
r...@brocade.commailto:r...@brocade.com wrote:

Hi Tim, All,

The document is in great shape! Any global policies such as those impacting 
compute and network (e.g. CPU utilization and network bandwidth utilization) 
would be handled in Congress and not delegated. It would be worthwhile to 
capture this.

Thanks,
Ramki

From: Tim Hinrichs [mailto:thinri...@vmware.com]
Sent: Monday, February 23, 2015 11:28 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [Congress][Delegation] Initial workflow design


Hi all,



I made a heavy editing pass of the Delegation google doc, incorporating many of 
your comments and my latest investigations into VM-placement.  I left the old 
stuff in place at the end of the doc and put the new stuff at the top.



My goal was to propose an end-to-end workflow for a PoC that we could put 
together quickly to help us explore the delegation interface.  We should 
iterate on this design until we have something that we think is workable.   And 
by all means pipe up if you think we need a totally different starting point to 
begin the iteration.



(BTW I'm thinking of the integration with solver-scheduler as a long-term 
solution to VM-placement, once we get the delegation interface sorted out.)



https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit#https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1ksDilJYXV-2D5AXWON8PLMedDKr9NpS8VbT0jIy-5FMIEtI_editd=AwMFAgc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=CDHVp-YQeYXMYsCvaPXTGCHW_AvnWo0kL01u9sbygWEs=72yePfPFTFqBwUU4wSM095IqCvt4olhGxbgPjsMdx2Ue=



Tim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress][Delegation] Initial workflow design

2015-02-26 Thread Tim Hinrichs
 
of adoption.



So you’re saying we won’t have fresh enough data to make policy decisions?  If 
the data changes so frequently that we can’t get an accurate view, then I’m 
guessing we shouldn’t be migrating based on that data anyway.

Could you point me to some of these discussions?


7) While in this document you have changed the “example” policy, could we 
drill down the set of policies for the PoC (the server under utilization ?)


ð  As a reference

Sure.  The only reason I chose this policy was because it doesn’t have 
aggregation.  I’m guessing we’ll want to avoid aggregation for the POC because 
we don’t yet have it in Congress, and it complicates the problem of translating 
Datalog to LP substantially.

Tim

Ruby

De : Yathiraj Udupi (yudupi) [mailto:yud...@cisco.com]
Envoyé : mardi 24 février 2015 20:01
À : OpenStack Development Mailing List (not for usage questions); Tim Hinrichs
Cc : Debo Dutta (dedutta)
Objet : Re: [openstack-dev] [Congress][Delegation] Initial workflow design

Hi Tim,

Thanks for your updated doc on Delegation from Congress to a domain-specific 
policy engine, in this case, you are planning to build a LP-based VM-Placement 
engine to be the domain specific policy engine.
I agree your main goal is to first get the delegation interface sorted out.  It 
will be good so that external services (like Solver-Scheduler) can also easily 
integrate to the delegation model.

From the Solver-Scheduler point of view,  we would actually want to start 
working on a PoC effort to start integrating Congress and the Solver-Scheduler.
We believe rather than pushing this effort to a long-term,  it would add value 
to both the Solver Scheduler effort, as well as the Congress effort to try some 
early integration now, as most of the LP solver work for VM placements is ready 
available now in Solver scheduler, and we need to spend some time thinking 
about translating your domain-agnostic policy to constraints that the Solver 
scheduler can use.

I would definitely need your help from the Congress interfaces and I hope you 
will share your early interfaces for the delegation, so I can start the effort 
from the Solver scheduler side for integration.
I will reach out to you to get some initial help for integration w.r.t. 
Congress, and also keep you posted about the progress from our side.

Thanks,
Yathi.



On 2/23/15, 11:28 AM, Tim Hinrichs 
thinri...@vmware.commailto:thinri...@vmware.com wrote:


Hi all,



I made a heavy editing pass of the Delegation google doc, incorporating many of 
your comments and my latest investigations into VM-placement.  I left the old 
stuff in place at the end of the doc and put the new stuff at the top.



My goal was to propose an end-to-end workflow for a PoC that we could put 
together quickly to help us explore the delegation interface.  We should 
iterate on this design until we have something that we think is workable.   And 
by all means pipe up if you think we need a totally different starting point to 
begin the iteration.



(BTW I'm thinking of the integration with solver-scheduler as a long-term 
solution to VM-placement, once we get the delegation interface sorted out.)



https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit#https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1ksDilJYXV-2D5AXWON8PLMedDKr9NpS8VbT0jIy-5FMIEtI_editd=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=kF8jMOpogOhk8MJWvNMKJC3PiNImxWpZeD2o642YM2ss=8PV5EW-kz8Q9aP9riFbIjJXJNZXchx2NsL-Z3Y7E5Vge=



Tim

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Info on Currently Supported Data Drivers

2015-03-24 Thread Tim Hinrichs
Hi Bryan,

Inline.

On Mar 24, 2015, at 12:13 PM, Bryan Sullivan 
bls...@hotmail.commailto:bls...@hotmail.com wrote:

Hi, I have a question about where to find the complete set of currently 
supported data drivers (with table/row details) for Congress. The info at 
http://congress.readthedocs.org/en/latest/cloudservices.html#drivershttps://urldefense.proofpoint.com/v2/url?u=http-3A__congress.readthedocs.org_en_latest_cloudservices.html-23driversd=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=7K6Tq1uqgCY-gmJMedkayeCZgjUEWp7QD_HaJg6pHmYs=rEUZxD7aLCxwILxm2ugsR4Qr-4JXCdZ_-9BWcvG5F_ke=
 seems to cover only Nova and Neutron.  I'm sure that I can pull this info from 
the source at 
https://github.com/stackforge/congress/tree/master/congress/datasourceshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_congress_tree_master_congress_datasourcesd=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=7K6Tq1uqgCY-gmJMedkayeCZgjUEWp7QD_HaJg6pHmYs=XeNw_NiKGg3HrR5R6oVbN9EVV2RhB3CQ4waMWGvBAHMe=
 but I was looking for any documentation on the current state of that code's 
design. The existing blueprints (e.g. at 
https://github.com/stackforge/congress-specs)https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_congress-2Dspecs-29d=AwMFAwc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7sm=7K6Tq1uqgCY-gmJMedkayeCZgjUEWp7QD_HaJg6pHmYs=AjqWychcmYYsMd9oOGytLq3rXOqzhJGLbUTi2OuX2_0e=
 do not go into this detail AFAICT.


I’d definitely just do a directory listing on the source code.  Syncing the 
docs is something we do before a major release.  Things are just changing too 
quickly to do anything else.

Also, since I'm looking into how the current supported Congress tables will 
support use cases of the OPNFV Copper project [1], it will be also helpful to 
know where I would look for the set of potential policy-related data items 
inside OpenStack docs/code for the various components. If I find a gap against 
a use case, I would want to be able to specify related patches (e.g. where data 
source driver extensions should get the additional data) so I need to know 
where the data comes from (and what's available) inside OpenStack.


One of Congress’s goals is to make any data available via an API call to 
policy-related.  So anything the API calls of OpenStack returns are reasonable 
candidates for writing policy over.

Tim

[1] https://wiki.opnfv.org/copper/use_cases

Thanks for your help!
Bryan





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] Cores

2015-04-28 Thread Tim Hinrichs
Hi all,

I'm nominating Alex Yip for core status in Congress.  He's active on IRC and 
gerrit, fixes important bugs routinely, and developed a number of key features 
for the project:
- added a DSL that makes writing datasource-drivers much easier
- improved scale-up performance by orders of magnitude in several dimensions
- designed and implemented a distributed architecture to give Congress 
high-availability

I'm also planning to remove Derick Winkworth, who hasn't been active and has 
moved on to other things.

If there are no objections, I'll process these changes on Friday.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone] On dynamic policy, role hierarchies/groups/sets etc.

2015-05-06 Thread Tim Hinrichs
Hi all,

Inline.

From: Adam Young ayo...@redhat.commailto:ayo...@redhat.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, May 5, 2015 at 8:34 PM
To: 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [keystone] On dynamic policy, role 
hierarchies/groups/sets etc.

On 05/05/2015 07:05 AM, Henry Nash wrote:
We’ve been discussing changes to these areas for a while - and although I think 
there is general agreement among the keystone cores that we need to change 
*something*, we’ve been struggling to get agreement on exactly how..  So to try 
and ground the discussion that will (I am sure) occur in Vancouver, here’s an 
attempt to take a step back, look at what we have now, as well as where, 
perhaps, we want to get to.

This is a great summary.  Thanks Henry.

Super helpful for sure!



The core functionality all this is related to is that of how does keystone  
policy allow the checking of whether a given API call to an OpenStack service 
should be allowed to take place or not. Within OpenStack this is a two step 
process for an API caller….1) Get yourself a token by authentication and 
getting authorised for a particular scope (e.g. a given project), and then 2) 
Use that token as part of your API call to the service you are interested in. 
Assuming you do, indeed, have the rights to execute this API, somehow steps 1) 
and 2) give the policy engine enough info to say yes or no.

So first, how does this work today and (conceptually) how should we describe 
that?  Well first of all, in fact, strictly we don’t control access at the raw 
API level.  In fact, each service defines a series “capabilities” (which 
usually, but not always, map one-to-one with an API call).  These capabilities 
represent the finest grained access control we support via the policy engine. 
Now, in theory, the most transparent way we could have implemented steps 1) and 
2) above would have been to say that users should be assigned capabilities to 
projects….and then those capabilities would be placed in the token….allowing 
the policy engine to check if they match what is needed for a given capability 
to be executed. We didn’t do that since, a) this would probably end up being 
very laborious for the administrator (there would be lots of capabilities any 
given user would need), and b) the tokens would get very big storing all those 
capabilities. Instead, it was recognised that, usually, there are sets of these 
capabilities that nearly always go together - so instead let’s allow the 
creation of such sets….and we’ll assign those to users instead. So far, so 
good. What is perhaps unusual is how this was implemented. These capability 
sets are, today, called Roles…but rather than having a role definition that 
describes the capabilities represented by that role….instead roles are just 
labels - which can be assigned to users/projects and get placed in a tokens.  
The expansion to capabilities happens through the definition of a json policy 
file (one for each service) which must be processed by the policy engine in 
order to work out what whether the roles in a token and the role-capability 
mapping means that a given API can go ahead. This implementation leads to a 
number issues (these have all been raised by others, just pulling them together 
here):
As I understand how this works conceptually, a policy makes go/no-go decisions 
based on two kinds of properties: (1) properties about the user making the API 
call  (which are encoded in the token) and (2) the API call name and arguments. 
  Is that right?


i) The role-capability mapping is rather static. Until recently it had to be 
stored in service-specific files pushed out to the service nodes out-of-band. 
Keystone does now provide some REST APIs to store and retrieve whole policy 
files, but these are a) course-grained and b) not really used by services 
anyway yet.

ii) As more and more clouds become multi-customer (i.e. a cloud provider 
hosting multiple companies on a single OpenStack installation), cloud providers 
will want to allow those customers to administer “their bit of the cloud”. 
Keystone uses the Domains concept to allow a cloud provider to create a 
namespace for a customer to create their own projects, users and groups….and 
there is a version of the keystone policy file that allows a cloud provider to 
effectively delegate management of these items to an administrator of that 
customer (sometimes called a domain administrator).  However, Roles are not 
part of that namespace - they exists in a global namespace (within a keystone 
installation). Diverse customers may have different interpretations of what a 
“VM admin” or a “net admin” should be allowed to do for their bit of the cloud 
- but  right now that 

Re: [openstack-dev] [keystone] On dynamic policy, role hierarchies/groups/sets etc. - Role Assignment

2015-05-08 Thread Tim Hinrichs
Hi David,

See below.

On 5/7/15, 1:01 AM, David Chadwick d.w.chadw...@kent.ac.uk wrote:

Hi Tim

On 06/05/2015 21:53, Tim Hinrichs wrote:
 I wondered if we could properly protect the API call for adding a new
 Role using the current mechanism.  So I came up with a simple example.
 
 Suppose we want to write policy about the API call: addRole(user,
 role-name).  If we¹re hosting both Pepsi and Coke, we want to write a
 policy that says that only someone in the Pepsi admin role can change
 roles for Pepsi users (likewise for Coke).  We¹d want to write something
 likeŠ
 
 addRole(user, role) is permitted for caller if
 caller belongs to the Pepsi-admin role and
 user belongs to the Pepsi role
 
 The policy engine knows if ³caller belongs to the Pepsi-admin role²
 because that¹s part of the token.  But the policy engine doesn¹t know if
 ³user belongs to the Pepsi role² because user is just an argument to
 the API call, so we don¹t have role info about user.  This helps me
 understand *why* we can¹t handle the multi-customer use case right now:
 the policy engine doesn¹t have all the info it needs.
 
 But otherwise, it seems, we could handle the multi-customer use-case
 using mechanism that already exists.  Are there other examples where
 they can¹t write policy because the engine doesn¹t have enough info?
 

Your simple example does not work in the federated case. This is because
role and attribute assignments are not done by Keystone, or by any part
of Openstack, but by a remote IDP. It is assumed that the administrator
of this remote IDP knows who his users are, and will assign the correct
attributes to them. However, these are not necessarily OpenStack roles
(they most certainly wont be).

Therefore, we have built a perfectly good mechanism into Keystone, to
ensure that the users from any IDP (Coke, Pepsi or Virgin Cola etc.) get
the right Keystone/Openstack role(s), and this is via attibute mapping.
When the mapping takes place, the user is in the process of logging in,
therefore Keystone knows the attributes of the user (assigned by the
IDP) and can therefore know which Openstack role to assign to him/her.

I understand the idea of mapping attributes from a remote IDP to
OpenStack/Keystone roles.  But I don¹t understand the impact on my
example.  In my example, the policy statement fails to work for one of 2
reasons:

1. there¹s no such thing as a Pepsi-admin role
2. The policy engine can¹t check if ³user belongs to Pepsi

The policy statement fails to work because of (2) for sure.  But are you
saying it also fails to work because of (1) in the federated case?  I
would have thought that the Keystone roles used to represent the Pepsi IDP
attributes would be separate from the Keystone roles used to represent
Coke IDP attributes, and therefore there¹s be some role corresponding to
Pepsi-admin and Coke-admin.

Sorry if this is obvious.

Tim


I hope this helps.

regards

David

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] summit events

2015-05-12 Thread Tim Hinrichs
For those of you interested in the Congress events at the summit, here's a 
list.  That list is also available at:

https://libertydesignsummit.sched.org/?s=congress

1. Talks

(On delegation)
Helping Telcos go Green and save OpEx via Policy
Wed, May 20 2:40-3:20, Room 110
https://libertydesignsummit.sched.org/event/e449fb7aca1e5d8da3a555ef03c535a9#.VUOcf61VhBc

(Lab)
Congress Hands On Lab
Wed, May 20 4:30-6:00, Room 111/112
https://libertydesignsummit.sched.org/event/ea281f113990403ad416bcf192c9f509#.VUOcjK1VhBc

(Overview)
Congress: Introduction, Status, and Future Plans
Thu, May 21 3:10-3:50, Room 306
 
https://libertydesignsummit.sched.org/event/c64db4197dfd158458e3dd59f223a6e5#.VUOcRq1VhBc


2. Design sessions, where we discuss the work we plan to do in the next 6 
months.

Wed, May 20 1:50-2:30, Room 304
https://libertydesignsummit.sched.org/event/1d71a641d25ad3a6772fc7da918a59fc#.VUOZgq1VhBc

Thu, May 21 5:00-5:40p, Room 223
https://libertydesignsummit.sched.org/event/616217aebc4a00ddb85eb4df0fc91049#.VUOZyK1VhBc


Hope to see you there!
Tim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] PTL candidacy

2015-04-06 Thread Tim Hinrichs
Hi all,

I'm writing to announce my candidacy for PTL of Congress for the Liberty cycle. 
 We've made a lot of progress in Kilo, and I'm excited by what we'll achieve in 
Liberty!  To give us some perspective, I compiled a (partial) list of 
improvements we made in Kilo:

* Officially became part of OpenStack (moved from stackforge to openstack)
* Matured the Congress community (commits/LOC from outside VMware: Juno:8%, 
Kilo:26%)
* Integrated with our first external project: Murano
* Improved scale-up by orders of magnitude (see 
http://ruleyourcloud.com/2015/03/12/scaling-up-congress.html)
* Added the first version of reactive enforcement (correcting policy violations 
after they happen)
* Created a Horizon GUI for writing policy
* Built a DSL for integrating external cloud services and enabled on-demand 
(de)installation of services

We achieved just about everything we set out to do in Kilo, thanks to the hard 
work of many outstanding folks.  For the Liberty cycle, I'm planning on similar 
success in the following areas.

* Production deployments
* Scale-out: enabling high-availability and increased throughput
* Delegation: enabling Congress to interoperate with other policy engines to 
share the burden of policy enforcement
* Community: increasing non-VMware contributions, in terms of commits/LOC and 
especially reviews

While Congress is an early-stage project, we're on a great track, and I'll make 
sure we stay on it.  As always I'm happy to chat about future directions, past 
progress, and anything in between.

Thanks!
Tim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] [Delegation] Progress on PoC

2015-04-08 Thread Tim Hinrichs
Hi all,

I've been making steady progress on the pieces of the delegation PoC that I 
volunteered for.  Here's the usual design doc for reference.

https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit#

My main tasks were to spearhead the creation of a skeleton for this work and to 
work on the conversion of Datalog to LP.
- The skeleton has already been merged; you can find it in 
congress/policy_engines/vm_placement.py.
- The quite rough implementation of converting Datalog to LP can be found in 
review.

This note aims to give some details of the Datalog-to-LP conversion code so 
that others working on the PoC have the context for working on their pieces.  
Here are the relevant patches.

https://review.openstack.org/#/c/155537/
https://review.openstack.org/#/c/157612/
https://review.openstack.org/#/c/170311/

The toplevel routine for converting Datalog to LP is 
vm_placement.py:ComputePlacementEngine.policy_to_lp().  It returns two things: 
a formula to optimize and a list of constraints that must be satisfied.

For example, the test in 
congress/tests/policy_engines/test_vmplacement.py:TestPolicyToLp.test_policy_to_lp
 starts with the following policy.

warning(id) :-
nova:host(id, zone, memory_capacity),
legacy:special_zone(zone),
ceilometer:mem_consumption(id, avg),
mul(0.75, memory_capacity, three_quarters_mem),
lt(avg, three_quarters_mem)

and the following data

nova:host(123, 1, 10)
nova:host(456, 1, 10)
nova:server(789, alice, 123)
nova:server(101, bob, 123)
legacy:special_zone(1)
ceilometer:mem_consumption(123, 15)
ceilometer:mem_consumption(456, 20)

The call to policy_to_lp() returns (1) the optimization criteria, which aims to 
minimize the numer of warnings and (2) the set of hard constraints that must be 
satisfied.  The code returns the following.  (The syntax needs to be changed so 
that the VM placement engine utilizes the LP solver interface that we decide 
on.  So the main thing to take away from what follows is just seeing that we 
can take Datalog as input and output what is conceptually an LP program.)

1. the optimization criteria: (u'warning', 456) + (u'warning', 123)

Here the term (u'warning, 456) is an LP variable that is defined to be true 
exactly when warning(456) is true in the policy.  And we're using + in the 
usual LP way, meaning that we want to minimize the sum of those variables.  
(These are the Yi in the google doc.)

2. the hard constraints:

First we're defining the (u'warning, 456) LP decision variables to be true 
exactly when warning(456) is true in the policy: when the host's memory usage 
is less than 75% of its capacity.

(u'warning', 456) = (u'lt', u'memUsage[456]', 7.5)
(u'warning', 123) = (u'lt', u'memUsage[123]', 7.5)

In this case (u'lt', u'memUsage[456]', 7.5) is equivalent to memUsage[456]  
7.5, where memUsage[456] is an LP decision variable.  (Just noticed that I'm 
using 2 different syntaxes to represent LP decision variables, but that's not a 
huge deal since the syntax is just a placeholder anyway.)

Second we're codifying the relationship between memory usage for guests and 
memory usage for hosts and the assignment of guests to hosts.  These are the 
domain specific axioms we discussed.

('hMemUse', 456) = ('assign', 789, 456) * ('gMemUse', 789) + ('assign', 101, 
456) * ('gMemUse', 101)
('hMemUse', 123) = ('assign', 789, 123) * ('gMemUse', 789) + ('assign', 101, 
123) * ('gMemUse', 101)

Here ('assign', 789, 456) is the LP decision variable that says guest 789 is 
assigned host 456.
('gMemUse', 789) is the memory usage of guest 789.

(I'm sure we're missing some hard constraints, but I think we've got the 
hardest ones worked out.)

I'm on vacation all week and will be back at work next Tuesday.  I'll try to 
check email, but I won't be checking into any code/bugs in depth.  I just 
wanted to let everyone know what I've been up to so I'm not holding anyone up 
working on their piece of the PoC.

Tim?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Dynamic Policy for Access Control Subteam Meeting

2015-06-03 Thread Tim Hinrichs
As long as there's some way to get the *declarative* policy from the system
(as a data file or as an API call) that sounds fine.  But I'm dubious that
it will be easy to keep the API call that returns the declarative policy in
sync with the actual code that implements that policy.

Tim

On Wed, Jun 3, 2015 at 1:53 PM, Doug Hellmann d...@doughellmann.com wrote:

 Excerpts from Sean Dague's message of 2015-06-03 13:34:11 -0400:
  On 06/03/2015 12:10 PM, Tim Hinrichs wrote:
   I definitely buy the idea of layering policies on top of each other.
   But I'd worry about the long-term feasibility of putting default
   policies into code mainly because it ensures we'll never be able to
   provide any tools that help users (or other services like Horizon) know
   what the effective policy actually is.  In contrast, if the code is
 just
   an implementation of the API, and there is some (or perhaps several)
   declarative description(s) of which of those APis are permitted to be
   executed by whom, we can build tools to analyze those policies.  Two
   thoughts.
  
   1) If the goal is to provide warnings to the user about questionable
 API
   policy choices, I'd suggest adding policy-analysis functionality to say
   oslo_policy.  The policy-analysis code would take 2 inputs: (i) the
   policy and (ii) a list of policy properties, and would generate a
   warning if any of the properties are true for the given policy.   Then
   each project could provide a file that describes which policy
 properties
   are questionable, and anyone wanting to see the warnings run the
   functionality on that project's policy and the project's policy
 property
   file.
  
   It would definitely help me if we saw a handful of examples of the
   warnings we'd want to generate.
 
  WARN: server create permissions have been restricted from the default,
  this may impede operation and interoperability of your OpenStack
  installation
 
   2) If the goal is to provide sensible defaults so the system functions
   if there's no policy.json (or a dynamic policy cached from Keystone),
   why not create a default_policy.json file and use that whenever
   policy.json doesn't exist (or more precisely to use policy.json to
   override default_policy.json in some reasonable way).
 
  Because it's still a file, living in /etc. Files living in etc are
  things people feel they can modify. They are also things that don't
  always get deployed correctly with code deploys. People might not
  realize that default_policy.json is super important to be updated every
  time the code is rolled out.

 It doesn't have to live in /etc, though. It could be packaged in the
 nova code namespace as a data file, and accessed from there.

 Doug

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Dynamic Policy for Access Control Subteam Meeting

2015-06-03 Thread Tim Hinrichs
I definitely buy the idea of layering policies on top of each other.  But
I'd worry about the long-term feasibility of putting default policies into
code mainly because it ensures we'll never be able to provide any tools
that help users (or other services like Horizon) know what the effective
policy actually is.  In contrast, if the code is just an implementation of
the API, and there is some (or perhaps several) declarative description(s)
of which of those APis are permitted to be executed by whom, we can build
tools to analyze those policies.  Two thoughts.

1) If the goal is to provide warnings to the user about questionable API
policy choices, I'd suggest adding policy-analysis functionality to say
oslo_policy.  The policy-analysis code would take 2 inputs: (i) the policy
and (ii) a list of policy properties, and would generate a warning if any
of the properties are true for the given policy.   Then each project could
provide a file that describes which policy properties are questionable, and
anyone wanting to see the warnings run the functionality on that project's
policy and the project's policy property file.

It would definitely help me if we saw a handful of examples of the warnings
we'd want to generate.

2) If the goal is to provide sensible defaults so the system functions if
there's no policy.json (or a dynamic policy cached from Keystone), why not
create a default_policy.json file and use that whenever policy.json doesn't
exist (or more precisely to use policy.json to override default_policy.json
in some reasonable way).

Tim




On Wed, Jun 3, 2015 at 3:47 AM, Sean Dague s...@dague.net wrote:

 On 06/02/2015 06:27 PM, Morgan Fainberg wrote:
 
 
  On Tue, Jun 2, 2015 at 12:09 PM, Adam Young ayo...@redhat.com
  mailto:ayo...@redhat.com wrote:
 
  Since this a cross project concern, sending it out to the wider
  mailing list:
 
  We have a sub-effort in Keystone to do better access control policy
  (not the  Neutron or  Congress based policy efforts).
 
  I presented on this at the summit, and the effort is under full
  swing.  We are going to set up a subteam meeting for this, but would
  like to get some input from outside the Keystone developers working
  on it.  In particular, we'd like input from the Nova team that was
  thinking about hard-coding policy decisions in Python, and ask you,
  instead, to work with us to come up with a solution that works for
  all the service.
 
 
  I want to be sure we look at what Nova is presenting here. While
  building policy into python may not (on the surface) look like an
  approach that is wanted due to it restricting the flexibility that we've
  had with policy.json, I don't want to exclude the concept without
  examination. If there is a series of base level functionality that is
  expected to work with Nova in all cases - is that something that should
  be codified in the policy rules? This doesn't preclude having a mix
  between the two approaches (allowing custom roles, etc, but having a
  baseline for a project that is a known quantity that could be
 overridden).
 
  Is there real value (from a UX and interoperability standpoint) to have
  everything 100% flexible in all the ways? If we are working to redesign
  how policy works, we should be very careful of excluding the (more)
  radical ideas without consideration. I'd argue that dynamic policy does
  fall on the opposite side of the spectrum from the Nova proposal. In
  truth I'm going to guess we end up somewhere in the middle.

 I also don't think it's removing any flexibility at all. Moving the
 default policy into code is about having sane defaults encoded somewhere
 that we can analyze what people did with the policy, and WARN them when
 they did something odd. That odd might be an interop thing, it might
 also be 'you realize you disabled server creation, right, probably want
 to go look at that'.

 Our intent is this applies in layers.

 You start with policy in code, that's a set of defaults, which can be
 annotated with (WARN if policy is restricted further than these
 defaults) for specific rules.

 Then you apply policy.json as a set of overrides. Compute and emit any
 warnings.

 Where this comes into dynamic policy I think is interesting, because
 dynamic policy seems to require a few things.

 Where is the start of day origin seed for policy?

 There are a few options here. But if we think about a world where
 components are releasing on different schedules, and being upgraded at
 different times, it seems like the Nova installation has to be that
 source of original truth.

 So having a GET /policy API call that would provide the composite policy
 that Nova knows about (code + json patch) would make a lot of sense. It
 would make that discoverable to all kinds of folks on the network, not
 just Keystone. Win.

 This also seems like the only sane thing in a big tent world where
 Keystone might have a *ton* of projects in it's 

Re: [openstack-dev] [Congress] summit events

2015-06-09 Thread Tim Hinrichs
Hi Joe,

The telco slides are powerpoint, but are hosted on google drive:
https://docs.google.com/a/styra.com/file/d/0ByDz-eYOtswScUFUc1ZrVVhmQlk/edit

The slides themselves may not include all the content you want without the
soundtrack.  Here's the corresponding design doc.
https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit

Both these links are now on the Congress wiki as well.

Tim

On Tue, Jun 9, 2015 at 5:53 AM D'ANDREA, JOE (JOE) 
jdand...@research.att.com wrote:

 Tim,

 I really appreciated the Intro/Status talk, and I'm sorry to have missed
 the Telco talk. Are slides for those talks available online?

  On May 12, 2015, at 1:34 PM, Tim Hinrichs thinri...@vmware.com wrote:
 
  (On delegation)
  Helping Telcos go Green and save OpEx via Policy
  Wed, May 20 2:40-3:20, Room 110
 
 https://libertydesignsummit.sched.org/event/e449fb7aca1e5d8da3a555ef03c535a9#.VUOcf61VhBc
 
  (Overview)
  Congress: Introduction, Status, and Future Plans
  Thu, May 21 3:10-3:50, Room 306
 
 https://libertydesignsummit.sched.org/event/c64db4197dfd158458e3dd59f223a6e5#.VUOcRq1VhBc

 jd

 —
 Joe D’Andrea
 ATT Labs - Research
 Cloud Technologies  Services
 Bedminster, NJ




 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Dynamic Policy for Access Control Subteam Meeting

2015-06-04 Thread Tim Hinrichs
Inline.

On Thu, Jun 4, 2015 at 6:40 AM, Sean Dague s...@dague.net wrote:

 On 06/04/2015 08:52 AM, Adam Young wrote:
  On 06/04/2015 06:32 AM, Sean Dague wrote:
  On 06/03/2015 08:40 PM, Tim Hinrichs wrote:
  As long as there's some way to get the *declarative* policy from the
  system (as a data file or as an API call) that sounds fine.  But I'm
  dubious that it will be easy to keep the API call that returns the
  declarative policy in sync with the actual code that implements that
  policy.
  Um... why? Nova (or any other server project) needs to know what the
  currently computed policy is to actually enforce it internally. Turning
  around and spitting that back out on the wire is pretty straight
 forward.
 
  Is there some secret dragon I'm missing here?
 
  No.  But it is a significant bit of coding to do;  you would need to
  crawl every API and make sure you hit every code path that could enforce
  policy.

 Um, I don't understand that.

 I'm saying that you'd GET https://my.nova.api.server/policy;

 And it would return basically policy.json. There is no crawling every
 bit, this is a standard entry point to return a policy representation.
 Getting all services to implement this would mean that Keystone could
 support interesting policy things with arbitrary projects, not just a
 small curated list, which is going to be really important in a big tent
 world. Monasca and  Murano are just as important to support here as Nova
 and Swift.



Definitely agree it'd be great to have an API call that returns policy.
The question that I think Adam and I are trying to answer is how do
projects implement that call?  We've (perhaps implicitly) suggested 3
different options.

1. Have a data file called say 'default_policy.json' that the oslo-policy
engine knows how to use (and override with policy.json or whatever).  The
policy-API call that returns policy then just reads in this file and
returns it.

2. Hard-code the return value of the Python function that implements the
policy-API call.  Different options as to how to do this.

3. Write code that automatically generates the policy-API result by
analyzing the code that implements the rest of the API calls (like
create_vm, delete_vm) and extracting the policy that they implement.  This
would require hitting all code paths that implement policy, etc.

I'm guessing you had option (2) in mind.  Is that right?  Assuming that's
the case I see two possibilities.

a. The policy-API call is used internally by Nova to check that an API call
is permitted before executing it.  (I'm talking conceptually.  Obviously
you'd not go through http.)

b. The policy-API call is never used internally; rather, each of the other
API calls (like create-server, delete-server) just use arbitrary Python
logic to decide whether an API call is permitted or not.  This requires the
policy-API call implementation to be kept in sync manually with the other
API calls to ensure the policy-API call returns the actual policy.

I'd be happy with (a) and doubt the practicality of (b).

Tim




 However, I've contemplated doing something like that with
  oslo.policy already;  run a workload through a server with policy
  non-enforcing (Permissive mode) and log the output to a file, then use
  that output to modify either the policy or the delegations (role
  assignments or trusts) used in a workflow.
 
  The Hard coded defaults worry me, though.  Nova is one piece (a big one,
  admittedly) of a delicate dance across multiple (not-so-micro) services
  that make up OpenStack.  Other serivces are going to take their cue from
  what Nova does, and that would make the overall flow that much harder to
  maintain.

 I don't understand why having hard coded defaults makes things harder,
 as long as they are discoverable. Defaults typically make things easier,
 because people then only change what they need, instead of setting a
 value for everything, having the deployment code update, and making
 their policy miss an important thing, or make something wrong because
 they didn't update it correctly at the same time as code.

  I think we need to break some very ingrained patterns in out policy
  enforcement.  I would worry that enforcing policy in code would give us
  something that we could not work around.  Instead, I think we need to
  ensure that the  Nova team leads the rest of the OpenStack core services
  in setting up best practices, and that is primarily a communication
  issue.  Getting to a common understanding of RBAC, and making it clear
  how roles are modified on a per-api basis will make Nova more robust.

 So I feel like I understand the high level dynamic policy end game. I
 feel like what I'm proposing for policy engine with encoded defaults
 doesn't negatively impact that. I feel there is a middle chunk where
 perhaps we've got different concerns or different dragons that we see,
 and are mostly talking past each other. And I don't know how to bridge
 that. All the keystone specs I've

Re: [openstack-dev] [Congress] Mid-cycle sprint

2015-06-18 Thread Tim Hinrichs
Hi Adam,

Thanks for the offer!  It'd definitely be great to join you all in Boston,
but I'm guessing the logistics aren't going to work out.  Most of the
people I've heard from are in the Bay Area, which makes it hard to host in
Boston.But I'll keep it in mind and float it by everyone.

Tim



On Wed, Jun 17, 2015 at 7:20 PM Adam Young ayo...@redhat.com wrote:

  How many people do you think you will have?  We have a midcycle in
 Boston University, July 15-17 and you are welcome to join in.  I am pretty
 sure we will have more than enough capacity, considering the size of the
 Congress team.

 Hotels might be a bit of an issue, as we are getting close, and they are
 starting to book up, but the BU admin has let us know that we can get dorm
 space is people so desire.



 On 06/16/2015 05:13 PM, Tim Hinrichs wrote:

 Hi all,

  In the last couple of IRCs we've been talking about running a mid-cycle
 sprint focused on enabling our message bus to span multiple processes and
 multiple hosts.  The message bus is what allows the Congress policy engine
 to communicate with the Congress wrappers around external services like
 Nova, Neutron.  This cross-process, cross-host message bus is the platform
 we'll use to build version 2.0 of our distributed architecture.

  If you're interested in participating, drop me a note.  Once we know
 who's interested we'll work out date/time/location details.

  Thanks!
 Tim



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

  __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Mid-cycle sprint

2015-06-19 Thread Tim Hinrichs
Hi Zhou,

The requirements for the request_refresh API call came from a customer.  It
wasn't intended as a replacement for the functionality we'd get by
integrating with oslo.messaging.  But it was a simple feature to add, with
nice properties from Congress's perspective, driven by a real use case.
It'd be great to support a more standard kind of streaming as well, such as
with oslo.messaging.

Tim

On Thu, Jun 18, 2015 at 11:10 PM Zhou, Zhenzan zhenzan.z...@intel.com
wrote:

  Hi, Tim



 I have looked at the request_fresh_action. One big question is that it
 requires external services to queue up changes and then call this Congress
 API at some point. I’m not sure they would buy in this design as many
 projects have notification mechanism ready now and Ceilometer heavily
 depends on these notifications.  So basically I prefer to use
 oslo.messaging. But this API is still useful for other external services
 that don’t use oslo.messaging notification to publish changes and are
 willing to integrate with Congress this way.

 Anyway, I can upload my draft bp for review at first.

 Thanks.



 BR

 Zhou Zhenzan



 *From:* Tim Hinrichs [mailto:t...@styra.com]
 *Sent:* Wednesday, June 17, 2015 21:57


 *To:* OpenStack Development Mailing List (not for usage questions)

 *Subject:* Re: [openstack-dev] [Congress] Mid-cycle sprint



 Hi Zhenzan,

 Yes the oslo.messaging integration task is relevant--oslo.messaging is one
 way of achieving cross-process, cross-host messaging.  So I'll count you as
 interested for the mid-cycle sprint.



 Have you looked at the API call that forces a datasource driver to pull
 immediately?   See
 congress/api/datasource_model.py:DatasourceModel.request_refresh_action.  We
 had envisioned using that to implement a kind of notification from external
 services as follows.  The external service queues up a list of changes and
 when the queue is long enough runs the API call to force the datasource
 driver  hooked up to that service to pull those changes and then publish
 them on the bus.  So it's not exactly streaming updates from the external
 service (which is good so that Congress can easily rate-limit updates), but
 it has almost the same effect.



 Tim



 On Tue, Jun 16, 2015 at 6:02 PM Zhou, Zhenzan zhenzan.z...@intel.com
 wrote:

  Hi, Tim



 Is this the oslo.messaging integration task? I’m interested in
 participating. Actually I am working on a bp to receive notifications from
 external services in datasource driver at first. I’m ok to change if the
 direction is to integrate oslo.messaging thoroughly (even replacing DSE).

 Thanks.



 BR

 Zhou Zhenzan



 *From:* Tim Hinrichs [mailto:t...@styra.com]
 *Sent:* Wednesday, June 17, 2015 05:14
 *To:* OpenStack Development Mailing List (not for usage questions)
 *Subject:* [openstack-dev] [Congress] Mid-cycle sprint



 Hi all,



 In the last couple of IRCs we've been talking about running a mid-cycle
 sprint focused on enabling our message bus to span multiple processes and
 multiple hosts.  The message bus is what allows the Congress policy engine
 to communicate with the Congress wrappers around external services like
 Nova, Neutron.  This cross-process, cross-host message bus is the platform
 we'll use to build version 2.0 of our distributed architecture.



 If you're interested in participating, drop me a note.  Once we know who's
 interested we'll work out date/time/location details.



 Thanks!

 Tim



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] Hands-on-lab setup instructions

2015-06-24 Thread Tim Hinrichs
At the IRC this week, Bryan asked how he could replicate the environment we
used at the Vancouver hands-on-lab.  So we added some basic instructions at
the end of the lab write-up.

https://docs.google.com/document/u/1/d/1lXmMkUhiSZYK45POd5ungPjVR--Fs_wJHeQ6bXWwP44/pub

Bryan: hope that helps.  Let us know if you have trouble.  We're happy to
help.  And if you're interested, it'd be great to work with you to get
those instructions in better shape!

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] IRC on 6/30 canceled

2015-06-26 Thread Tim Hinrichs
Hi all,

Since almost all the cores are on vacation next week, we're canceling the
IRC meeting on 6/30.  Expect slow responses for reviews and email all next
week.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [congress] table with named parameters in rule's left side ?

2015-06-11 Thread Tim Hinrichs
Hi Radek,

1. You can't use column references on any table except those created by a
datasource, mainly because Congress doesn't know the name of the columns
for those tables.  We've kicked around the idea of letting policy-writers
declare the column names of any table, but it's trickier than it sounds.

2. Even if we solved (1), there's not much point in using column-references
in the head (left-hand-side) of a rule because you always need to include
all of the columns anyway.  That restriction is baked deeply into the
semantics of the language: if you leave out a column in the head the size
of the table is infinite.  The only benefit to column-references in the
head would be enabling you to specify the columns in any order.

3. I've thought about tweaking the semantics of Datalog so that it includes
both tuples and dictionaries.  But that is a major effort.

4. We don't actually allow column-references even within execute[
action(...)] because of (2).

5.  In your example, can you write it without the column references?  I
don't see the predeploy_modify rule to understand why you need column
references in the query.

 predeploy_modify(uuid,uuid,'add_property', 5, name=image”,
value=uuid)

Tim



On Thu, Jun 11, 2015 at 6:44 AM Pospisil, Radek radek.pospi...@hp.com
wrote:

  Hello,



 Is it possible to have named parameters on rule’s left side, for example:



 1.   predeploy_modify(eid,oid,add_property,5, name=”image”,
 value=pvalue) :- …. Glancev2:images(name=pvalue,…)

 2.   predeploy_modify(eid,oid,add_object, 10, type=”monitoring”,
 port=8190, path=”/aa/bb”) :- ….



 My Usecase is following – I want to use Congress to provide me list of
 ‘actions’ that I have to do on Murano environment prior it is deployed –
 for example:

 · I want to enforce/use specific image for given component(s) in
 environment, so I set corresponding property (example 1)

 · I want to add an object (monitoring) with properties to
 environment  (example 2)



 Thus generally “I want to query for one rule with variable set/dictionary
 of arguments” – for example using simulation API:

 $ openstack congress policy simulate test 'predeploy_modify()' '' action

 Result will be

 predeploy_modify(uuid,uuid,'add_property', 5, name=image”,
 value=uuid)

 predeploy_modify(uuid,uuid,'add_object', 10, type=monitoring,
 port=…, path=…)







 The problem is that if I do (as an example):



 $  openstack congress policy rule create test a(o,p,q=4) :- b(o),b(p)

 ERROR: openstack Syntax error for rule::Compiler found errors:Atom a(q=4)
 uses named parameters but the columns for table a have not been declared.

 Atom a(q=4) uses named parameters but the columns for table a have not
 been declared. (HTTP 400) (Request-ID:
 req-b05683e6-766a-4be8-a238-5811eb6f7125)





 So is it possible to use named parameters on not-declared tables?

 I know that it is possible to do for “execute[ action(…) ] :- “, where
 positional and keyword/named set of parameters is passed to action-executor.





   Regards,



 Radek

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] summit events

2015-06-15 Thread Tim Hinrichs
Hi Joe,

Here's the link to the Intro to Congress slide deck.
https://docs.google.com/file/d/0ByDz-eYOtswScTlmamlhLXpmTXc/edit

While I'm at it, here are the instructions for the Congress Hands On Lab.
https://docs.google.com/document/u/1/d/1lXmMkUhiSZYK45POd5ungPjVR--Fs_wJHeQ6bXWwP44/pub

All these links are available on the wiki.
https://wiki.openstack.org/wiki/Congress

Tim


On Mon, Jun 15, 2015 at 7:51 AM D'ANDREA, JOE (JOE) 
jdand...@research.att.com wrote:

 Tim,

  On Jun 9, 2015, at 10:04 AM, Tim Hinrichs t...@styra.com wrote:
 
  Hi Joe,
 
  The telco slides are powerpoint, but are hosted on google drive ...

 Thanks very much for the OpEx slides (and corresponding design doc)!

 Is there a deck link for Congress: Introduction, Status, and Future
 Plans as well?

 The OpEx slides look similar to a degree, but I recall the other deck went
 into different detail and made for a really nice slide-based summary of
 things.

 Please advise. Thanks!

 jd

 —
 Joe D’Andrea
 ATT Labs - Research
 Cloud Technologies  Services
 Bedminster, NJ



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] Mid-cycle sprint

2015-06-16 Thread Tim Hinrichs
Hi all,

In the last couple of IRCs we've been talking about running a mid-cycle
sprint focused on enabling our message bus to span multiple processes and
multiple hosts.  The message bus is what allows the Congress policy engine
to communicate with the Congress wrappers around external services like
Nova, Neutron.  This cross-process, cross-host message bus is the platform
we'll use to build version 2.0 of our distributed architecture.

If you're interested in participating, drop me a note.  Once we know who's
interested we'll work out date/time/location details.

Thanks!
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] Mid-cycle sprint

2015-06-17 Thread Tim Hinrichs
Hi Zhenzan,

Yes the oslo.messaging integration task is relevant--oslo.messaging is one
way of achieving cross-process, cross-host messaging.  So I'll count you as
interested for the mid-cycle sprint.

Have you looked at the API call that forces a datasource driver to pull
immediately?   See
congress/api/datasource_model.py:DatasourceModel.request_refresh_action.  We
had envisioned using that to implement a kind of notification from external
services as follows.  The external service queues up a list of changes and
when the queue is long enough runs the API call to force the datasource
driver  hooked up to that service to pull those changes and then publish
them on the bus.  So it's not exactly streaming updates from the external
service (which is good so that Congress can easily rate-limit updates), but
it has almost the same effect.

Tim

On Tue, Jun 16, 2015 at 6:02 PM Zhou, Zhenzan zhenzan.z...@intel.com
wrote:

  Hi, Tim



 Is this the oslo.messaging integration task? I’m interested in
 participating. Actually I am working on a bp to receive notifications from
 external services in datasource driver at first. I’m ok to change if the
 direction is to integrate oslo.messaging thoroughly (even replacing DSE).

 Thanks.



 BR

 Zhou Zhenzan



 *From:* Tim Hinrichs [mailto:t...@styra.com]
 *Sent:* Wednesday, June 17, 2015 05:14
 *To:* OpenStack Development Mailing List (not for usage questions)
 *Subject:* [openstack-dev] [Congress] Mid-cycle sprint



 Hi all,



 In the last couple of IRCs we've been talking about running a mid-cycle
 sprint focused on enabling our message bus to span multiple processes and
 multiple hosts.  The message bus is what allows the Congress policy engine
 to communicate with the Congress wrappers around external services like
 Nova, Neutron.  This cross-process, cross-host message bus is the platform
 we'll use to build version 2.0 of our distributed architecture.



 If you're interested in participating, drop me a note.  Once we know who's
 interested we'll work out date/time/location details.



 Thanks!

 Tim



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [congress] mid cycle Sprint dates

2015-07-01 Thread Tim Hinrichs
Hi all!

We settled on dates and location for the Congress mid cycle sprint:

Aug 6-7
VMware campus, Palo Alto, CA

Please RSVP if you plan to come so we can get a headcount.

Hope to see you there!

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Murano][Congress] Application placement use case

2015-07-06 Thread Tim Hinrichs
Sorry--hit the Send button by accident.


 Hi Gosha,

 This definitely sounds like an interesting use case for Congress.  Keep in
 mind that Congress doesn't itself do placement (though we did some
 experimentation with that [1][2]).  Some thoughts.

 1. Let's suppose Murano/Congress/etc. allow us to figure out which app
 should be deployed in which region.  Is there a separate Nova for each
 region that can do the actual scheduling?  If not, how would Murano force
 the app to be deployed in the proper region?

 2. Let's assume Murano can force the app to the proper region.  One option
 for using Congress to compute the proper region would be to write policy
 statements that enumerate the permitted regions for a given application,
 and then ask Congress for one of those regions.  For your suggested
 policies above, we could write the following datalog statements



 Solaris is required then select Region_Solaris. 

 permitted_region(app, region_solaris) :-
 murano:app_requires(app, solaris)


A. If Solaris is required then select region_solaris

permitted_region(app, region_solaris) :-
murano:app_requires(app, solaris)

B. If Solaris is required then select less loaded regions from
[Region_Solaris1, Region_Solaris2]

This one requires additional expressiveness in the language than we
currently have (because it asks to minimize over several options).  But it
would be something like...

best_region(app, min(y)) :-
permitted_region(app, x),
ceilometer:region_load(x, y)




 [1] Design doc:
 https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit
 [2] Slides:
 https://drive.google.com/a/styra.com/file/d/0ByDz-eYOtswScUFUc1ZrVVhmQlk/view



Tim


 On Thu, Jul 2, 2015 at 7:36 AM Georgy Okrokvertskhov 
 gokrokvertsk...@mirantis.com wrote:

 Hi,

 All applications are monolithic so they don't need to be split over
 multiple regions. It is necessary to have an ability to select a region
 based on requirements and for now I don't care how they are placed inside
 the region.
 I am not sure how region's capabilities will be stored and actually this
 is a reason why I am asking if Congress will solve this. I can imagine a
 policy which says if Solaris is required then select Region_Solaris. Or
 more complex If Solaris is required then select less loaded regions from
 [Region_Solaris1, Region_Solaris2]

 In this use case Murano will deploy complex environment which consist of
 multiple atomic applications with different requirements, so deployment
 will be across clouds but for whole environment. Imagine IBM MQ on AIX and
 PowerPC + Oracle DB on Solaris + Microsoft IIS on Windows 2012  HyperV +
 WebSphere on RHEL  KVM.

 Thanks
 Gosha



 On Wed, Jul 1, 2015 at 10:17 PM, ruby.krishnasw...@orange.com wrote:

  Hi





 Did you mean placement at “two levels”. First to select the region and
 then within each region, Nova scheduler will place on hosts.



 But where will the capabilities of each region (based on which placement
 decision will be made) be stored? Will each region be queried to obtain
 this information?

  Will a single application need to be placed (split across) different
 regions?



  Ruby



 *De :* Georgy Okrokvertskhov [mailto:gokrokvertsk...@mirantis.com]
 *Envoyé :* mercredi 1 juillet 2015 21:26
 *À :* OpenStack Development Mailing List
 *Objet :* [openstack-dev] [Murano][Congress] Application placement use
 case



 Hi,



 I would like to share with the community one of the real use case which
 we saw while working with one of the Murano customer and ask an advice.
 This customer has multiple OpenStack regions which are serving for
 different hypervisors. The reason for that is Oracle OpenStack which is
 used to work with Solaris on top of SPARC architecture. There are other
 hypervisors KVM and VMWare which are used.

 There are multiple applications which should be placed to different
 regions based on their requirements (OS, Hypervisor, networking
 restrictions). As there is no single cloud, Nova scheduler can’t be used
 (at least in my understanding) so we need to have some placement policies
 to put applications properly. And obviously we don’t want to ask end user
 about the placement.



 Right now in Murano we can do this by:

 1.Hardcoding placement inside application. This approach will work
 and does not require any significant change in Murano. But there are
 obvious limitations like if we have two options for placement which one
 should be hardcoded.

 2.Create special placement scheduler application\class in Murano
 which will use some logic to place applications properly. This is better
 approach as nothing is hard coded in applications except their
 requirements. Applications will just have a workflow to ask placement
 scheduler for a decision and then will just use this decision.

 3.Use some external tool or OpenStack component for placement
 decision. This is a very generic use case which we saw multiple times.
 Tools like 

Re: [openstack-dev] [congress] mid cycle Sprint dates

2015-07-06 Thread Tim Hinrichs
I added Congress to the list of Liberty sprints and built a wiki page with
basic info:

https://wiki.openstack.org/wiki/Sprints/CongressLibertySprint

Please RSVP using eventbrite:
http://www.eventbrite.com/e/congress-liberty-midcycle-sprint-tickets-17654731778

Tim


On Wed, Jul 1, 2015 at 8:58 AM Jeremy Stanley fu...@yuggoth.org wrote:

 On 2015-07-01 15:40:31 + (+), Tim Hinrichs wrote:
  We settled on dates and location for the Congress mid cycle sprint:
 [...]

 Invoking the spirit of Thierry, please remember to add it to the
 list at:

 https://wiki.openstack.org/wiki/Sprints#Liberty_sprints

 --
 Jeremy Stanley

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Murano][Congress] Application placement use case

2015-07-06 Thread Tim Hinrichs
Hi Gosha,

This definitely sounds like an interesting use case for Congress.  Keep in
mind that Congress doesn't itself do placement (though we did some
experimentation with that [1][2]).  Some thoughts.

1. Let's suppose Murano/Congress/etc. allow us to figure out which app
should be deployed in which region.  Is there a separate Nova for each
region that can do the actual scheduling?  If not, how would Murano force
the app to be deployed in the proper region?

2. Let's assume Murano can force the app to the proper region.  One option
for using Congress to compute the proper region would be to write policy
statements that enumerate the permitted regions for a given application,
and then ask Congress for one of those regions.  For your statements above,
we could write the following policy statements

Solaris is required then select Region_Solaris. Or more complex If Solaris
is required then select less loaded regions from [Region_Solaris1,
Region_Solaris2]

permitted_region(app, region) :-
murano:app_requires(app, solaris),

[1] Design doc:
https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit
[2] Slides:
https://drive.google.com/a/styra.com/file/d/0ByDz-eYOtswScUFUc1ZrVVhmQlk/view


On Thu, Jul 2, 2015 at 7:36 AM Georgy Okrokvertskhov 
gokrokvertsk...@mirantis.com wrote:

 Hi,

 All applications are monolithic so they don't need to be split over
 multiple regions. It is necessary to have an ability to select a region
 based on requirements and for now I don't care how they are placed inside
 the region.
 I am not sure how region's capabilities will be stored and actually this
 is a reason why I am asking if Congress will solve this. I can imagine a
 policy which says if Solaris is required then select Region_Solaris. Or
 more complex If Solaris is required then select less loaded regions from
 [Region_Solaris1, Region_Solaris2]

 In this use case Murano will deploy complex environment which consist of
 multiple atomic applications with different requirements, so deployment
 will be across clouds but for whole environment. Imagine IBM MQ on AIX and
 PowerPC + Oracle DB on Solaris + Microsoft IIS on Windows 2012  HyperV +
 WebSphere on RHEL  KVM.

 Thanks
 Gosha



 On Wed, Jul 1, 2015 at 10:17 PM, ruby.krishnasw...@orange.com wrote:

  Hi





 Did you mean placement at “two levels”. First to select the region and
 then within each region, Nova scheduler will place on hosts.



 But where will the capabilities of each region (based on which placement
 decision will be made) be stored? Will each region be queried to obtain
 this information?

  Will a single application need to be placed (split across) different
 regions?



  Ruby



 *De :* Georgy Okrokvertskhov [mailto:gokrokvertsk...@mirantis.com]
 *Envoyé :* mercredi 1 juillet 2015 21:26
 *À :* OpenStack Development Mailing List
 *Objet :* [openstack-dev] [Murano][Congress] Application placement use
 case



 Hi,



 I would like to share with the community one of the real use case which
 we saw while working with one of the Murano customer and ask an advice.
 This customer has multiple OpenStack regions which are serving for
 different hypervisors. The reason for that is Oracle OpenStack which is
 used to work with Solaris on top of SPARC architecture. There are other
 hypervisors KVM and VMWare which are used.

 There are multiple applications which should be placed to different
 regions based on their requirements (OS, Hypervisor, networking
 restrictions). As there is no single cloud, Nova scheduler can’t be used
 (at least in my understanding) so we need to have some placement policies
 to put applications properly. And obviously we don’t want to ask end user
 about the placement.



 Right now in Murano we can do this by:

 1.Hardcoding placement inside application. This approach will work
 and does not require any significant change in Murano. But there are
 obvious limitations like if we have two options for placement which one
 should be hardcoded.

 2.Create special placement scheduler application\class in Murano
 which will use some logic to place applications properly. This is better
 approach as nothing is hard coded in applications except their
 requirements. Applications will just have a workflow to ask placement
 scheduler for a decision and then will just use this decision.

 3.Use some external tool or OpenStack component for placement
 decision. This is a very generic use case which we saw multiple times.
 Tools like CIRBA are often used for this. Murano will need an interface to
 ask these tools. I think about this solution as an extension of 2.



 I am aware that Murano is working on integration with Congress and I am
 looking for an opportunity here to address real use case of Murano usage in
 real customer environment. It will be great to know if OpenStack can offer
 something here without involving 3rd party tools. I suspect that this is a
 good use case for Congress, but I would like 

[openstack-dev] [Congress] [Murano] multiple numbers of arguments for table

2015-08-18 Thread Tim Hinrichs
Hi all,

We're contemplating a small syntax change in the Congress policy language
and wanted to see if it would cause anyone problems.

Currently you can write rules that give a single table differing numbers of
columns.  In the following example, the 'error' table has both 1 column and
2 columns.

error(vm) :- blah blah
error(vm, net) :- blah blah

We're contemplating removing this flexibility and making every table have a
fixed number of columns.  (We only support differing numbers of columns in
very special cases anyway.)

Would this cause any problems?  Murano team?

Thanks,
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [congress] [murano] Congress jenkins job is failing

2015-08-18 Thread Tim Hinrichs
Looks like we merged a database schema change without the migration
script.  I'm on it.  (We'll get our tempest tests running in gate again
ASAP.)

Tim


On Tue, Aug 18, 2015 at 7:36 AM Filip Blaha filip.bl...@hp.com wrote:

 Hi Congress team.

 Our jenkins job testing integration congress and murano is failing.
 Congress fails to start on some DB error [1] . Does anyone know what
 could be the problem?

 [1]

 http://logs.openstack.org/08/211608/3/check/gate-murano-congress-devstack-dsvm/db1b682/logs/screen-congress.txt.gz#_2015-08-18_10_17_05_033

 Thanks
 Filip

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [congress] [murano] Congress jenkins job is failing

2015-08-18 Thread Tim Hinrichs
Hi Filip,

I just submitted a revert of the problematic change.  Once it merges, all
should be well again.  Sorry for the trouble.

Tim

On Tue, Aug 18, 2015 at 7:57 AM Tim Hinrichs t...@styra.com wrote:

 Looks like we merged a database schema change without the migration
 script.  I'm on it.  (We'll get our tempest tests running in gate again
 ASAP.)

 Tim


 On Tue, Aug 18, 2015 at 7:36 AM Filip Blaha filip.bl...@hp.com wrote:

 Hi Congress team.

 Our jenkins job testing integration congress and murano is failing.
 Congress fails to start on some DB error [1] . Does anyone know what
 could be the problem?

 [1]

 http://logs.openstack.org/08/211608/3/check/gate-murano-congress-devstack-dsvm/db1b682/logs/screen-congress.txt.gz#_2015-08-18_10_17_05_033

 Thanks
 Filip

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [congress][requirements] Requirements proposal job fails

2015-08-20 Thread Tim Hinrichs
Thanks for sorting that out.  +1ed the change.

Tim


On Thu, Aug 20, 2015 at 12:53 PM Andreas Jaeger a...@suse.com wrote:

 On 08/20/2015 09:32 PM, Robert Collins wrote:
  It was just removed from global requirements, because it was not used.
  That's clearly wrong. So let's refer that and add it back.

 Ah, that explains it, let's revert the change:
 https://review.openstack.org/#/c/215306/

 Thanks,
 Andreas
 --
   Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
 GF: Felix Imendörffer, Jane Smithard, Graham Norton,
 HRB 21284 (AG Nürnberg)
  GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] meeting time change

2015-07-31 Thread Tim Hinrichs
Hi all,

We managed to find a day/time where all the active contributors can attend
(without being up too early/late).  The room, day, and time have all
changed.

Room: #openstack-meeting-2
Time: Wednesday 5p Pacific = Thursday midnight UTC

Next week we begin with this new schedule.

And don't forget that next week Thu/Fri is our Mid-cycle sprint.  Hope to
see you there!

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Congress] meeting time change

2015-07-31 Thread Tim Hinrichs
Peter pointed out that no one uses #openstack-meeting-2.  So we'll go with
#openstack-meeting.  Here are the updated meeting details.

Room: #openstack-meeting
Time: Wednesday 5p Pacific = Thursday midnight UTC

There's a change out for review that will update the meeting website once
it merges.
http://eavesdrop.openstack.org/#Congress_Team_Meeting
https://review.openstack.org/#/c/207981/

Tim

On Fri, Jul 31, 2015 at 9:24 AM Tim Hinrichs t...@styra.com wrote:

 Hi all,

 We managed to find a day/time where all the active contributors can attend
 (without being up too early/late).  The room, day, and time have all
 changed.

 Room: #openstack-meeting-2
 Time: Wednesday 5p Pacific = Thursday midnight UTC

 Next week we begin with this new schedule.

 And don't forget that next week Thu/Fri is our Mid-cycle sprint.  Hope to
 see you there!

 Tim


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


  1   2   3   >