Re: [openstack-dev] [congress] Jenkins failure

[Tim Hinrichs]

Some docs got merged last night.  I think that if you fetch the latest and 
rebase that you’ll end up with conflicts that you need to resolve manually.  
The Readme file has changed a lot, so I’m sure git doesn’t know how to merge 
your changes in.

Tim

On Aug 18, 2014, at 10:21 PM, Rajdeep Dua 
mailto:rajdeep@gmail.com>> wrote:

my branch already has the latest changes.
it is not able to merge two rst files hence it failed


On Tue, Aug 19, 2014 at 10:02 AM, Akash Gangil 
mailto:akashg1...@gmail.com>> wrote:
"This change was unable to be automatically merged with the current state of 
the repository. Please rebase your change and upload a new patchset."

Rebase and commit?


On Tue, Aug 19, 2014 at 12:10 AM, Rajdeep Dua 
mailto:rajdeep@gmail.com>> wrote:
One of my CL - updation of  README.rst seems to be failing jenkins

https://review.openstack.org/#/c/114896/1

Any idea how to get this passed?

Thanks
Rajdeep

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Akash

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Policy Enforcement logic

[Tim Hinrichs]

Hi Madhu,

For the alpha release (due soon), we’re focusing on just monitoring policy 
violations—we’ve disabled all the enforcement code in master.  (Though we never 
actually hooked up the enforcement policy to the real world, so all Congress 
has ever done is compute what actions to take to enforce policy.)  There’s a 
ton of interest in enforcement, so we’re planning to add enforcement features 
to the beta release.

Tim


On Aug 21, 2014, at 7:07 AM, Madhu Mohan 
mailto:mmo...@mvista.com>> wrote:

Hi,

I am quite new to the Congress and Openstack as well and this question may seem 
very trivial and basic.

I am trying to figure out the policy enforcement logic,

Can some body help me understand how exactly, a policy enforcement action is 
taken.

>From the example policy there is an action defined as:

action("disconnect_network")
nova:network-(vm, network) :- disconnect_network(vm, network)

I assume that this statement when applied would translate to deletion of entry 
in the database.

But, how does this affect the actual setup (i.e) How is this database update 
translated to actual disconnection of the VM from the network.
How does nova know that it has to disconnect the VM from the network ?

Thanks and Regards,
Madhu Mohan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Policy Enforcement logic

[Tim Hinrichs]

Hi Jay,

We have a tutorial in review right now.  It should be merged in a couple of 
days.  Thanks for the suggestion!

Tim


On Aug 21, 2014, at 7:54 AM, Jay Lau 
mailto:jay.lau@gmail.com>> wrote:

I know that Congress is still under development, but it is better that it can 
provide some info for "How to use it" just like docker 
https://wiki.openstack.org/wiki/Docker , this might attract more people 
contributing to it.


2014-08-21 22:07 GMT+08:00 Madhu Mohan 
mailto:mmo...@mvista.com>>:
Hi,

I am quite new to the Congress and Openstack as well and this question may seem 
very trivial and basic.

I am trying to figure out the policy enforcement logic,

Can some body help me understand how exactly, a policy enforcement action is 
taken.

>From the example policy there is an action defined as:

action("disconnect_network")
nova:network-(vm, network) :- disconnect_network(vm, network)

I assume that this statement when applied would translate to deletion of entry 
in the database.

But, how does this affect the actual setup (i.e) How is this database update 
translated to actual disconnection of the VM from the network.
How does nova know that it has to disconnect the VM from the network ?

Thanks and Regards,
Madhu Mohan




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Ramp-up strategy

[Tim Hinrichs]

Hi Madhu,

We have an end-user tutorial in review right now.  That should help you get 
started understanding the end-to-end flow a bit better.  Look for it to be 
merged today or tomorrow.

Tim



On Aug 21, 2014, at 2:44 AM, Madhu Mohan 
mailto:mmo...@mvista.com>> wrote:

Hi,

Since a few weeks I am trying to get a hold on Congress code base and 
understand the flow.

Here is a brief summary what I am trying out:

Prepared a dummy client to send the policy strings to congress_server listening 
at the path "/policies". This is now changed to "v1/policies". I am using 
"POST" request to send the policy string to the server.

The call to server somehow seems to get converted to an action with the name 
"create_policies"
Added a new API "create_policies" in the api model policy_model.py which gets 
the policy string in "params".

I am able to call compile.parse() and runtime.initialize() functions from this 
API.
The compilation produces a result in the format below:

Rule(head=[Literal(table=u'error', arguments=[Variable(name=u'vm')], 
negated=False)], body=[Literal(table=u'nova:virtual_machine', 
arguments=[Variable(name=u'vm')],.

I am not really sure about how to go about from here to see the policies 
actually getting applied and monitored.

Any resource or instructions on getting through the code flow will be of great 
help to proceed further.

Thanks in Advance,
Madhu Mohan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Upcoming alpha release

[Tim Hinrichs]

Hi all,

We're aiming for an alpha release of Congress tomorrow (Friday).  If you have a 
spare server and a little time, it’d be great if you could try it out: install 
it, write some policies, run tests, etc.  If you could send some feedback along 
the following lines, that would be helpful.

1. What problems did you run into?  File a bug if you like, or drop me an email.

2. Which operating system did you use, and was the install successful or not?

Here are some docs that we hope are enough to get you started.

- README with install instructions:
https://github.com/stackforge/congress/blob/master/README.rst

- Tutorial in the form of an end-to-end example:
https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst

- Troubleshooting guide:
https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rst

Thanks!
Tim


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Policy Enforcement logic

[Tim Hinrichs]

The tutorial is now merged.

https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst



Tim

On Aug 21, 2014, at 3:02 PM, Jay Lau 
mailto:jay.lau@gmail.com>> wrote:

Hi Tim,

That's great! Does the tutorial is uploaded to Gerrit for review?

Thanks.


2014-08-21 23:56 GMT+08:00 Tim Hinrichs 
mailto:thinri...@vmware.com>>:
Hi Jay,

We have a tutorial in review right now.  It should be merged in a couple of 
days.  Thanks for the suggestion!

Tim


On Aug 21, 2014, at 7:54 AM, Jay Lau 
mailto:jay.lau@gmail.com>> wrote:

I know that Congress is still under development, but it is better that it can 
provide some info for "How to use it" just like docker 
https://wiki.openstack.org/wiki/Docker , this might attract more people 
contributing to it.


2014-08-21 22:07 GMT+08:00 Madhu Mohan 
mailto:mmo...@mvista.com>>:
Hi,

I am quite new to the Congress and Openstack as well and this question may seem 
very trivial and basic.

I am trying to figure out the policy enforcement logic,

Can some body help me understand how exactly, a policy enforcement action is 
taken.

>From the example policy there is an action defined as:

action("disconnect_network")
nova:network-(vm, network) :- disconnect_network(vm, network)

I assume that this statement when applied would translate to deletion of entry 
in the database.

But, how does this affect the actual setup (i.e) How is this database update 
translated to actual disconnection of the VM from the network.
How does nova know that it has to disconnect the VM from the network ?

Thanks and Regards,
Madhu Mohan




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Alpha release of Congress

[Tim Hinrichs]

Hi all,

The alpha release of Congress is now available!  We'd love any and all feedback.

Components and Features
- Support for policy monitoring
- Policy engine implementation
- Message-passing architecture
- Drivers for Nova and Neutron
- Devstack integration
- Documentation
- REST API

README (with install instructions):
https://github.com/stackforge/congress/blob/master/README.rst

Tutorial:
https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst

Troubleshooting:
https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rst

Contributors (IRC, Reviews, Code):
Sergio Cazzolato (sjcazzol)
Peter Balland (pballand)
Mohammad Banikazemi (banix)
Rajdeep Dua (rajdeep)
Conner Ferguson (Radu)
Tim Hinrichs (thinrichs)
Gokul Kandiraju (gokul)
Harrison Kelly (harrisonkelly)
Prabhakar Kudva (kudva)
Susanta Nanda (skn)
Sean Roberts (sarob)
Iben Rodriguez (iben)
Aaron Rosen (arosen)
Derick Winkworth (cloudtoad)
Alex Yip (alexsyip)


Sincerely,
The Congress team


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Alpha release of Congress

[Tim Hinrichs]

Congress isn’t incubated yet, so it won’t be a part of the official release.

There’s a command-line interface in review right now.  The horizon integration 
will be along not long after.

Tim

On Sep 3, 2014, at 7:28 PM, Baohua Yang 
mailto:yangbao...@gmail.com>> wrote:

Congrats!
Will appear in J or K?
And will there's an interface in horizon?
Thanks!


On Thu, Sep 4, 2014 at 5:41 AM, Tim Hinrichs 
mailto:thinri...@vmware.com>> wrote:
Hi all,

The alpha release of Congress is now available!  We'd love any and all feedback.

Components and Features
- Support for policy monitoring
- Policy engine implementation
- Message-passing architecture
- Drivers for Nova and Neutron
- Devstack integration
- Documentation
- REST API

README (with install instructions):
https://github.com/stackforge/congress/blob/master/README.rst<https://urldefense.proofpoint.com/v1/url?u=https://github.com/stackforge/congress/blob/master/README.rst&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=6iCMeALTCrx1m6hWlIjoRMUKoRQoAX1IF1NOgy3X2Mo%3D%0A&s=b457bfe87b47b60df3fc9e8e2e5ebcbaa930693a5eb3d948210792ec428c83bd>

Tutorial:
https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst<https://urldefense.proofpoint.com/v1/url?u=https://github.com/stackforge/congress/blob/master/doc/source/tutorial-tenant-sharing.rst&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=6iCMeALTCrx1m6hWlIjoRMUKoRQoAX1IF1NOgy3X2Mo%3D%0A&s=2780defe2dde27a72faff00ba4120122e9954511d66f9a2b1b802e8d2e00f8c4>

Troubleshooting:
https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rst<https://urldefense.proofpoint.com/v1/url?u=https://github.com/stackforge/congress/blob/master/doc/source/troubleshooting.rst&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=6iCMeALTCrx1m6hWlIjoRMUKoRQoAX1IF1NOgy3X2Mo%3D%0A&s=da150867c461b3e743a3e32560417e454fbcbf42f8a2217ecb8576176daae325>

Contributors (IRC, Reviews, Code):
Sergio Cazzolato (sjcazzol)
Peter Balland (pballand)
Mohammad Banikazemi (banix)
Rajdeep Dua (rajdeep)
Conner Ferguson (Radu)
Tim Hinrichs (thinrichs)
Gokul Kandiraju (gokul)
Harrison Kelly (harrisonkelly)
Prabhakar Kudva (kudva)
Susanta Nanda (skn)
Sean Roberts (sarob)
Iben Rodriguez (iben)
Aaron Rosen (arosen)
Derick Winkworth (cloudtoad)
Alex Yip (alexsyip)


Sincerely,
The Congress team


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Best wishes!
Baohua
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

[Tim Hinrichs]

+1

Tim

On Sep 12, 2014, at 11:47 AM, Sean Roberts  wrote:

> +1
> 
> ~sean
> 
>> On Sep 12, 2014, at 11:28 AM, Peter Balland  wrote:
>> 
>> Hello,
>> 
>> I would like to nominate Aaron Rosen for the congress-core team.
>> 
>> Aaron has been involved in congress for the last few months providing
>> valuable reviews as well as leading the keystone integration and
>> congressclient work.
>> 
>> References:
>> 
>> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress,
>> n,z
>> 
>> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
>> ngressclient,n,z
>> 
>> https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
>> ss,n,z
>> 
>> Please respond with +1's or any concerns.
>> 
>> Thanks,
>> 
>> Peter
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[Congress]

[Tim Hinrichs]

You are right that we don't have those built ins hooked up yet. But there are 
patches in review that do that. If you would like to help let us know!

Tim

P. S. Pardon the brevity. Sent from my mobile. 

> On Sep 17, 2014, at 10:52 PM, "Madhu Mohan"  wrote:
> 
> Hi Congress Team,
> 
> Has anyone tried writing policies with Builtin operators like gt(), lt().
> 
> I see the class CongressBuiltinCategoryMap() to add/update the start map 
> defined in the same file.
> 
> However, these classes are never referred in any part of the code. So 
> clearly, I can assume Congress does not support such operators at this point
> 
> Is there a plan to implement this feature or someone looking into it already ?
> 
> Thanks and Regards,
> Madhu Mohan
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Group-based Policy language

[Tim Hinrichs]

At the Neutron group-based policy proposal meeting today, we discussed whether 
or not the proposal should include a concrete policy language.  We decided to 
send a note to the list to get additional feedback.

The proposed API extension includes the ability to insert/delete policy 
statements.  But we do not say which policy statements are valid.  The benefit 
of leaving the policy language unspecified is that each plugin can support a 
custom policy language, leading to maximum flexibility in terms of writing 
plugins.  The drawback of leaving the policy language unspecified is that 
there's no way for any person or other OS component to know which API calls are 
valid, unless we know which plugin is being used.  Said another way, the 
current proposal says there are API calls like insert-policy-statement and 
delete-policy-statement, but does not say which arguments are valid to give to 
those calls (and the valid arguments can differ from plugin to plugin).

The thought experiment we went through was to imagine writing a super 
stripped-down version of Heat that only builds applications with a DB tier and 
a Web tier, and the template for the app only specifies how many DB servers and 
how many Web servers we want.  We should be able to implement a function that 
takes the number of DB servers and the number of web servers as input and 
executes a sequence of Nova/Neutron API calls that deploys that app.  But 
without a concrete policy language, we can't use the Neutron policy API  b/c we 
don't know what arguments to give the insert-policy-statement call.

In the end, we discussed adding a concrete language to the proposal.  Does 
anyone see a better alternative?

Thanks,
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [policy] Policy-group relationship

[Tim Hinrichs]

Inline.

- Original Message -
| From: "Mohammad Banikazemi" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Cc: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Tuesday, December 17, 2013 11:42:53 AM
| Subject: Re: [openstack-dev] [neutron] [policy] Policy-group relationship
| 
| 
| 
| 
| Stephen Wong  wrote on 12/15/2013 12:00:32 PM:
| 
| > From: Stephen Wong 
| > To: "OpenStack Development Mailing List (not for usage questions)"
| > ,
| > Date: 12/15/2013 12:04 PM
| > Subject: Re: [openstack-dev] [neutron] [policy] Policy-group
| > relationship
| > 
| > Hi Mohammad,
| > 
| > Good writeup, one minor comment at the end below (look for
| > [s3wong]).
| > 
| > On Thu, Dec 12, 2013 at 3:42 PM, Mohammad Banikazemi
| >  wrote:
| > > Continuing the discussion we had earlier today during the Neutron
| > > Group
| > > Policy weekly meeting [0], I would like to initiate a couple of
| > > email
| > > threads and follow up on a couple of important issues we need to
| > > agree on so
| > > we can move forward. In this email thread, I would like to
| > > discuss the
| > > policy-group relationship.
| > > 
| > > I want to summarize the discussion we had during the meeting [1]
| > > and see if
| > > we have reached an agreement:
| > > 
| > > There are two models for expressing the relationship between
| > > Groups and
| > > Policies that were discussed:
| > > 1- Policies are defined for a source Group and a destination
| > > Group
| > > 2- Groups specify the Policies they "provide" and the Policies
| > > they
| > > "consume"
| > > 
| > > As expressed during the IRC meeting, both models have strong
| > > support and we
| > > decided to have a resource model that can be used to express both
| > > models.
| > > The solution we came up with was rather simple:
| > > 
| > > Update the resource model (shown in the attribute tables and the
| > > taxonomy in
| > > the google doc [2]) such that policy can refer to a "list" of
| > > source Groups
| > > and a "list" of destination Groups.
| > > This boils down to having two attributes namely, src_groups and
| > > destination_groups (both list of uuid-str type) replacing the
| > > current
| > > attributes src_group and dest_group, respectively.
| > > 
| > > This change simply allows the support for both models. For
| > > supporting model
| > > 1, specify a single source Group and a single destination Group.
| > > For model
| > > 2, specify the producers of a Policy in the source Group list and
| > > specify
| > > the consumers of the Policy in the destination Group list.
| > 
| > [s3wong] this is interesting. Let's say we have two groups A & B,
| > and
| > A wants to send traffic to B, so in this case, B is providing a
| > policy
| > which A will consume. In your proposal above, I would have to put A
| > in
| > destination group list and B in source group list although the
| > traffic
| > direction is the reverse. Would that cause a bit of a confusion?
| > 
| 
| Yeah, that's a good point. Producers are the destination of traffic
| flows.
| So should we have them under destination group? Even that is a bit
| confusing.
| May be we need different names for the two groups.
| 
| -Mohammad
| 

If we're not sure what the 2 groups represent (source and destination), perhaps 
that means we ought to have any number of groups and not assign them names.  A 
policy would then be applied to any number of groups, and it would be up to the 
policy itself to dictate source/destination semantics if that is what the 
groups represent.

For example, if we had a DENY action, it might take several arguments, one of 
which is a source and one of which is a destination.  Then by applying groups 
properly to that DENY action, we could dictate which group is playing the role 
of SOURCE and which group is playing the role of DESTINATION.

Tim

| > Thanks,
| > - Stephen
| > 
| > 
| > > 
| > > If there is agreement, I will update the taxonomy and the
| > > attribute tables
| > > in the doc.
| > > 
| > > Best,
| > > 
| > > Mohammad
| > > 
| > > 
| > > [0] https://wiki.openstack.org/wiki/Meetings/Neutron_Group_Policy
| > > [1]
| > > http://eavesdrop.openstack.org/meetings/networking_policy/2013/
| > networking_policy.2013-12-12-16.01.log.html
| > > [2]
| > > https://docs.google.com/document/d/
| > 1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit#heading=h.x1h06xqhlo1n
| > > (Note the new additions are at the end of the document.)
| > > 
| > > 
| > > ___
| > > OpenStack-dev mailing list
| > > OpenStack-dev@lists.openstack.org
| > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
| > > 
| > 
| > ___
| > OpenStack-dev mailing list
| > OpenStack-dev@lists.openstack.org
| > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
| > 
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.

Re: [openstack-dev] [neutron][policy] Policy-Rules discussions based on Dec.12 network policy meeting

[Tim Hinrichs]

- Original Message -
| From: "Prasad Vellanki" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Monday, December 16, 2013 2:11:37 PM
| Subject: Re: [openstack-dev] [neutron][policy] Policy-Rules discussions based 
on Dec.12 network policy meeting
| 
| 
| 
| Hi
| Please see inline 
| 
| 
| 
| On Sun, Dec 15, 2013 at 8:49 AM, Stephen Wong < s3w...@midokura.com >
| wrote:
| 
| 
| Hi,
| 
| During Thursday's group-policy meeting[1], there are several
| policy-rules related issues which we agreed should be posted on the
| mailing list to gather community comments / consensus. They are:
| 
| (1) Conflict resolution between policy-rules
| --- a priority field was added to the policy-rules attributes
| list[2]. Is this enough to resolve conflict across policy-rules (or
| even across policies)? Please state cases where a cross policy-rules
| conflict can occur.
| --- conflict resolution was a major discussion point during
| Thursday's meeting - and there was even suggestion on setting
| priority
| on endpoint groups; but I would like to have this email thread
| focused
| on conflict resolution across policy-rules in a single policy first.
| 

There was interest in having a single policy that could include different 
actions so that a single flow might be both redirected and QOSed 
simultaneously.  For me this rules out a total ordering on the policy 
statements.  Here's a proposal that relies on the fact that we're fixing the 
meaning of actions within the language: the language specifies a partial order 
on the *actions*.  For example, DENY takes precedence over ALLOW, so if we both 
ALLOW and DENY, then the conflict resolution dictates DENY wins. But {DENY, 
ALLOW} and QOS and REDIRECT are all unrelated, so there is no problem with a 
policy that both DENYs and QOSes and REDIRECTs.

| (2) Default policy-rule actions
| --- there seems to be consensus from the community that we need to
| establish some basic set of policy-rule actions upon which all
| plugins/drivers would have to support
| --- just to get the discussion going, I am proposing:
| 
| 
| 
| Or should this be a query the plugin for supported actions and thus
| the user knows what functionality the plugin can support. Hence
| there is no default supported list.
| 

I think the important part is that the language defines what the actions mean.  
Whether each plugin supports them all is a different issue.  If the language 
doesn't define the meaning of the actions, there's no way for anyone to use the 
language.  We might be able to write down policies, but we don't know what 
those policies actually mean because 2 plugins might assign very different 
meanings to the same action name.

| 
| 
| a.) action_type: 'security' action: 'allow' | 'drop'
| b.) action_type: 'qos' action: {'qos_class': {'critical' |
| 'low-priority' | 'high-priority' |
| 
| 'low-immediate' | 'high-immediate' |
| 
| 'expedite-forwarding'}
| (a subset of DSCP values - hopefully in language that can
| be well understood by those performing application deployments)
| c.) action_type:'redirect' action: {UUID, [UUID]...}
| (a list of Neutron objects to redirect to, and the list
| should contain at least one element)
| 
| 
| 
| 
| I am not sure making the UUIDs a list of neutron objects or endpoints
| will work well. It seems that it should more higher level such as
| list of services that form a chain. Lets say one forms a chain of
| services, firewall, IPS, LB. It would be tough to expect user to
| derive the neutron ports create a chain of them. It could be a VM
| UUID.
| 
| 

Perhaps we could use our usual group mechanism here and say that the redirect 
action operates on 3 groups: source, destination, and the group to which we 
want to redirect.

| 
| Please discuss. In the document, there is also 'rate-limit' and
| 'policing' for 'qos' type, but those can be optional instead of
| required for now
| 

It would be nice if we had some rationale for deciding which actions to include 
and which to leave out.  Maybe if we found a 
standard/spec/collection-of-use-cases and included exactly the same actions.  
Or if we go with the action-based conflict resolution scheme from (1), we might 
want to think about whether we have at least complementary actions (e.g. ALLOW 
and DENY, WAYPOINT -- route traffic through a group of middleboxes-- and FORBID 
-- prohibit traffic from passing through middleboxes).

| (3) Prasad asked for clarification on 'redirect' action, I propose to
| add the following text to document regarding 'redirect' action:
| 
| "'redirect' action is used to mirror traffic to other destinations
| - destination can be another endpoint group, a service chain, a port,
| or a network. Note that 'redirect' action type can be used with other
| forwarding related action type such as 'security'; therefore, it is
| entirely possible that one can specify {'security':'deny'} and still
| do {'redirect':{'uuid-1', 'uuid-2'...}. Note that the desti

[openstack-dev] [neutron] [policy] Complementary or alternative semantics?

[Tim Hinrichs]

Over the holidays I realized there's something about the proposed Neutron 
policy API that I don't understand.  Is the proposed API complementary to the 
core API, or is it intended to be an alternative?  By complementary, I mean 
that a user can create a bunch of networks, subnets, and ports and then 
constrain how those things interoperate by writing policy.  By an alternative, 
I mean that a user must choose either networks/subnets/ports or policy, but 
cannot choose both.  I had always assumed we were talking about a complementary 
API but wanted to double-check.

Thanks,
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] [policy] More conflict resolution

[Tim Hinrichs]

Hi all,

Today on the Neutron-policy IRC, we ran into some issues that we wanted all 
your feedback on (and as I was writing this note several more occurred to me). 
It's the problem of conflict resolution: different policies making different 
decisions about what to do.  We have a conflict resolution scheme for when 
conflicts arise within a single policy, but we have yet to solve the problem of 
conflicts arising because different policies make different decisions.  The 
important thing to know is that each policy is attached to a tenant (and a 
single tenant can have multiple policies).

1) We need conflict resolution for the following scenarios.  Suggestions?

- Two policies for a single tenant have a conflict.

- Two policies for different tenants with different Keystone roles have a 
conflict.  For example, we expect that admin policies should supersede 
non-admin policies (i.e. that if the admin policy makes a decision to either 
allow or drop a packet, then the non-admin's policy is ignored; otherwise, the 
non-admin's policy makes the final decision).  Are there other roles we need to 
think about?

- Two policies for different tenants with the same Keystone roles have a 
conflict.  For example, if there are two admins whose policies conflict, which 
one wins?

2) Do we perform conflict resolution at the time policies are created (i.e. at 
the API level) or do we resolve conflicts more dynamically at run-time?  

To me, API-level conflict resolution doesn't seem like a good option.  Suppose 
a non-admin writes a perfectly reasonable policy.  Then a month later an admin 
writes a policy that conflicts with the non-admin policy.  There's no way to 
reject the non-admin's policy at this point (and we can't reject the admin's 
policy).  It seems the best we can do is inform the non-admin (via email?) that 
her policy has been overruled.  But if we do that, it may be possible for a 
tenant to learn what the admin's policy is--whether that is a security problem 
or not I don't know.

3) What do we do when roles change, e.g. a non-admin user gets promoted to an 
admin user or vice versa?  And how do we find out about role changes if the 
users do not log in after their policies have been created?  That is, 
role-changes seem to affect the overall policy that is enforced at any point in 
time and thus role-changes ought to be factored into policy enforcement.  

Role-changes make me even more dubious that API-level conflict resolution is a 
good choice.


Hopefully that's a reasonable summary.  Others will chime in where not.

Thoughts?
Thanks,
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [policy] More conflict resolution

[Tim Hinrichs]

ar user's policies are all suddenly promoted 
to admin priority, no one will be able to predict how the policy changes will 
affect the network.  Demoting admin rules to non-admin priority has a similar 
effect.  Hmmm... leaving the priorities the same seems problematic, but 
changing the priorities seems problematic too; not sure what to do here.

Tim

| 
| my 2 cents,
| Manuel
| 
| From: Tim Hinrichs [thinri...@vmware.com]
| Sent: Thursday, January 23, 2014 6:57 PM
| To: OpenStack Development Mailing List (not for usage questions)
| Subject: [openstack-dev] [neutron] [policy] More conflict resolution
| 
| Hi all,
| 
| Today on the Neutron-policy IRC, we ran into some issues that we
| wanted all your feedback on (and as I was writing this note several
| more occurred to me). It's the problem of conflict resolution:
| different policies making different decisions about what to do.  We
| have a conflict resolution scheme for when conflicts arise within a
| single policy, but we have yet to solve the problem of conflicts
| arising because different policies make different decisions.  The
| important thing to know is that each policy is attached to a tenant
| (and a single tenant can have multiple policies).
| 
| 1) We need conflict resolution for the following scenarios.
|  Suggestions?
| 
| - Two policies for a single tenant have a conflict.
| 
| - Two policies for different tenants with different Keystone roles
| have a conflict.  For example, we expect that admin policies should
| supersede non-admin policies (i.e. that if the admin policy makes a
| decision to either allow or drop a packet, then the non-admin's
| policy is ignored; otherwise, the non-admin's policy makes the final
| decision).  Are there other roles we need to think about?
| 
| - Two policies for different tenants with the same Keystone roles
| have a conflict.  For example, if there are two admins whose
| policies conflict, which one wins?
| 
| 2) Do we perform conflict resolution at the time policies are created
| (i.e. at the API level) or do we resolve conflicts more dynamically
| at run-time?
| 
| To me, API-level conflict resolution doesn't seem like a good option.
|  Suppose a non-admin writes a perfectly reasonable policy.  Then a
| month later an admin writes a policy that conflicts with the
| non-admin policy.  There's no way to reject the non-admin's policy
| at this point (and we can't reject the admin's policy).  It seems
| the best we can do is inform the non-admin (via email?) that her
| policy has been overruled.  But if we do that, it may be possible
| for a tenant to learn what the admin's policy is--whether that is a
| security problem or not I don't know.
| 
| 3) What do we do when roles change, e.g. a non-admin user gets
| promoted to an admin user or vice versa?  And how do we find out
| about role changes if the users do not log in after their policies
| have been created?  That is, role-changes seem to affect the overall
| policy that is enforced at any point in time and thus role-changes
| ought to be factored into policy enforcement.
| 
| Role-changes make me even more dubious that API-level conflict
| resolution is a good choice.
| 
| 
| Hopefully that's a reasonable summary.  Others will chime in where
| not.
| 
| Thoughts?
| Thanks,
| Tim
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=GCQOsyh5WyZr4Lx2t%2Bs5Kt6S9fWkBV9dfeL%2B1xUUBag%3D%0A&s=c06f67fbd826026f4abc79fda10240d25bd756e237f4c121a43e06de8cdfc5e1
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=GCQOsyh5WyZr4Lx2t%2Bs5Kt6S9fWkBV9dfeL%2B1xUUBag%3D%0A&s=c06f67fbd826026f4abc79fda10240d25bd756e237f4c121a43e06de8cdfc5e1
| 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] IRC meeting moved

[Tim Hinrichs]

Hi all,

Last week we moved the Congress IRC meeting time to every other Tuesday at 
17:00 UTC in #openstack-meeting-3 (that's an hour earlier than it was 
previously).  But we neglected to mail out the new time, and it doesn't look 
like anyone remembered the time change.  So I'll hang around at both our old 
and new time slots this week.

Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] status and feedback

[Tim Hinrichs]

Hi all,

I wanted to send out a quick status update on Congress and start a discussion 
about short-term goals.

1) Logistics

IRC Meeting time
Tuesday 1700 UTC in openstack-meeting-3
Every other week starting Feb 25
(Note this is a new meeting time/location.)

2) We have two design docs, which we would like feedback on.

Toplevel design doc:
https://docs.google.com/a/vmware.com/document/d/1f2xokl9Tc47aV67KEua0PBRz4jsdSDLXth7dYe-jz6Q/edit

Data integration design doc:
https://docs.google.com/document/d/1K9RkQuBSPN7Z2TmKfok7mw3E24otEGo8Pnsemxd5544/edit

3) Short term goals.

I think it's useful to get an end-to-end system up and running to make it 
easier to communicate what we're driving at.  To that end, I'm suggesting we 
take the following policy and build up enough of Congress to make it work.

"Every network connected to a VM must either be public or owned by someone in 
the same group as the VM owner"

This example is compelling because it combines information from Neutron, Nova, 
and Keystone (or another group-management data-source such as ActiveDirectory). 
 To that end, I suggest focusing on the following tasks in the short-term.

- Data integration framework, including read-only support for 
Neutron/Nova/Keystone
- Exposing policy engine via API
- Policy engine error handling
- Simple scaling tests
- Basic user docs
- Execution framework: it would be nice if we could actually execute some of 
the actions we tell Congress about, but this is lowest priority on the list, 
for me.

Thoughts?
Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

[Tim Hinrichs]

Hi Jay,

The Congress project aims to handle something similar to your use cases.  I 
just sent a note to the ML with a Congress status update with the tag 
[Congress].  It includes links to our design docs.  Let me know if you have 
trouble finding it or want to follow up.

Tim

- Original Message -
| From: "Sylvain Bauza" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Tuesday, February 25, 2014 3:58:07 PM
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage
| compute/storage resource
| 
| 
| 
| Hi Jay,
| 
| 
| Currently, the Nova scheduler only acts upon user request (either
| live migration or boot an instance). IMHO, that's something Gantt
| should scope later on (or at least there could be some space within
| the Scheduler) so that Scheduler would be responsible for managing
| resources on a dynamic way.
| 
| 
| I'm thinking of the Pets vs. Cattles analogy, and I definitely think
| that Compute resources could be treated like Pets, provided the
| Scheduler does a move.
| 
| 
| -Sylvain
| 
| 
| 
| 2014-02-26 0:40 GMT+01:00 Jay Lau < jay.lau@gmail.com > :
| 
| 
| 
| 
| Greetings,
| 
| 
| Here I want to bring up an old topic here and want to get some input
| from you experts.
| 
| 
| Currently in nova and cinder, we only have some initial placement
| polices to help customer deploy VM instance or create volume storage
| to a specified host, but after the VM or the volume was created,
| there was no policy to monitor the hypervisors or the storage
| servers to take some actions in the following case:
| 
| 
| 1) Load Balance Policy: If the load of one server is too heavy, then
| probably we need to migrate some VMs from high load servers to some
| idle servers automatically to make sure the system resource usage
| can be balanced.
| 
| 2) HA Policy: If one server get down for some hardware failure or
| whatever reasons, there is no policy to make sure the VMs can be
| evacuated or live migrated (Make sure migrate the VM before server
| goes down) to other available servers to make sure customer
| applications will not be affect too much.
| 
| 3) Energy Saving Policy: If a single host load is lower than
| configured threshold, then low down the frequency of the CPU to save
| energy; otherwise, increase the CPU frequency. If the average load
| is lower than configured threshold, then shutdown some hypervisors
| to save energy; otherwise, power on some hypervisors to load
| balance. Before power off a hypervisor host, the energy policy need
| to live migrate all VMs on the hypervisor to other available
| hypervisors; After Power on a hypervisor host, the Load Balance
| Policy will help live migrate some VMs to the new powered
| hypervisor.
| 
| 4) Customized Policy: Customer can also define some customized
| policies based on their specified requirement.
| 
| 5) Some run-time policies for block storage or even network.
| 
| 
| 
| I borrow the idea from VMWare DRS (Thanks VMWare DRS), and there
| indeed many customers want such features.
| 
| 
| 
| I have filed a bp here [1] long ago, but after some discussion with
| Russell, we think that this should not belong to nova but other
| projects. Till now, I did not find a good place where we can put
| this in, can any of you show some comments?
| 
| 
| 
| [1]
| https://blueprints.launchpad.net/nova/+spec/resource-optimization-service
| 
| --
| 
| 
| Thanks,
| 
| Jay
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
| 
| 
| 
| ___
| OpenStack-dev mailing list
| OpenStack-dev@lists.openstack.org
| 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=XDB3hT4WE2iDrNVK0sQ8qKooX2r1T4E%2BVHek3GREhnE%3D%0A&s=e2346cd017c9d8108c12a101892492e2ac75953e4a5ea5c17394c775cf086d7f
| 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

[Tim Hinrichs]

Hi Jay and Sylvain,

The solver-scheduler sounds like a good fit to me as well.  It clearly 
provisions resources in accordance with policy.  Does it monitor those 
resources and adjust them if the system falls out of compliance with the policy?

I mentioned Congress for two reasons. (i) It does monitoring.  (ii) There was 
mention of compute, networking, and storage, and I couldn't tell if the idea 
was for policy that spans OS components or not.  Congress was designed for 
policies spanning OS components.

Tim

- Original Message -
| From: "Jay Lau" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Tuesday, February 25, 2014 10:13:14 PM
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage
| compute/storage resource
| 
| 
| 
| 
| 
| Thanks Sylvain and Tim for the great sharing.
| 
| @Tim, I also go through with Congress and have the same feeling with
| Sylvai, it is likely that Congress is doing something simliar with
| Gantt providing a holistic way for deploying. What I want to do is
| to provide some functions which is very similar with VMWare DRS that
| can do some adaptive scheduling automatically.
| 
| @Sylvain, can you please show more detail for what "Pets vs. Cattles
| analogy" means?
| 
| 
| 
| 
| 2014-02-26 9:11 GMT+08:00 Sylvain Bauza < sylvain.ba...@gmail.com > :
| 
| 
| 
| Hi Tim,
| 
| 
| As per I'm reading your design document, it sounds more likely
| related to something like Solver Scheduler subteam is trying to
| focus on, ie. intelligent agnostic resources placement on an
| holistic way [1]
| IIRC, Jay is more likely talking about adaptive scheduling decisions
| based on feedback with potential counter-measures that can be done
| for decreasing load and preserving QoS of nodes.
| 
| 
| That said, maybe I'm wrong ?
| 
| 
| [1] https://blueprints.launchpad.net/nova/+spec/solver-scheduler
| 
| 
| 
| 2014-02-26 1:09 GMT+01:00 Tim Hinrichs < thinri...@vmware.com > :
| 
| 
| 
| 
| Hi Jay,
| 
| The Congress project aims to handle something similar to your use
| cases. I just sent a note to the ML with a Congress status update
| with the tag [Congress]. It includes links to our design docs. Let
| me know if you have trouble finding it or want to follow up.
| 
| Tim
| 
| 
| 
| - Original Message -
| | From: "Sylvain Bauza" < sylvain.ba...@gmail.com >
| | To: "OpenStack Development Mailing List (not for usage questions)"
| | < openstack-dev@lists.openstack.org >
| | Sent: Tuesday, February 25, 2014 3:58:07 PM
| | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal
| | for OpenStack run time policy to manage
| | compute/storage resource
| | 
| | 
| | 
| | Hi Jay,
| | 
| | 
| | Currently, the Nova scheduler only acts upon user request (either
| | live migration or boot an instance). IMHO, that's something Gantt
| | should scope later on (or at least there could be some space within
| | the Scheduler) so that Scheduler would be responsible for managing
| | resources on a dynamic way.
| | 
| | 
| | I'm thinking of the Pets vs. Cattles analogy, and I definitely
| | think
| | that Compute resources could be treated like Pets, provided the
| | Scheduler does a move.
| | 
| | 
| | -Sylvain
| | 
| | 
| | 
| | 2014-02-26 0:40 GMT+01:00 Jay Lau < jay.lau@gmail.com > :
| | 
| | 
| | 
| | 
| | Greetings,
| | 
| | 
| | Here I want to bring up an old topic here and want to get some
| | input
| | from you experts.
| | 
| | 
| | Currently in nova and cinder, we only have some initial placement
| | polices to help customer deploy VM instance or create volume
| | storage
| | to a specified host, but after the VM or the volume was created,
| | there was no policy to monitor the hypervisors or the storage
| | servers to take some actions in the following case:
| | 
| | 
| | 1) Load Balance Policy: If the load of one server is too heavy,
| | then
| | probably we need to migrate some VMs from high load servers to some
| | idle servers automatically to make sure the system resource usage
| | can be balanced.
| | 
| | 2) HA Policy: If one server get down for some hardware failure or
| | whatever reasons, there is no policy to make sure the VMs can be
| | evacuated or live migrated (Make sure migrate the VM before server
| | goes down) to other available servers to make sure customer
| | applications will not be affect too much.
| | 
| | 3) Energy Saving Policy: If a single host load is lower than
| | configured threshold, then low down the frequency of the CPU to
| | save
| | energy; otherwise, increase the CPU frequency. If the average load
| | is lower than configured threshold, then shutdown some hypervisors
| | to save energy; otherwise, power on some hypervisors to load
| | balance. Before power off a hypervisor host, the energy policy need
| | to live migrate all VMs on the hypervisor to

Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

[Tim Hinrichs]

Hi Jay,

I think the Solver Scheduler is a better fit for your needs than Congress 
because you know what kinds of constraints and enforcement you want.  I'm not 
sure this topic deserves an entire design session--maybe just talking a bit at 
the summit would suffice (I *think* I'll be attending).

Tim

- Original Message -
| From: "Jay Lau" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Wednesday, February 26, 2014 6:30:54 PM
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage
| compute/storage resource
| 
| 
| 
| 
| 
| 
| Hi Tim,
| 
| I'm not sure if we can put resource monitor and adjust to
| solver-scheduler (Gantt), but I have proposed this to Gantt design
| [1], you can refer to [1] and search "jay-lau-513".
| 
| IMHO, Congress does monitoring and also take actions, but the actions
| seems mainly for adjusting single VM network or storage. It did not
| consider migrating VM according to hypervisor load.
| 
| Not sure if this topic deserved to be a design session for the coming
| summit, but I will try to propose.
| 
| 
| 
| 
| [1] https://etherpad.openstack.org/p/icehouse-external-scheduler
| 
| 
| 
| Thanks,
| 
| 
| Jay
| 
| 
| 
| 2014-02-27 1:48 GMT+08:00 Tim Hinrichs < thinri...@vmware.com > :
| 
| 
| Hi Jay and Sylvain,
| 
| The solver-scheduler sounds like a good fit to me as well. It clearly
| provisions resources in accordance with policy. Does it monitor
| those resources and adjust them if the system falls out of
| compliance with the policy?
| 
| I mentioned Congress for two reasons. (i) It does monitoring. (ii)
| There was mention of compute, networking, and storage, and I
| couldn't tell if the idea was for policy that spans OS components or
| not. Congress was designed for policies spanning OS components.
| 
| 
| Tim
| 
| - Original Message -
| 
| | From: "Jay Lau" < jay.lau@gmail.com >
| | To: "OpenStack Development Mailing List (not for usage questions)"
| | < openstack-dev@lists.openstack.org >
| 
| 
| | Sent: Tuesday, February 25, 2014 10:13:14 PM
| | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal
| | for OpenStack run time policy to manage
| | compute/storage resource
| | 
| | 
| | 
| | 
| | 
| | Thanks Sylvain and Tim for the great sharing.
| | 
| | @Tim, I also go through with Congress and have the same feeling
| | with
| | Sylvai, it is likely that Congress is doing something simliar with
| | Gantt providing a holistic way for deploying. What I want to do is
| | to provide some functions which is very similar with VMWare DRS
| | that
| | can do some adaptive scheduling automatically.
| | 
| | @Sylvain, can you please show more detail for what "Pets vs.
| | Cattles
| | analogy" means?
| | 
| | 
| | 
| | 
| | 2014-02-26 9:11 GMT+08:00 Sylvain Bauza < sylvain.ba...@gmail.com >
| | :
| | 
| | 
| | 
| | Hi Tim,
| | 
| | 
| | As per I'm reading your design document, it sounds more likely
| | related to something like Solver Scheduler subteam is trying to
| | focus on, ie. intelligent agnostic resources placement on an
| | holistic way [1]
| | IIRC, Jay is more likely talking about adaptive scheduling
| | decisions
| | based on feedback with potential counter-measures that can be done
| | for decreasing load and preserving QoS of nodes.
| | 
| | 
| | That said, maybe I'm wrong ?
| | 
| | 
| | [1] https://blueprints.launchpad.net/nova/+spec/solver-scheduler
| | 
| | 
| | 
| | 2014-02-26 1:09 GMT+01:00 Tim Hinrichs < thinri...@vmware.com > :
| | 
| | 
| | 
| | 
| | Hi Jay,
| | 
| | The Congress project aims to handle something similar to your use
| | cases. I just sent a note to the ML with a Congress status update
| | with the tag [Congress]. It includes links to our design docs. Let
| | me know if you have trouble finding it or want to follow up.
| | 
| | Tim
| | 
| | 
| | 
| | - Original Message -
| | | From: "Sylvain Bauza" < sylvain.ba...@gmail.com >
| | | To: "OpenStack Development Mailing List (not for usage
| | | questions)"
| | | < openstack-dev@lists.openstack.org >
| | | Sent: Tuesday, February 25, 2014 3:58:07 PM
| | | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A
| | | proposal
| | | for OpenStack run time policy to manage
| | | compute/storage resource
| | | 
| | | 
| | | 
| | | Hi Jay,
| | | 
| | | 
| | | Currently, the Nova scheduler only acts upon user request (either
| | | live migration or boot an instance). IMHO, that's something Gantt
| | | should scope later on (or at least there could be some space
| | | within
| | | the Scheduler) so that Scheduler would be responsible for
| | | managing
| | | resources on a dynamic way.
| | | 
| | | 
| | | I'm thinking of the Pets vs. Cattles analogy, and I definitely
| | | think
| | | that Compute resources could b

Re: [openstack-dev] [Keystone] New Policy Administration Service

[Tim Hinrichs]

Based on the thread entitled [all][policy][keystone] Better Policy Model and 
Representing Capabilites from October 20, I wrote some code to pull a 
policy.json file into Congress and figure out what roles are necessary to give 
access to a specific API call.

So if bundling this kind of functionality into Congress is a reasonable way 
forward, it seems doable technically.  We’re happy to help in any case, so let 
us know!

Tim


-- Forwarded message --
From: Ioram Schechtman Sette mailto:i...@cin.ufpe.br>>
Date: Tue, Nov 18, 2014 at 5:52 AM
Subject: [openstack-dev] [Keystone] New Policy Administration Service
To: openstack-dev@lists.openstack.org


Hi all,

In Paris, on the last day, we listed the new features that we would like to see 
in the next release of Keystone.
The top 3 were chosen as high priority.

Further down the list was a policy administration service that will collect 
policies from all the Openstack services and allow the Keystone administrator 
to ask the question "what role do I need to assign to a user to give access to 
these services?" and will allow users to ask the question "what can I access 
with my roles?".

We have now started to design and build this service. An important design 
decision is "should this service be integrated with Keystone or be a separated 
standalone Openstack service?" What does the Keystone group think?

If policy administration should be a separate service, what is the process to 
register blueprints, apis and code reviews?

Regards,
Ioram and David

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Reactive enforcement specs

[Tim Hinrichs]

Hi all,

Recently there’s been quite a bit of interest in adding reactive enforcement to 
Congress: the ability to write policies that tell Congress to execute actions 
to correct policy violations.  We’re planning to add this feature in the next 
release.  I wrote a few specs that split this work into several bite-sized 
pieces (one was accidentally merged prematurely—it’s still up for discussion).

Let’s discuss these over Gerrit (the usual spec process).  We’re trying to 
finalize these specs by the middle of next week (a little behind the usual 
OpenStack schedule).  For those of you who haven’t left comments via Gerrit, 
you need to ...

1) log in to Gerrit using your Launchpad ID,
2) leave comments on specific lines in individual files by double-clicking the 
line you’d like to comment on,
3) click the Review button on the initial page
4) click the Publish Comments button.

Add triggers to policy engine (a programmatic interface useful for implementing 
reactive enforcement)
https://review.openstack.org/#/c/130010/

Add modal operators to policy language (how we might express reactive 
enforcement policies within Datalog)
https://review.openstack.org/#/c/134376/

Action-execution interface (how we might modify data-source drivers so they can 
execute actions)
https://review.openstack.org/#/c/134417/

Explicit reactive enforcement (pulling all the pieces together)
https://review.openstack.org/#/c/134418/


There are a number of additional specs generated since the summit.  Feel free 
to chime in on those too.
https://review.openstack.org/#/q/status:open+project:stackforge/congress-specs,n,z

Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [policy] [congress] Protocol for Congress --> Enactor

[Tim Hinrichs]

Thanks for the summary Greg—that was great!  Here’s my take.

It would be great if all the services Congress interacts with implemented the 
same protocol and used the same policy/data language.  It is worth our time to 
figure out what that protocol and language should be.

But we should not forget that there will always be legacy services that people 
are unwilling or unable to change that don’t speak that protocol/language. And 
right now no services speak that protocol/language (since it doesn’t exist).  
So it’s useful today and in the future to have an adapter/wrapper framework 
that enables Congress to  interact with other protocols and languages.

That means we need to push on 2 fronts: (i) designing the ideal 
protocol/language and (ii) designing the adapter framework.  I’ve been focused 
on (ii) since it’s absolutely necessary today, but if anyone would like to 
spearhead (i) I’d be happy to help.

Tim


On Nov 1, 2014, at 11:13 AM, Gregory Lebovitz 
mailto:gregory.i...@gmail.com>> wrote:


Summary from IRC chat 10/14/2014 on weekly meeting [1] [2]

Topic:  Declarative Language for Congress —> Enactor/Enforcer

Question: Shall we specify a declarative language for communicating policy 
configured in Congress to enactors / enforcement systems

Hypothesis (derived at conclusion of discussion):
 - Specify declarative protocol and framework for describing policy with 
extensible attributes/value fields described in a base ontology, with 
additional affinity ontologies, is what is needed earlier than later, to be 
able to achieve it as an end-state, before too many Enactors dive into one-offs.
 - We could achieve that specification once we know the right structure


Discussion:

  *   Given the following framework:
 *   Elements:
*   Congress - The policy description point, a place where:
   *   (a) policy inputs are collected
   *   (b) collected policy inputs are integrated
   *   (c) policy is defined
   *   (d) declares policy intent to enforcing / enacting systems
   *   (e) observes state of environment, noting policy violations
*   Feeders - provides policy inputs to Congress
*   Enactors / Enforcers - receives policy declarations from Congress 
and enacts / enforces the policy according to its capabilities
   *   E.g. Nova for VM placement, Neutron for interface connectivity, 
FWaaS for access control, etc.

What will the protocol be for the Congress —> Enactors / Enforcers?


thinrichs:  we’ve we've been assuming that Congress will leverage whatever the 
Enactors (policy engines) and Feeders (and more generally datacenter services) 
that exist are using. For basic datacenter services, we had planned on teaching 
Congress what their API is and what it does. So there's no new protocol 
there—we'd just use HTTP or whatever the service expects. For Enactors, there 
are 2 pieces: (1) what policy does Congress push and (2) what protocol does it 
use to do that? We don't know the answer to (1) yet.  (2) is less important, I 
think. For (2) we could use opflex, for example, or create a new one. (1) is 
hard because the Enactors likely have different languages that they understand. 
I’m not aware of anyone thinking about (2). I’m not thinking about (2) b/c I 
don't know the answer to (1). The *really* hard thing to understand IMO is how 
these Enactors should cooperate (in terms of the information they exchange and 
the functionality they provide).  The bits they use to wrap the messages they 
send while cooperating is a lower-level question.


jasonsb & glebo: feel the need to clarify (2)


glebo: if we come out strongly with a framework spec that identifies a protocol 
for (2), and make it clear that Congress participants, including several data 
center Feeders and Enactors, are in consensus, then the other Feeders & 
Enactors will line up, in order to be useful in the modern deployments. Either 
that, or they will remain isolated from the new environment, or their customers 
will have to create custom connectors to the new environment. It seems that we 
have 2 options. (a) Congress learns any language spoken by Feeders and 
Enactors, or (b) specifies a single protocol for Congress —> Enactors policy 
declarations, including a highly adaptable public registry(ies) for defining 
the meaning of content blobs in those messages. For (a) Congress would get VERY 
bloated with an abstraction layer, modules, semantics and state for each 
different language it needed to speak. And there would be 10s of these 
languages. For (b), there would be one way to structure messages that were 
constructed of blobs in (e.g.) some sort of Type/Length/Value (TLV) method, 
where the Types and Values were specified in some Internet registry.


jasonsb: Could we attack this from the opposite direction? E.g. if Congress 
wanted to provide an operational dashboard to show if things are in compliance, 
it would be better served by receiving the

[openstack-dev] [Congress] Re: Placement and Scheduling via Policy

[Tim Hinrichs]

[Adding openstack-dev to this thread.  For those of you just joining… We 
started kicking around ideas for how we might integrate a special-purpose VM 
placement engine into Congress.]

Kudva: responses inline.


On Dec 16, 2014, at 6:25 AM, Prabhakar Kudva 
mailto:ku...@us.ibm.com>> wrote:

Hi,

I am very interested in this.

So, it looks like there are two parts to this:
1. Policy analysis when there are a significant mix of logical and builtin 
predicates (i.e.,
runtime should identify a solution space when there are arithmetic operators). 
This will
require linear programming/ILP type solvers.  There might be a need to have a 
function
in runtime.py that specifically deals with this (Tim?)


I think it’s right that we expect there to be a mix of builtins and standard 
predicates.  But what we’re considering here is having the linear solver be 
treated as if it were a domain-specific policy engine.  So that solver wouldn’t 
be embedded into the runtime.py necessarily.  Rather, we’d delegate part of the 
policy to that domain-specific policy engine.

2. Enforcement. That is with a large number of constraints in place for 
placement and
scheduling, how does the policy engine communicate and enforce the placement
constraints to nova scheduler.


I would imagine that we could delegate either enforcement or monitoring or 
both.  Eventually we want enforcement here, but monitoring could be useful too.

And yes you’re asking the right questions.  I was trying to break the problem 
down into pieces in my bullet (1) below.  But I think there is significant 
overlap in the questions we need to answer whether we’re delegating monitoring 
or enforcement.

Both of these require some form of mathematical analysis.

Would be happy and interested to discuss more on these lines.


Maybe take a look at how I tried to breakdown the problem into separate 
questions in bullet (1) below and see if that makes sense.

Tim

Prabhakar






From:        Tim Hinrichs mailto:thinri...@vmware.com>>
To:"ruby.krishnasw...@orange.com<mailto:ruby.krishnasw...@orange.com>" 
mailto:ruby.krishnasw...@orange.com>>
Cc:"Ramki Krishnan (r...@brocade.com<mailto:r...@brocade.com>)" 
mailto:r...@brocade.com>>, Gokul B 
Kandiraju/Watson/IBM@IBMUS, Prabhakar Kudva/Watson/IBM@IBMUS
Date:12/15/2014 12:09 PM
Subject:Re: Placement and Scheduling via Policy




[Adding Prabhakar and Gokul, in case they are interested.]

1) Ruby, thinking about the solver as taking 1 matrix of [vm, server] and 
returning another matrix helps me understand what we’re talking about—thanks.  
I think you’re right that once we move from placement to optimization problems 
in general we’ll need to figure out how to deal with actions.  But if it’s a 
placement-specific policy engine, then we can build VM-migration into it.

It seems to me that the only part left is figuring out how to take an arbitrary 
policy, carve off the placement-relevant portion, and create the inputs the 
solver needs to generate that new matrix.  Some thoughts...

- My gut tells me that the placement-solver should basically say “I enforce 
policies having to do with the schema nova:location.”  This way the Congress 
policy engine knows to give it policies relevant to nova:location (placement).  
If we do that, I believe we can carve off the right sub theory.

- That leaves taking a Datalog policy where we know nova:location is important 
and converting it to the input language required by a linear solver.  We need 
to remember that the Datalog rules may reference tables from other services 
like Neutron, Ceilometer, etc.  I think the key will be figuring out what class 
of policies we can actually do that for reliably.  Cool—a concrete question.


2) We can definitely wait until January on this.  I’ll be out of touch starting 
Friday too; it seems we all get back early January, which seems like the right 
time to resume our discussions.  We have some concrete questions to answer, 
which was what I was hoping to accomplish before we all went on holiday.

Happy Holidays!
Tim


On Dec 15, 2014, at 5:53 AM, 
mailto:ruby.krishnasw...@orange.com>> 
mailto:ruby.krishnasw...@orange.com>> wrote:

Hi Tim

“Questions:
1) Is there any more data the solver needs?  Seems like it needs something 
about CPU-load for each VM.
2) Which solver should we be using?  What does the linear program that we feed 
it look like?  How do we translate the results of the linear solver into a 
collection of ‘migrate_VM’ API calls?”



  Question (2) seems to me the first to address, in particular:
 “how to prepare the input (variables, constraints, goal) and invoke the 
solver”
=>  We need rules that represent constraints to give the solver (e.g. a 
technical constraint that a VM should not be assigned to more than one server 
or that more than maximum resource (cpu / mem …) of a server canno

Re: [openstack-dev] [Congress] Re: Placement and Scheduling via Policy

[Tim Hinrichs]

Hi all,

Responses inline.

On Dec 16, 2014, at 10:57 PM, 
mailto:ruby.krishnasw...@orange.com>> 
mailto:ruby.krishnasw...@orange.com>> wrote:

Hi Tim & All

@Tim: I did not reply to openstack-dev. Do you think we could have an openstack 
list specific for “congress” to which anybody may subscribe?

Sending to openstack-dev is the right thing, as long as we put [Congress] in 
the subject.  Everyone I know sets up filters on openstack-dev so they only get 
the mail they care about.  I think you’re the only one in the group who isn’t 
subscribed to that list.



1) Enforcement:
   By this we mean “how will the actions computed by the policy engine 
be executed by the concerned OpenStack functional module”.


  In this case, it is better to first work this out for a “simpler” case, 
e.g. your running example concerning the network/groups.
Note: some actions concern only some data base (e.g. insert the 
user within some group).



2)  From Prabhakar’s mail

“Enforcement. That is with a large number of constraints in place for placement 
and
scheduling, how does the policy engine communicate and enforce the placement
constraints to nova scheduler. “

Nova scheduler (current): It assigns VMs to servers based on the 
policy set by the administrator (through filters and host aggregates).

  The administrator also configures a scheduling heuristic (implemented as a 
driver), for example “round-robin” driver.
 Then the computed assignment 
is sent back to the requestor (API server) that interacts with nova-compute to 
provision the VM.
 The current nova-scheduler has 
another function: It updates the allocation status of each compute node on the 
DB (through another indirection called nova-conductor)

So it is correct to re-interpret your statement as follows:

-   What is the entity with which the policy engine interacts for either 
proactive or reactive placement management?

-   How will the output from the policy engine (for example the placement 
matrix) be communicated back?

oProactive: this gives the mapping of VM to host

oReactive: this gives the new mapping of running VMs to hosts

-   How starting from the placement matrix, the correct migration plan will 
be executed? (for reactive case)



3) Currently openstack does not have “automated management of reactive 
placement”:  Hence if the policy engine is used for reactive placement, then 
there is a need for another “orchestrator” that can interpret the new proposed 
placement configuration (mapping of VM to servers) and execute the 
reconfiguration workflow.


4) So with a policy-based “placement engine” that is integrated with 
external solvers, then this engine will replace nova-scheduler?

Could we converge on this?



The notes from Yathiraj say that there is already a policy-based Nova scheduler 
we can use.  I suggest we look into that.  It could potentially simplify our 
problem to the point where we need only figure out how to convert a fragment of 
the Congress policy language into their policy language.  But those of you who 
are experts in placement will know better.

   
https://github.com/stackforge/nova-solver-scheduler<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dscheduler&d=AAMGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7s&m=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0M&s=mdMcHh7nMTJv8PmY0i8NpQXP9_gpUpI3gxEec6zyt7A&e=>

Tim


Regards
Ruby

De : Tim Hinrichs [mailto:thinri...@vmware.com]
Envoyé : mardi 16 décembre 2014 19:25
À : Prabhakar Kudva
Cc : KRISHNASWAMY Ruby IMT/OLPS; Ramki Krishnan 
(r...@brocade.com<mailto:r...@brocade.com>); Gokul B Kandiraju; openstack-dev
Objet : [Congress] Re: Placement and Scheduling via Policy

[Adding openstack-dev to this thread.  For those of you just joining… We 
started kicking around ideas for how we might integrate a special-purpose VM 
placement engine into Congress.]

Kudva: responses inline.


On Dec 16, 2014, at 6:25 AM, Prabhakar Kudva 
mailto:ku...@us.ibm.com>> wrote:


Hi,

I am very interested in this.

So, it looks like there are two parts to this:
1. Policy analysis when there are a significant mix of logical and builtin 
predicates (i.e.,
runtime should identify a solution space when there are arithmetic operators). 
This will
require linear programming/ILP type solvers.  There might be a need to have a 
function
in runtime.py that specifically deals with this (Tim?)

I think it’s right that we expect there to be a mix of builtins and standard 
predicates.  But what we’re considering here is having the linear solver be 
treated as if it were a domain-specific policy engine.  So that solver wouldn’t 
be embedded into the runtime.py necessarily.  Rather, we’d d

Re: [openstack-dev] [Congress] Re: Placement and Scheduling via Policy

[Tim Hinrichs]

Hi Yathi,

Thanks for the reminder about the nova solver scheduler.  It’s definitely a 
cool idea to look at integrating the two systems!

Ramki is definitely involved in this discussion.  We thought placement was a 
good first example of a broad class of problems that a linear solver could help 
address, esp. in the NFV context.  I like the idea of integrating Congress and 
the Nova solver scheduler and then generalizing what we learned to handle other 
kinds of optimization problems.  So that’s what I’m thinking long term.

Tim





On Dec 16, 2014, at 11:28 AM, Yathiraj Udupi (yudupi) 
mailto:yud...@cisco.com>> wrote:

To add to what I mentioned below… We from the Solver Scheduler team are a small 
team here at Cisco, trying to drive this project and slowly adding more complex 
use cases for scheduling and policy–driven placements.We would really love 
to have some real contributions from everyone in the community and build this 
the right way.
If it may interest – some interesting scheduler use cases are here based on one 
of our community meetings in IRC - 
https://etherpad.openstack.org/p/SchedulerUseCases  This could apply to 
Congress driving some of this too.

I am leading the effort for the  Solver Scheduler project ( 
https://github.com/stackforge/nova-solver-scheduler<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dscheduler&d=AAMGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7s&m=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0M&s=mdMcHh7nMTJv8PmY0i8NpQXP9_gpUpI3gxEec6zyt7A&e=>
 ) , and if any of you are willing to contribute code, API, benchmarks, and 
also work on integration, my team and I can help you guide through this.   We 
would be following the same processes under Stackforge at the moment.

Thanks,
Yathi.





On 12/16/14, 11:14 AM, "Yathiraj Udupi (yudupi)" 
mailto:yud...@cisco.com>> wrote:

Tim,

I read the conversation thread below and this got me interested as it relates 
to our discussion we had in the Policy Summit (mid cycle meet up) held in Palo 
Alto a few months ago.

This relates to our project – Nova Solver Scheduler, which I had talked about 
at the Policy summit.   Please see this - 
https://github.com/stackforge/nova-solver-scheduler<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stackforge_nova-2Dsolver-2Dscheduler&d=AAMGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7s&m=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0M&s=mdMcHh7nMTJv8PmY0i8NpQXP9_gpUpI3gxEec6zyt7A&e=>

We already have a working constraints-based solver framework/engine that 
handles Nova placement, and we are currently active in Stackforge, and aim to 
get this integrated into the Gantt project 
(https://blueprints.launchpad.net/nova/+spec/solver-scheduler), based on our 
discussions in the Nova scheduler sub group.

When I saw discussions around using Linear programming (LP) solvers, PULP, etc, 
 I thought of pitching in here to say, we already have demonstrated integrating 
a LP based solver for Nova compute placements.   Please see: 
https://www.youtube.com/watch?v=7QzDbhkk-BI#t=942<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3D7QzDbhkk-2DBI-23t-3D942&d=AAMGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=B6BWd4kFfgOzAREgThxkmTZKy7dDXE2-eBAmL0PBK7s&m=gSzCqpS6tRMB8r5xNbeWoNcpobYiFYvOFpo3QBmvm0M&s=EH2bJNaqNXJoAXzDbMtWkwQaJDUj-IC9QDgie_xdA8E&e=>
 for a demo of this (from our talk at the Atlanta Openstack summit).
 Based on this email thread,  I believe Ramki, one of our early collaborators 
is driving a similar solution in the NFV ETSI research group.  Glad to know our 
Solver scheduler project is getting interest now.

As part of Congress integration,  at the policy summit, I had suggested, we can 
try to translate a Congress policy into our Solver Scheduler’s constraints,  
and use this to enforce Nova placement policies.
We can already demonstrate policy-driven nova placements using our pluggable 
constraints model.  So it should be easy to integrate with Congress.

The Nova solver scheduler team would be glad to help with any efforts wrt to 
trying out a Congress integration for Nova placements.

Thanks,
Yathi.



On 12/16/14, 10:24 AM, "Tim Hinrichs" 
mailto:thinri...@vmware.com>> wrote:

[Adding openstack-dev to this thread.  For those of you just joining… We 
started kicking around ideas for how we might integrate a special-purpose VM 
placement engine into Congress.]

Kudva: responses inline.


On Dec 16, 2014, at 6:25 AM, Prabhakar Kudva 
mailto:ku...@us.ibm.com>> wrote:

Hi,

I am very interested in this.

So, it looks like there are two parts to this:
1. Policy analysis when there are a significant mix of logical and builtin 
predicates (i.e.,
runtime should identify a solutio

Re: [openstack-dev] [Congress] simulate examples

[Tim Hinrichs]

Here's a description.  We need to get this added to the docs.


Below is a full description of how you might utilize the Action-centric version 
of simulate.  The idea is that if you describe the effects that an 
action/API-call will have on the basic tables of nova/neutron/etc. (below 
called an Action Description policy) then you can ask Congress to simulate the 
execution of that action and answer a query in the resulting state.  The only 
downside to the action-centric application of simulate is writing the Action 
Policy for all of the actions you care about.

The other way to utilize simulate is to give it the changes in 
nova/neutron/etc. directly that you’d like to make.  That is, instead of an 
action, you’ll tell simulate what rows should be inserted and which ones should 
be deleted.  An insertion is denoted with a plus (+) and deletion is denoted 
with a minus (-).

For example, to compute all the errors after

  1.  inserting a row into the nova:servers table with ID uuid1, 2TB of disk, 
and 10GB of memory (this isn’t the actual schema BTW) and
  2.  deleting the row from neutron:security_groups with the ID “uuid2” and 
name “alice_default_group” (again not the real schema),

you’d write something like the following.

openstack congress policy simulate classification 'error(x)’ 
‘nova:servers+(“uuid1”, “2TB”, “10 GB”) neutron:security_groups-(“uuid2”, 
“alice_default_group”)' action

But I’d suggest reading the following to see some of the options.

=
1. CREATE ACTION DESCRIPTION POLICY
=

Suppose the table 'p' is a collection of key-value pairs:  p(key, value).

Suppose we have a single action 'set(key, newvalue)’ that changes the existing 
value of 'key' to 'newvalue' or sets the value of 'key' to 'newvalue' if 'key' 
was not already assigned.  We can describe the effects of ‘set’ using the 
following 3 Datalog rules.

p+(x,y) :- set(x,y)
p-(x,oldy) :- set(x,y), p(x,oldy)
action("set")

The first thing we do is add each of these 3 rules to the policy named 'action'.

$ openstack congress policy rule create action 'p+(x,y) :- set(x,y)'
$ openstack congress policy rule create action 'p-(x,oldy) :- set(x,y), 
p(x,oldy)'
$ openstack congress policy rule create action 'action("set")'


=
2. ADD SOME KEY/VALUE PAIRS FOR TESTING
=

Here’s we’ll populate the ‘classification’ policy with a few key/value pairs.

$ openstack congress policy rule create classification 'p(101, 0)'
$ openstack congress policy rule create classification 'p(202, "abc")'
$ openstack congress policy rule create classification 'p(302, 9)'


==
3. DEFINE POLICY
==

There's an error if a key's value is 9.

$ openstack congress policy rule create classification 'error(x) :- p(x, 9)'


===
4. RUN SIMULATION QUERIES
===

Each of the following is an example of a simulation query you might want to run.

a) Simulate changing the value of key 101 to 5 and query the contents of p.

$ openstack congress policy simulate classification 'p(x,y)' 'set(101, 5)' 
action
p(101, 5)
p(202, "abc")
p(302, 9)


b) Simulate changing the value of key 101 to 5 and query the error table

$ openstack congress policy simulate classification 'error(x)' 'set(101, 5)' 
action
error(302)


c) Simulate changing the value of key 101 to 9 and query the error table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9)' 
action
error(302)
error(101)


d) Simulate changing the value of key 101 to 9 and query the *change* in the 
error table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9)' 
action --delta
error+(101)


e) Simulate changing 101:9, 202:9, 302:1 and query the *change* in the error 
table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9) 
set(202, 9) set(302, 1)' action --delta
error+(202)
error+(101)
error-(302)


f) Simulate changing 101:9, 202:9, 302:1, and finally 101:15 (in that order).  
Then query the *change* in the error table.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9) 
set(202, 9) set(302, 1) set(101, 15)' action --delta
error+(202)
error-(302)


g) Simulate changing 101:9 and query the *change* in the error table, while 
asking for a debug trace of the computation.

$ openstack congress policy simulate classification 'error(x)' 'set(101, 9)' 
action --delta --trace

error+(101)
RT: ** Simulate: Querying error(x)
Clas  : Call: error(x)
Clas  : | Call: p(x, 9)
Clas  : | Exit: p(302, 9)
Clas  : Exit: error(302)
Clas  : Redo: error(302)
Clas  : | Redo: p(302, 9)
Clas  : | Fail: p(x, 9)
Clas  : Fail: error(x)
Clas  : Found answer [error(302)]
RT: Original result of error(x) is [error(302)]
RT: ** Simulate: Applying sequence [set(101, 9)]
Action: Call: action(x)
...

Tim




___

Re: [openstack-dev] [NFV] - follow up on scheduling discussion

[Tim Hinrichs]

Hi all,

I see that many of the use cases require information from different OS 
components, e.g. networking, compute, and storage.  One thing to think about is 
where those constraints are written/stored and how the data the constraints 
depend on is pulled together.  The Congress project might be helpful here, and 
I’m happy to help explore options.  Let me know if you’re interested.

https://wiki.openstack.org/wiki/Congress

Tim  



On Jun 4, 2014, at 11:25 AM, ramki Krishnan  wrote:

> All,
> 
> Thanks for the interest in the NFV scheduling topic. Please find a proposal 
> on "Smart Scheduler (Solver Scheduler) enhancements for NFV: Use Cases, 
> Constraints etc.". 
> https://urldefense.proofpoint.com/v1/url?u=https://docs.google.com/document/d/1k60BQXOMkZS0SIxpFOppGgYp416uXcJVkAFep3Oeju8/edit%23heading%3Dh.wlbclagujw8c&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=vTulCeloS8Hc59%2FeAOd32Ri4eqbNqVE%2FeMgNRzGZnz4%3D%0A&s=836991d6daab66b519de3b670db8af001144ddb20e636665b395597aa118538f
> 
> Based on this proposal, we are planning to enhance the existing 
> solver-scheduler blueprint 
> https://urldefense.proofpoint.com/v1/url?u=https://blueprints.launchpad.net/nova/%2Bspec/solver-scheduler&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=vTulCeloS8Hc59%2FeAOd32Ri4eqbNqVE%2FeMgNRzGZnz4%3D%0A&s=d1883d82f8d09081b35986987b5f2f9f1d7731f16b2a5251fdbf26c1b26b294d.
> 
> Would love to hear your comments and thoughts. Would be glad to arrange a 
> conference call if needed.
> 
> Thanks,
> Ramki
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=vTulCeloS8Hc59%2FeAOd32Ri4eqbNqVE%2FeMgNRzGZnz4%3D%0A&s=7df001977efa968a09f3fae30b16ae35f4278a7bc82fcb3cdbbc9639cc505503


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

[Tim Hinrichs]

Hi Gokul, 

Thanks for working out how all these policy initiatives relate to each other. 
I'll be spending some time diving into the ones I hadn't heard about. 

I made some additional comments about Congress below. 

Tim 

- Original Message -

From: "Jay Lau"  
To: "OpenStack Development Mailing List (not for usage questions)" 
 
Sent: Wednesday, March 5, 2014 7:31:55 AM 
Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage compute/storage resource 

Hi Gokul, 




2014-03-05 3:30 GMT+08:00 Gokul Kandiraju < gokul4o...@gmail.com > : 





Dear All, 

We are working on a framework where we want to monitor the system and take 
certain actions when specific events or situations occur. Here are two examples 
of ‘different’ situations: 

Example 1: A VM’s-Owner and N/W’s-owner are different ==> this could mean a 
violation ==> we need to take some action 

Example 2: A simple policy such as (VM-migrate of all VMs on possible node 
failure) OR (a more complex Energy Policy that may involve optimization). 

Both these examples need monitoring and actions to be taken when certain events 
happen (or through polling). However, the first one falls into the Compliance 
domain with Boolean conditions getting evaluated while the second one may 
require a more richer set of expression allowing for sequences or algorithms. 





So far, based on this discussion, it seems that these are *separate* 
initiatives in the community. I am understanding the Congress project to be in 
the domain of Boolean conditions (used for Compliance, etc.) where as the 
Run-time-policies (Jay's proposal) where policies can be expressed as rules, 
algorithms with higher-level goals. Is this understanding correct? 

Also, looking at all the mails, this is what I am reading: 

1. Congress -- Focused on Compliance [ is that correct? ] (Boolean constraints 
and logic) 



[Tim] Your characterization of boolean constraints for Congress is probably a 
good one. Congress won't be solving optimization/numeric problems any time soon 
if ever. However, I could imagine that down the road we could tell Congress 
here's the policy (optimization or Boolean) that we want to enforce, and it 
would carve off say the Load-balancing part of the policy and send it to the 
Runtime-Policies component; or it would carve off the placement policy and send 
it to the SolverScheduler. Not saying I know how to do this today, but that's 
always been part of the goal for Congress: to have a central point for admins 
to control the global policy being enforced throughout the datacenter/cloud. 

The other delta here is that the Congress policy language is general-purpose, 
so there's not a list of policy types that it will handle (Load Balancing, 
Placement, Energy). That generality comes with a price: that Congress must rely 
on other enforcement points, such as the ones below, to handle complicated 
policy enforcement problems. 







2. Runtime-Policies --  -- Focused on Runtime policies for Load 
Balancing, Availability, Energy, etc. (sequences of actions, rules, algorithms) 



[Jay] Yes, exactly. 







3. SolverScheduler -- Focused on Placement [ static or runtime ] and will be 
invoked by the (above) policy engines 




4. Gantt – Focused on (Holistic) Scheduling 



[Jay] For 3 and 4, I was always thinking Gantt is doing something for 
implementing SolverScheduler, not sure if run time policy can be included. 







5. Neat -- seems to be a special case of Runtime-Policies (policies based on 
Load) 



Would this be correct understanding? We need to understand this to contribute 
to the right project. :) 



Thanks! 

-Gokul 


On Fri, Feb 28, 2014 at 5:46 PM, Jay Lau < jay.lau@gmail.com > wrote: 



Hi Yathiraj and Tim, 

Really appreciate your comments here ;-) 

I will prepare some detailed slides or documents before summit and we can have 
a review then. It would be great if OpenStack can provide "DRS" features. 

Thanks, 

Jay 



2014-03-01 6:00 GMT+08:00 Tim Hinrichs < thinri...@vmware.com > : 



Hi Jay, 

I think the Solver Scheduler is a better fit for your needs than Congress 
because you know what kinds of constraints and enforcement you want. I'm not 
sure this topic deserves an entire design session--maybe just talking a bit at 
the summit would suffice (I *think* I'll be attending). 

Tim 

- Original Message - 
| From: "Jay Lau" < jay.lau@gmail.com > 
| To: "OpenStack Development Mailing List (not for usage questions)" < 
openstack-dev@lists.openstack.org > 
| Sent: Wednesday, February 26, 2014 6:30:54 PM 
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for 
OpenStack run time policy to manage 
| compute/storage resource 
| 
| 
| 
| 
| 
| 
| Hi Tim, 
| 
| I'm not sure if we can put resource monitor 

[openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi all, 

We started a discussion on IRC yesterday that I'd like to continue. The main 
question is what kind of policy does a Congress user actually write? I can see 
three options. The first two focus on actions (API calls that make changes to 
the state of the cloud) and the last focuses on just the cloud state. (By 
"state of the cloud" I mean all the information Congress can see about all the 
cloud services it is managing, e.g. all the information we can get through API 
calls to Nova, Neutron, Cinder, Heat, ...). 

1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed by 
other cloud services (for each state of the cloud) 
2) Condition Action: which *actions* Congress should execute (for each state of 
the cloud) 
3) Classification (currently supported in Congress): which *states* violate 
real-world policy. [For those of you who have read docs/white-papers/etc. I'm 
using "Classification" in this note to mean the combination of the current 
"Classification" and "Action Description" policies.] 

The important observation is that each of these policies could contain 
different information from each of the others. 

- Access Control vs Condition Action. The Access Control policy tells *other 
cloud services* which actions they are *allowed* to execute. The Condition 
Action policy tells *Congress* which actions it *must* execute. These policies 
differ because they constrain different sets of cloud services. 

- Access Control vs. Classification. The Access Control policy might permit 
some users to violate the Classification policy in some situations (e.g. to fix 
violation A, we might need to cause violation B before eliminating both). These 
policies differ because a violation in one policy might be be a violation in 
the other. 

- Classification vs. Condition Action. The Classification policy might imply 
which actions *could* eliminate a given violation, but the Condition Action 
policy would dictate which of those actions *should* be executed (e.g. the 
Classification policy might tell us that disconnecting a network and deleting a 
VM would both eliminate a particular violation, but the Condition Action policy 
would tell us which to choose). And the Condition Action policy need not 
eliminate all the violations present in the Classification policy. Again these 
policies differ because a violation in one policy might not be a violation in 
the other. 

I'm proposing that for the first release of Congress we support all 3 of these 
policies. When a user inserts/deletes a policy statement, she chooses which 
policy it belongs to. All would be written in basically the same syntax but 
would be used in 3 different scenarios: 

- Prevention: If a component wants to consult Congress before taking action to 
see if that action is allowed, Congress checks the Access Control policy. 

- Reaction: When Congress learns of a change in the cloud's state, it checks 
the Condition Action policy to see which actions should be executed (if any). 

- Monitoring: If a user wants to simply check if the cloud's state is in 
compliance and monitor compliance over time, she writes and queries the 
Classification policy. 

There are several benefits to this proposal. 
- It allows users to choose any of the policy types, if they only want one of 
them. From our discussions with potential users, most seem to want one of these 
3 policy types (and are uninterested in the others). 
- It makes the introduction to Congress relatively simple. We describe 3 
different uses of policy (Prevention, Reaction, Monitoring) and then explain 
which policy to use in which case. 
- This allows us to focus on implementing a single policy-engine technology (a 
Datalog policy language and evaluation algorithms), which gives us the 
opportunity to make it solid. 

There are drawbacks to this proposal as well. 
- We will have 3 separate policies that are conceptually very similar. As the 
policies grow larger, it will become increasingly difficult to keep the 
policies synchronized. This problem can be mitigated to some extent by having 
all 3 share a library of policy statements that they all apply in different 
ways (and such a library mechanism is already implemented). 
- As cloud services change their behavior, policies may need to be re-written. 
For example, right now Nova does not consult Congress before creating a VM; 
thus, to enforce policy surrounding VMs, the best we could do is write a 
Condition-Action policy that adjusts VM configuration when it learns about new 
VMs being created. If we later make Nova consult with Congress before creating 
a VM, we need to write an Access-control policy that puts the proper controls 
in place. 

These drawbacks were the original motivation for supporting only the 
Classification policy and attempting to derive the Access Control and Condition 
Action policies from it. But given that we can't always derive the proper 
Access Control and Condition Action policies from t

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi Prabhakar, 

Thanks for the feedback. I'd be interested to hear what other policy types you 
have in mind. 

To answer your questions... 

We're planning on extending our policy language in such a way that you can use 
Python functions as conditions ("" in the grammar) in rules. That's on my 
todo-list but didn't mention it yesterday as we were short on time. There will 
be some syntactic restrictions so that we can properly execute those Python 
functions (i.e. we need to always be able to compute the inputs to the 
function). I had thought it was just an implementation detail I hadn't gotten 
around to (all Datalog implementations I've seen have such things), but it 
sounds like it's worth writing up a proposal and sending it around before 
implementing. If that's a pressing concern for you, let me know and I'll bump 
it up the stack (a little). If you'd like, feel free to draft a proposal (or 
remind me to do it once in a while). 

As for actions, I typically think of them as API calls to other OS components 
like Nova. But they could just as easily be Python functions. But I would want 
to avoid an action that changes Congress's internal data structures directly 
(e.g. adding a new policy statement). Such actions have caused trouble in the 
past for policy languages (though for declarative programming languages like 
Prolog they are less problematic). I don't think there's anyway we can stop 
people from creating such actions, but I think we should advocate against them. 

Tim 

- Original Message -

From: "prabhakar Kudva"  
To: "OpenStack Development Mailing List (not for usage questions)" 
 
Sent: Wednesday, March 12, 2014 11:34:04 AM 
Subject: Re: [openstack-dev] [Congress] Policy types 

Hi Tim, All, 

I was in the discussion yesterday (kudva), and would like to start gradually 
contributing to the code base. 

So, this discussion below is based on my limited exploration of Congress 
code, running it. I am trying some small pieces to implement to familiarize. 
Please view it as such. As I start adding code, I am sure, my thoughts will 
be more evolved. 

I agree with the three types you outline. I also agree that these will grow. 
We are already thinking of expanding congress for various other types of 
policies. But those would be a manageable start. 

Regarding the comment below. I was wondering if all conditions, and actions 
could be both: 
1. python functions (for conditions they eval 
2. policy primitives. 

The advantage of 1, is that it is just executed and a True or False returned 
by Python for conditions. For actions, python functions are executed to respond 
to conditions. 
This controls the growth of policies and adding more primitives, and makes it 
flexible (say 
to use alarms, monitors, os clients, nova actions etc). 

The advantage of 2, is the ability to use unification (as in unify.py) and do 
some logic reduction. This gives us the full strength of extensive and mature 
logic reasoning and reduction methods. 

One possibility is that it checks which one the two it is and does the 
appropriate 
evaluation for condition and action. 


>There are drawbacks to this proposal as well. 
>- We will have 3 separate policies that are conceptually very similar. As the 
>policies grow larger, it will become >increasingly difficult to keep the 
>policies synchronized. This problem can be mitigated to some extent by having 
>>all 3 share a library of policy statements that they all apply in different 
>ways (and such a library mechanism is >already implemented). 
>- As cloud services change their behavior, policies may need to be re-written. 
>For example, right now Nova does >not consult Congress before creating a VM; 
>thus, to enforce policy surrounding VMs, the best we could do is >write a 
>Condition-Action policy that adjusts VM configuration when it learns about new 
>VMs being created. If we >later make Nova consult with Congress before 
>creating a VM, we need to write an Access-control policy that puts >the proper 
>controls in place. 

Thanks, 

Prabhakar Kudva 




Date: Wed, 12 Mar 2014 10:05:23 -0700 
From: thinri...@vmware.com 
To: openstack-dev@lists.openstack.org 
Subject: [openstack-dev] [Congress] Policy types 

Hi all, 

We started a discussion on IRC yesterday that I'd like to continue. The main 
question is what kind of policy does a Congress user actually write? I can see 
three options. The first two focus on actions (API calls that make changes to 
the state of the cloud) and the last focuses on just the cloud state. (By 
"state of the cloud" I mean all the information Congress can see about all the 
cloud services it is managing, e.g. all the information we can get through API 
calls to Nova, Neutron, Cinder, Heat, ...). 

1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed by 
other cloud services (for each state of the cloud) 
2) Condition Action: which *actions* Congress should execute (for each state of 
the cloud) 
3) Classifi

Re: [openstack-dev] [Congress][Data Integration]

[Tim Hinrichs]

Hi Rajdeep, 

This is an great problem to work on because it confronts one of the assumptions 
we're making in Congress: that cloud services can be represented as a 
collection of tables in a reasonable way. You're asking good questions here. 

More responses inline. 

Tim 


- Original Message -



From: "Rajdeep Dua"  
To: openstack-dev@lists.openstack.org 
Sent: Wednesday, March 12, 2014 11:54:28 AM 
Subject: [openstack-dev] [Congress][Data Integration] 

Need some guidance on how to convert nested types into flat tuples. 
Also should we reorder the tuple values in a particular sequence? 




Order of tuples doesn't matter. Order of columns (values) within a tuple 
doesn't really matter either, except that all tuples must use the same order 
and the policies we write must know which column is which. 




Thanks 
Rajdeep 

As an example i have shown networks and ports tuples with some nested types 

networks - tuple format 
--- 

keys (for reference) 

{'status','subnets', 
'name','test-network','provider:physical_network','admin_state_up', 
'tenant_id','provider:network_type','router:external', 
'shared',id,'provider:segmentation_id'} 

values 
--- 
('ACTIVE', ['4cef03d0-1d02-40bb-8c99-2f442aac6ab0'], 'test-network', None, 
True, 
'570fe78a1dc54cffa053bd802984ede2', 'gre', False, False, 
'240ff9df-df35-43ae-9df5-27fae87f2492', 4) 




Here we'd want to pull the List out and replace it with an ID. Then create 
another table that shows which subnets belong to the list with that ID. (You 
can think of the ID as a pointer to the list---in the C/C++ sense.) So 
something like... 

network( 'ACTIVE', 'ID1', 'test-network', None, True, 
'570fe78a1dc54cffa053bd802984ede2', 'gre', False, False, 
'240ff9df-df35-43ae-9df5-27fae87f2492', 4) 

element('ID1', '4cef03d0-1d02-40bb-8c99-2f442aac6ab0') 
element('ID1', ) 

The other thing to think about is whether we want 1 table with 10 columns or we 
want 10 tables with 2 columns each. In this example, we would have... 

network('net1') 
network.status('net1', 'ACTIVE' ) 
network.subnets('net1', 'ID1') 
network.name('net1', 'test-network') 
... 

The period is just another character in the tablename. Nothing fancy happening 
here. 

The ports example below would need a similar flattening. To handle 
dictionaries, I would use the dot-notation shown above. 
A single Neutron API call might populate several Congress tables. 

Tim 




ports - tuple format 
 
keys (for reference) 

{'status','binding:host_id', 'name', 'allowed_address_pairs', 
'admin_state_up', 'network_id', 
'tenant_id', 'extra_dhcp_opts': [], 
'binding:vif_type', 'device_owner', 
'binding:capabilities', 'mac_address', 
'fixed_ips' , 'id', 'security_groups', 
'device_id'} 

Values 

('ACTIVE', 'havana', '', [], True, '240ff9df-df35-43ae-9df5-27fae87f2492', 
'570fe78a1dc54cffa053bd802984ede2', [], 'ovs', 'network:router_interface', 
{'port_filter': True}, 'fa:16:3e:ab:90:df', [{'subnet_id': 
'4cef03d0-1d02-40bb-8c99-2f442aac6ab0', 'ip_address': '90.0.0.1'}], 
'0a2ce569-85a8-45ec-abb3-0d4b34ff69ba', [], 
'864e4acf-bf8e-4664-8cf7-ad5daa95681e') 



___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=A86YVKfBX5U3g6F7eNScJYjr6Qwjv4dyDyVxE9Im8g8%3D%0A&s=0345ab3711a58ec1ebcee08649f047826cec593f57e9843df0fec2f8cfb03b42
 






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi Prabhakar,

I'm not sure the functionality is split between 'policy' and 'server' as 
cleanly as you describe.

The 'policy' directory contains the Policy Engine.  At its core, the policy 
engine has a generic Datalog implementation that could feasibly be used by 
other OS components.  (I don't want to think about pulling it out into Oslo 
though.  There are just too many other things going on and no demand yet.)  But 
there are also Congress-specific things in that directory, e.g. the class 
Runtime in policy/runtime.py will be the one that we hook up external API calls 
to.

The 'server' directory contains the code for the API web server that calls into 
the Runtime class.

So if you're digging through code, I'd suggest focusing on the 'policy' 
directory and looking at compile.py (responsible for converting Datalog rules 
written as strings into an internal representation) and runtime.py (responsible 
for everything else).  The docs I mentioned in the IRC should have a decent 
explanation of the functions in Runtime that the API web server will hook into. 
 

Be warned though that unless someone raises some serious objections to the 
proposal that started this thread, we'll be removing some of the more 
complicated functions from Runtime.  The compile.py code won't change (much).  
All of the 3 new theories will be instances of MaterializedViewTheory.  That's 
also the code that must change to add in the Python functions we talked about 
(more specifically see MaterializedViewTheory::propagate_rule(), which calls 
TopDownTheory::top_down_evaluation(), which is what will need modification).

Tim
 



- Original Message -
| From: "prabhakar Kudva" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Wednesday, March 12, 2014 1:38:55 PM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| 
| 
| 
| Hi Tim,
| 
| Thanks for your comments.
| Would be happy to contribute to the propsal and code.
| 
| The existing code already reflects the thoughts below, and got me
| in the line of ideas. Please orrect me if I am wrong as I am
| learning with these discussions:
| 
| One part (reflected by code in "policy" directory is the generic
| "condition-> action engine" which could take logic primitives and
| (in the future) python functions, evaluate the conditions and
| execute the action. This portable core engine be used for any kind of
| policy enforcement
| (as by other OS projects), such as for data center monitoring and
| repair,
| service level enforcement, compliance policies, optimization (energy,
| performance) etc... at any level of the stack. This core engine seems
| possibly
| a combination of logic reasoning/unification and python function
| evaluation, and python code actions.
| 
| Second part (reflected by code in "server") are the applications
| for various purposes. These could be project specific, task specific.
| We could add a diverse set of examples. The example I have worked
| with seems closer to compliance (as in net owner, vm owner check),
| and we will add more.
| 
| Prabhakar
| 
| 
| 
| Date: Wed, 12 Mar 2014 12:33:35 -0700
| From: thinri...@vmware.com
| To: openstack-dev@lists.openstack.org
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| 
| 
| Hi Prabhakar,
| 
| 
| Thanks for the feedback. I'd be interested to hear what other policy
| types you have in mind.
| 
| 
| To answer your questions...
| 
| 
| We're planning on extending our policy language in such a way that
| you can use Python functions as conditions ("" in the grammar)
| in rules. That's on my todo-list but didn't mention it yesterday as
| we were short on time. There will be some syntactic restrictions so
| that we can properly execute those Python functions (i.e. we need to
| always be able to compute the inputs to the function). I had thought
| it was just an implementation detail I hadn't gotten around to (all
| Datalog implementations I've seen have such things), but it sounds
| like it's worth writing up a proposal and sending it around before
| implementing. If that's a pressing concern for you, let me know and
| I'll bump it up the stack (a little). If you'd like, feel free to
| draft a proposal (or remind me to do it once in a while).
| 
| 
| As for actions, I typically think of them as API calls to other OS
| components like Nova. But they could just as easily be Python
| functions. But I would want to avoid an action that changes
| Congress's internal data structures directly (e.g. adding a new
| policy statement). Such actions have caused trouble in the past for
| policy languages (though for declarative programming languages like
| Prolog they are less problematic). I don't think there's anyway we
| can stop people from creating such actions, but I think we should
| advocate against them.
| 
| 
| Tim
| 
| 
| 
| From: "prabhakar Kudva" 
| To: "OpenStack Development Mailing List (not for usage questions)"
| 
| Sent: Wednesday, March 12, 2014 11:34:04 AM
| Subject

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi Prabhakar,

One big piece we're missing in terms of code right now is the Data Integration 
component.  The job of this component is to integrate data sources available in 
the cloud so that tables like nova:virtual_machine, neutron:owner, etc. reflect 
the information stored in Nova, Neutron, etc.  Rajdeep is making progress on 
that (he's got some code up on review that we're iterating on), and Peter had 
worked on integrating AD a while back.

Typically I've seen the Python functions (which I usually call 'builtins') 
integrated into a Datalog system have explicit declarations (e.g. inputs, 
outputs, etc.).  This is good if we need to do deeper analysis of the policy 
(which is one of the reasons to use a policy language) and typically requires 
information about how that builtin works.  

I dug through some old (Lisp) code to see how I've done this in the past.

// (defbuiltin [])
(defbuiltin plus + (integer integer) integer)
(defbuiltin minus - (integer integer) integer)
(defbuiltin times * (integer integer) integer)
(defbuiltin div (lambda (x y) (floor (/ x y))) (integer integer) integer)
(defbuiltin lt numlessp (integer integer) nil)
(defbuiltin lte numleqp (integer integer) nil)
(defbuiltin gte numgeqp (integer integer) nil)
(defbuiltin gt numgreaterp (integer integer) nil)

But maybe you're right in that we could do away with these explicit 
declarations and just assume that everything that (i) is calleable, (ii) not 
managed by the Data Integration component, and (iii) does not appear in the 
head of a rule is a builtin.  My only worry is that I could imagine silent and 
weird problems showing up, e.g. someone forgot to define a table with rules and 
there happens to be a function in Python by that name.  Or someone supplies the 
wrong number of arguments, and we just get an error from Python, which we'd 
have no direct way to communicate to the policy-writer, i.e. there's no 
compile-time argument-length checking.

The other thing I've seen done is to have a single builtin 'evaluate' that lets 
us call an arbitrary Python function, e.g.

p(x, y) :- q(x), evaluate(mypyfunc(x), y)

Then we wouldn't need to declare the functions.  Errors would still be silent.  
But it would be clear whether we were using a Python function or not.

Thoughts?
Tim




- Original Message -
| From: "prabhakar Kudva" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Saturday, March 15, 2014 8:28:27 PM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| 
| 
| Hi Tim,
| 
| Here is a small change I wanted to try in runtime.py
| It may already exist in MaterializedViewTheory, but wasn't clear to
| me.
| Checking to see if this is something that:1. makes sense 2. already
| exists
| 3. worth implementing? in that order.
| 
| Let's take the example from private_public_network.classify
| 
| error(vm) :- nova:virtual_machine(vm), nova:network(vm, network),
| not neutron:public_network(network),
| neutron:owner(network, netowner), nova:owner(vm, vmowner), not
| same_group(netowner, vmowner)
| 
| same_group(user1, user2) :- cms:group(user1, group), cms:group(user2,
| group)
| 
| nova:virtual_machine("vm1")
| nova:virtual_machine("vm2")
| nova:virtual_machine("vm3")
| nova:network("vm1", "net_private")
| nova:network("vm2", "net_public")
| neutron:public_network("net_public")
| nova:owner("vm1", "tim")
| nova:owner("vm2", "pete")
| nova:owner("vm3", "pierre")
| neutron:owner("net_private", "martin")
| 
| 
| In this example, if as in Scenario 1:
| 
| Cloud services at our disposal:
| nova:virtual_machine(vm)
| nova:network(vm, network)
| nova:owner(vm, owner)
| neutron:public_network(network)
| neutron:owner(network, owner)
| cms:group(user, group)
| 
| are all python functions called through some nova/neutron api,
| then, we just execute them to get a true/false value in runtime.py
| They should be first checked to make sure they are python functions
| and
| not condition primitives using 'callable' and os.dir or some such
| combination.
| 
| If not, and they are assertions made in the file, not directly
| related to OS
| state, then in Scenario 2
| 
| nova:owner("vm1", "tim")
| nova:owner("vm2", "pete")
| nova:owner("vm3", "pierre")
| 
| Are assertions made in the file. In a dynamic environment,
| a python function could query an OS client to actually find the
| current owner,
| since some other OS command could have been used to change the owner
| without an entry being made in this file, i.e., without explicitly
| informing Congres.
| This may not occur currently with vms, but may be implemented
| in a future release. Similar other examples are possible
| 
https://ask.openstack.org/en/question/5582/how-to-change-ownership-between-tenants-of-volume/
| https://blueprints.launchpad.net/cinder/+spec/volume-transfer
| 
| 
| So, I was thinking that python_nova_owner("vm1"), is first checked as
| a python
| function which calls the appropriate OS client to check the current
| owner.

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi Prabhakar,

No IRC meeting this week.  Our IRC is every *other* week, and we had it last 
week.

Though there's been enough activity of late that maybe we should consider 
making it weekly.

I'll address the rest later.

Tim

- Original Message -
| From: "prabhakar Kudva" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Monday, March 17, 2014 7:45:53 PM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Tim,
|  
| Definitely would like to learn more about the Data Integration component
| and how best to contribute. Can't find it in the latest git, perhaps we can
| discuss during the meeting tomorrow. Where can I find some code and/or
| document?
|  
| I like the idea of using an 'evauate' wrapper.  And the idea of a builtin
| library of functions.  Having strict rules for builtins and an evaluate
| wrapper
| to do type checking (primitive or builtin) and so on reduces the silent
| errors or weird outcomes.
|  
| I also agree, raw python using methods like 'calleable' (or other ways
| of checking if the function can be called in python) may not always give
| definitive answers. Even the definition of callable says, that a True only
| says that the object 'appears' calleable.
|  
| So silent and weird problems are possible if we try to execute. On the other
| hand,
| if we are looking at module which contains only user defined functions or
| builtine
| , i.e., there  are strict rules to what exactly constitute allowed-functions,
| then there
| is a higher likelihood that it is correct.  But still a likelihood.
|  
| I like the 'evaluate' idea which could do the checking within
| to make sure the function is indeed calleable, and silent and weird side
| effects
| are filtered inside the 'evaluate'. In addition, it can do checking if it is
| a  primitive or some other valid format. Needs more thought on this.
|  
| Let's discuss implementation paths if possible at the meeting. Would like to
| carve out a small implementation goal either in the 'data integration' line
| or
| the discussion above.
|  
| Prabhakar
|  
|  
|  
| 
|  
| > Date: Mon, 17 Mar 2014 08:57:05 -0700
| > From: thinri...@vmware.com
| > To: openstack-dev@lists.openstack.org
| > Subject: Re: [openstack-dev] [Congress] Policy types
| > 
| > Hi Prabhakar,
| > 
| > One big piece we're missing in terms of code right now is the Data
| > Integration component.  The job of this component is to integrate data
| > sources available in the cloud so that tables like nova:virtual_machine,
| > neutron:owner, etc. reflect the information stored in Nova, Neutron, etc.
| > Rajdeep is making progress on that (he's got some code up on review that
| > we're iterating on), and Peter had worked on integrating AD a while back.
| > 
| > Typically I've seen the Python functions (which I usually call 'builtins')
| > integrated into a Datalog system have explicit declarations (e.g. inputs,
| > outputs, etc.).  This is good if we need to do deeper analysis of the
| > policy (which is one of the reasons to use a policy language) and
| > typically requires information about how that builtin works.
| > 
| > I dug through some old (Lisp) code to see how I've done this in the past.
| > 
| > // (defbuiltin of types of returns> [])
| > (defbuiltin plus + (integer integer) integer)
| > (defbuiltin minus - (integer integer) integer)
| > (defbuiltin times * (integer integer) integer)
| > (defbuiltin div (lambda (x y) (floor (/ x y))) (integer integer) integer)
| > (defbuiltin lt numlessp (integer integer) nil)
| > (defbuiltin lte numleqp (integer integer) nil)
| > (defbuiltin gte numgeqp (integer integer) nil)
| > (defbuiltin gt numgreaterp (integer integer) nil)
| > 
| > But maybe you're right in that we could do away with these explicit
| > declarations and just assume that everything that (i) is calleable, (ii)
| > not managed by the Data Integration component, and (iii) does not appear
| > in the head of a rule is a builtin.  My only worry is that I could imagine
| > silent and weird problems showing up, e.g. someone forgot to define a
| > table with rules and there happens to be a function in Python by that
| > name.  Or someone supplies the wrong number of arguments, and we just get
| > an error from Python, which we'd have no direct way to communicate to the
| > policy-writer, i.e. there's no compile-time argument-length checking.
| > 
| > The other thing I've seen done is to have a single builtin 'evaluate' that
| > lets us call an arbitrary Python function, e.g.
| > 
| > p(x, y) :- q(x), evaluate(mypyfunc(x), y)
| > 
| > Then we wouldn't need to declare the functions.  Errors would still be
| > silent.  But it would be clear whether we were using a Python function or
| > not.
| > 
| > Thoughts?
| > Tim
| > 
| > 
| > 
| > 
| > - Original Message -
| > | From: "prabhakar Kudva" 
| > | To: "OpenStack Development Mailing List (not for usage questions)"
| > | 
| > | Sent: Saturday, March 15, 2014 8:28:27 PM
| 

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi Prabhakar,

Found time for a more detailed response.  Comments are inline.

Tim

- Original Message -
| From: "Tim Hinrichs" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Tuesday, March 18, 2014 9:31:34 AM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Prabhakar,
| 
| No IRC meeting this week.  Our IRC is every *other* week, and we had it last
| week.
| 
| Though there's been enough activity of late that maybe we should consider
| making it weekly.
| 
| I'll address the rest later.
| 
| Tim
| 
| - Original Message -
| | From: "prabhakar Kudva" 
| | To: "OpenStack Development Mailing List (not for usage questions)"
| | 
| | Sent: Monday, March 17, 2014 7:45:53 PM
| | Subject: Re: [openstack-dev] [Congress] Policy types
| | 
| | Hi Tim,
| |  
| | Definitely would like to learn more about the Data Integration component
| | and how best to contribute. Can't find it in the latest git, perhaps we can
| | discuss during the meeting tomorrow. Where can I find some code and/or
| | document?

There are no docs yet, and the only code in git is a quick-and-dirty 
ActiveDirectory integration.  But conceptually there are two pieces of the data 
integration story.

1. We're representing the data stored in each cloud service (that is pertinent 
to policy) as a collection of TABLES.  Each table is a collection of rows that 
all have the same number of columns.  Each value in the table is a scalar 
(number or string).   

If you look at the last IRC, Rajdeep has started working on translating Neutron 
data into this table format.  There are a couple of emails on the mailing list 
as well where we had some discussion.  Here's his change set.

https://review.openstack.org/#/c/77835/

2. The policy engine will need to ask for the contents of tables (or updates to 
tables).  Or have the contents/updates pushed to it periodically.  The 
infrastructure for doing this is what Peter (pballand) volunteered to do in IRC 
last week.  Let's wait on this to see what he has to say.



| |  
| | I like the idea of using an 'evauate' wrapper.  And the idea of a builtin
| | library of functions.  Having strict rules for builtins and an evaluate
| | wrapper
| | to do type checking (primitive or builtin) and so on reduces the silent
| | errors or weird outcomes.
| |  
| | I also agree, raw python using methods like 'calleable' (or other ways
| | of checking if the function can be called in python) may not always give
| | definitive answers. Even the definition of callable says, that a True only
| | says that the object 'appears' calleable.
| |  
| | So silent and weird problems are possible if we try to execute. On the
| | other
| | hand,
| | if we are looking at module which contains only user defined functions or
| | builtine
| | , i.e., there  are strict rules to what exactly constitute
| | allowed-functions,
| | then there
| | is a higher likelihood that it is correct.  But still a likelihood.

Agreed: errors will always be possible.  The policy engine will treat them as 
failures (i.e. won't "re-throw" the error), so we just want to help the 
policy-writer as much as we can.

| |  
| | I like the 'evaluate' idea which could do the checking within
| | to make sure the function is indeed calleable, and silent and weird side
| | effects
| | are filtered inside the 'evaluate'. In addition, it can do checking if it
| | is
| | a  primitive or some other valid format. Needs more thought on this.
| |  
| | Let's discuss implementation paths if possible at the meeting. Would like
| | to
| | carve out a small implementation goal either in the 'data integration' line
| | or
| | the discussion above.


Great! 

One option is to coordinate with rajdeep and see if there's an OS component 
that you'd like to try to turn into tables.  His code is an example of what 
you'd be writing.

Or you could put together a proposal for adding Python builtins to the policy 
engine.  One thing to think about is that we'd want to make it easy to add 
additional builtins.  We'd also need to figure out what the syntax would look 
like from the policy-writer's point of view.  Here I've seen two options:

// +, *, and lt are builtins
p(x, y, z) :- q(x), r(z) (3*x)+2=y, lt(y, z)

// builtins are forced to look like any other table.
p(x, y, z) :- q(x), r(z), *(3, x, w), +(w, 2, y), lt(y, z)


The latter is a bit clunky, but it makes it easier to implement the policy 
engine.

Tim

   

| |  
| | Prabhakar
| |  
| |  
| |  
| | 
| |  
| | > Date: Mon, 17 Mar 2014 08:57:05 -0700
| | > From: thinri...@vmware.com
| | > To: openstack-dev@lists.openstack.org
| | > Subject: Re: [openstack-dev] [Congress] Policy types
| | > 
| | > Hi Prabhakar,
| | > 
| | > One big piece we're missing in terms of co

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Inline.

Tim

- Original Message -
| From: "prabhakar Kudva" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Wednesday, March 19, 2014 7:09:24 AM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Tim,
|  
| I am good with starting or own the Python __congress__builtin__ proposal, but
| will need help from the whole team to work through the details.
| Do you all envision, the data integration API (Pete's calls) to be part of
| this builtin?
| What are the exact aspects each member is interested in having in this
| builtin? It
| would be good to start a list and discuss.
| 1. Inherit all python builtins
| 2. Any builtins specific to datalog etc
| 3. builtins to test and manage primitives
|  

I'd say we should start simple.  The data types we support right now are 
integers, floats, and strings.  So let's add some basic arithmetic and string 
manipulation functions.  If we design this right, it will be easy to add more 
later.

Arithmetic: +, *, -, /, <, >, =, others?
Strings: concatenation, substring, indexof, others?

I'd say we definitely don't want class (3)--we don't want the policy to change 
the data upon which policy decisions are made, at least not directly.  
Conceptually, the tables derived from nova/Neutron/etc. represent *their* 
states.  We can change their states but only by executing *their* API calls, 
which if successful, will change their state, which will eventually be 
reflected in Congress tables.  The Condition-Action policy that I described 
below dictates which of their API calls Congress should execute and when; this 
policy *indirectly* will change the tables of Nova/Neutron/etc.  But I wouldn't 
want to change tables via builtins.



| Regarding data integration:
| My concern with periodically pushing the contents is consistency. Having two
| stores (the nova/OS
| database and a Congress specific increases the chances of the data being out
| of sync). For example
| nova:owner(vm1), might be changed in the nova database if the period is too
| large.  What would be
| the benefits of doing this?  The first one: 'ask for contents of the tables'
| reduces the consistency
| issue.  Do you see those API mapping betweeen table entries and OS calls as
| part of the builtin?

Conceptually, data integration handles the problem of grabbing information from 
other components.  Builtins handle the problem that some tables are  
hard/impossible to define within Datalog itself.  

More practically, I would NOT want a builtin that queries another cloud service 
because the policy writer does not (and is not supposed to) know the particular 
algorithms Congress will use to process those rules.  Congress could end up 
calling a builtin 100 times for a *single* computation of policy violations.
We wouldn't want to download all the data about all VM instances 100 times 
whenever someone asks for all violations.  So builtins are things that compute 
quickly.

I don't see a way around caching the contents of other OS components.  A single 
policy might utilize the same data but in different ways in multiple places in 
the policy.  The intent is that we'll always be in sync to the extent that is 
practically feasible.  And even if we were to ask for all the data that we need 
before each and every query someone asks, that data could still change before 
we answer the query, before the person asking the query looked at the answer, 
before the person asking the query made a decision based on the answer, etc.  
So staleness is a fact of life, and we're setting this up to minimize those 
problems by enabling cloud services to send us *updates* to their data whenever 
they get them.  If a cloud service does not send us updates, the best we can do 
is poll them periodically, which will make staleness a bigger problem.  But I 
don't see what else we can do.


|  
| >>2. The policy engine will need to ask for the contents of tables (or
| >>updates to tables).  Or have the >>contents/updates pushed to it
| >>periodically.  The infrastructure for doing this is what Peter (pballand)
| >>>>volunteered to do in IRC last week.  Let's wait on this to see what he
| >>has to say.
|  
| > Date: Tue, 18 Mar 2014 12:56:10 -0700
| > From: thinri...@vmware.com
| > To: openstack-dev@lists.openstack.org
| > CC: rajde...@vmware.com
| > Subject: Re: [openstack-dev] [Congress] Policy types
| > 
| > Hi Prabhakar,
| > 
| > Found time for a more detailed response.  Comments are inline.
| > 
| > Tim
| > 
| > - Original Message -
| > | From: "Tim Hinrichs" 
| > | To: "OpenStack Development Mailing List (not for usage questions)"
| > | 
| > | Sent: Tuesday, March 18, 2014 9:31:34 AM
| > | Subject: Re: [openstack-dev] [Congress] Policy types
| &g

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

I went ahead and made an editing pass over our design doc to reflect the 
changes described below.  (I moved most of the old content into the final 
section focused on future work.)  Overall, I'd say it simplifies the story and 
implementation considerably.  We can always revert if anyone has serious 
objections, but it was a useful exercise to see if everything hangs together 
still (and it seems to).

I've also happened to talk with several different people about Congress since 
the last IRC, and the proposed changes are well-received.  

https://docs.google.com/a/vmware.com/document/d/1f2xokl9Tc47aV67KEua0PBRz4jsdSDLXth7dYe-jz6Q/edit#

Tim



- Original Message -
| From: "Tim Hinrichs" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Wednesday, March 12, 2014 10:05:23 AM
| Subject: [Congress] Policy types
| 
| Hi all,
| 
| We started a discussion on IRC yesterday that I'd like to continue. The main
| question is what kind of policy does a Congress user actually write? I can
| see three options. The first two focus on actions (API calls that make
| changes to the state of the cloud) and the last focuses on just the cloud
| state. (By "state of the cloud" I mean all the information Congress can see
| about all the cloud services it is managing, e.g. all the information we can
| get through API calls to Nova, Neutron, Cinder, Heat, ...).
| 
| 1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed
| by other cloud services (for each state of the cloud)
| 2) Condition Action: which *actions* Congress should execute (for each state
| of the cloud)
| 3) Classification (currently supported in Congress): which *states* violate
| real-world policy. [For those of you who have read docs/white-papers/etc.
| I'm using "Classification" in this note to mean the combination of the
| current "Classification" and "Action Description" policies.]
| 
| The important observation is that each of these policies could contain
| different information from each of the others.
| 
| - Access Control vs Condition Action. The Access Control policy tells *other
| cloud services* which actions they are *allowed* to execute. The Condition
| Action policy tells *Congress* which actions it *must* execute. These
| policies differ because they constrain different sets of cloud services.
| 
| - Access Control vs. Classification. The Access Control policy might permit
| some users to violate the Classification policy in some situations (e.g. to
| fix violation A, we might need to cause violation B before eliminating
| both). These policies differ because a violation in one policy might be be a
| violation in the other.
| 
| - Classification vs. Condition Action. The Classification policy might imply
| which actions *could* eliminate a given violation, but the Condition Action
| policy would dictate which of those actions *should* be executed (e.g. the
| Classification policy might tell us that disconnecting a network and
| deleting a VM would both eliminate a particular violation, but the Condition
| Action policy would tell us which to choose). And the Condition Action
| policy need not eliminate all the violations present in the Classification
| policy. Again these policies differ because a violation in one policy might
| not be a violation in the other.
| 
| I'm proposing that for the first release of Congress we support all 3 of
| these policies. When a user inserts/deletes a policy statement, she chooses
| which policy it belongs to. All would be written in basically the same
| syntax but would be used in 3 different scenarios:
| 
| - Prevention: If a component wants to consult Congress before taking action
| to see if that action is allowed, Congress checks the Access Control policy.
| 
| - Reaction: When Congress learns of a change in the cloud's state, it checks
| the Condition Action policy to see which actions should be executed (if
| any).
| 
| - Monitoring: If a user wants to simply check if the cloud's state is in
| compliance and monitor compliance over time, she writes and queries the
| Classification policy.
| 
| There are several benefits to this proposal.
| - It allows users to choose any of the policy types, if they only want one of
| them. From our discussions with potential users, most seem to want one of
| these 3 policy types (and are uninterested in the others).
| - It makes the introduction to Congress relatively simple. We describe 3
| different uses of policy (Prevention, Reaction, Monitoring) and then explain
| which policy to use in which case.
| - This allows us to focus on implementing a single policy-engine technology
| (a Datalog policy language and evaluation algorithms), which gives us the
| opportunity to make it solid.
| 
| There are drawbacks to this proposal as well.
| - We will have 3 separate policies that are conceptually very similar. As the
| 

Re: [openstack-dev] [Congress] Policy types

[Tim Hinrichs]

Hi Prabhakar,

I looked into this a while back.  pydatalog is a cool project, and I'd like to 
find time to study some of its algorithms and architecture a bit more.  

The reason we rolled our own version of Datalog is that we knew we'd want the 
freedom to try out non-standard Datalog algorithms, and there are algorithms we 
will likely need that aren't usually included (query rewriting, skolemization, 
conversion to DNF, etc.).  I suppose we could have modified pydatalog, but 
given that it is basically an extension to Python, it seemed there would be 
significant overhead in figuring out the code, making sure to keep our changes 
compatible with the old, etc.  Rolling our own gives us the freedom to build 
exactly what we need with minimal distractions, which is important since we 
won't really know what we need until we start getting feedback from users.  

But like I said, there are probably some good ideas in there, so if the way 
they deal with builtins is useful to us, great!  I'd just keep in mind that the 
benefit of using Datalog instead of Python is mainly that it *limits* what 
policy authors can say (without limiting it too much).  

Tim

- Original Message -
| From: "prabhakar Kudva" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Wednesday, March 26, 2014 9:04:22 AM
| Subject: Re: [openstack-dev] [Congress] Policy types
| 
| Hi Tim, All,
|  
| As I was preparing the background for the proposal for the
| __Congress_builtins__,
| came across this link which uses Datalog with Python, and provides
| data integration facilities through SQLAlchemy.  The tool seems to have been
| used (from the web page claim) in production:
| Just want to run it by everyone to see if this is connected or useful in our
| builtin
| capability, and anything we can glean from it:
|  
| 
https://urldefense.proofpoint.com/v1/url?u=https://pypi.python.org/pypi/pyDatalog&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=WvQzzQA4hxW34TMc13h1FgKQQSxHsa1RgBpgfI5lO2U%3D%0A&s=6028d309c9c709635828e7dd33c3f614db1b6989544e05ad6a21da2e646144fe
| 
|  
https://urldefense.proofpoint.com/v1/url?u=https://sites.google.com/site/pydatalog/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=WvQzzQA4hxW34TMc13h1FgKQQSxHsa1RgBpgfI5lO2U%3D%0A&s=fb152fdc60b5925bb47145dd54973b8184a2b47fb51c28cc86afd6ecbe02b2c5
|  
| 
https://urldefense.proofpoint.com/v1/url?u=https://sites.google.com/site/pydatalog/home/datalog-applications&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=WvQzzQA4hxW34TMc13h1FgKQQSxHsa1RgBpgfI5lO2U%3D%0A&s=a4cb6eed4bf9d7d0a0bfa811c3c8d72f64b973924697f0e5e9e00681403a7664
|  
| Thanks,
|  
| Prabhakar
|  
| > Date: Tue, 18 Mar 2014 12:56:10 -0700
| > From: thinri...@vmware.com
| > To: openstack-dev@lists.openstack.org
| > CC: rajde...@vmware.com
| > Subject: Re: [openstack-dev] [Congress] Policy types
| > 
| > Hi Prabhakar,
| > 
| > Found time for a more detailed response.  Comments are inline.
| > 
| > Tim
| > 
| > - Original Message -
| > | From: "Tim Hinrichs" 
| > | To: "OpenStack Development Mailing List (not for usage questions)"
| > | 
| > | Sent: Tuesday, March 18, 2014 9:31:34 AM
| > | Subject: Re: [openstack-dev] [Congress] Policy types
| > | 
| > | Hi Prabhakar,
| > | 
| > | No IRC meeting this week.  Our IRC is every *other* week, and we had it
| > | last
| > | week.
| > | 
| > | Though there's been enough activity of late that maybe we should consider
| > | making it weekly.
| > | 
| > | I'll address the rest later.
| > | 
| > | Tim
| > | 
| > | - Original Message -
| > | | From: "prabhakar Kudva" 
| > | | To: "OpenStack Development Mailing List (not for usage questions)"
| > | | 
| > | | Sent: Monday, March 17, 2014 7:45:53 PM
| > | | Subject: Re: [openstack-dev] [Congress] Policy types
| > | | 
| > | | Hi Tim,
| > | |  
| > | | Definitely would like to learn more about the Data Integration
| > | | component
| > | | and how best to contribute. Can't find it in the latest git, perhaps we
| > | | can
| > | | discuss during the meeting tomorrow. Where can I find some code and/or
| > | | document?
| > 
| > There are no docs yet, and the only code in git is a quick-and-dirty
| > ActiveDirectory integration.  But conceptually there are two pieces of the
| > data integration story.
| > 
| > 1. We're representing the data stored in each cloud service (that is
| > pertinent to policy) as a collection of TABLES.  Each table is a
| > collection of rows that all have the same number of columns.  Each value
| > in the t

Re: [openstack-dev] [Congress] PTL Non-Candidacy + Nomination

[Tim Hinrichs]

I’m writing to announce my candidacy for the Congress Program Technical Lead 
(PTL).  (Congress is not yet incubated, but we follow the standard PTL election 
procedure.)

I’m an active contributor in terms of code commits, code reviews, spec writing, 
spec reviews, and IRC.  I’ve been serving as Chief Architect and have worked on 
Congress since the beginning.  In the upcoming cycle I propose focusing on the 
following issues.

1)  Deployments: I’d like to deploy Congress into real production clouds to get 
feedback from real operators and help us refine our development priorities.

2)  Community: I’d like us to understand what the developers in the OpenStack 
community want and need from Congress and how we can work together to improve 
the overall policy capabilities of OpenStack as a whole.  I’d like us to make a 
concerted effort to welcome new Congress developers and help them become 
productive.

3)  End users: I’d like us to make it easier for people to write policy; I’d 
like us to make it easier for users to add new cloud services over which they 
can write policy; and I’d like to give people more insight into what Congress 
can do with policy.

4) Capabilities: I’d like us to make our first foray into policy 
enforcement—having Congress actively make changes in the cloud to help it obey 
policy.

While there are many more things I’d like us to work on, at this point in the 
project making progress in each of these areas seems most crucial.

Thanks for your consideration!
Tim



On Sep 29, 2014, at 2:35 PM, Peter Balland  wrote:

> Yes, I should have mentioned, Congress is currently a StackForge project,
> but we follow many of the standard timelines and processes.  Sorry if I
> caused any confusion.
> 
> - Peter
> 
> On 9/29/14, 2:28 PM, "Anita Kuno"  wrote:
> 
>> On 09/29/2014 05:20 PM, Peter Balland wrote:
>>> It has been an honor to serve as the interim PTL for the Congress
>>> project during Juno.  Due to other commitments I have during the Kilo
>>> timeframe, I feel I would not be able to commit the time needed by the
>>> growing project.  In my place, I would like to nominate Tim Hinrichs for
>>> PTL of Congress.
>>> 
>>> Tim has been involved in the Congress project since its inception, and
>>> has been serving as its chief architect.  He is very active in policy
>>> research, code, and community, and has a great strategic vision for the
>>> project.
>>> 
>>> - Peter
>>> 
>>> 
>>> 
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>> Note:
>> 
>> Congress is not one of the programs or projects that is currently having
>> elections administered by the election process governed by the tc
>> charter, just to ensure readers are not confused. They can choose a
>> leader as suits the members involved in Congress.
>> 
>> Thanks,
>> Anita.
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][policy][keystone] Better Policy Model and Representing Capabilites

[Tim Hinrichs]

First, some truth in advertising: I work on Congress (policy as a service), so 
I’ve mostly given thought to this problem in that context.

1) I agree with the discussion below about creating a token that encodes all 
the permitted actions for the user.  The cons seem substantial.  

(i) The token will get stale, requiring us to either revoke it when 
policy/roles change or to live with incorrect access control enforcement until 
the token expires.  

(ii) The token could become large, complex, or both.  Suppose the policy is 
granular enough to put restrictions on the arguments a user is permitted to 
provide to an action.  The token might end up encoding a significant portion of 
the policy itself.  Checking if the token permits a given action could be 
similar computationally to checking the original policy.json file.  


2) I like the idea of an out-of-band service that caches a copy of all the 
policy.jsons and allows users to interrogate/edit them.  I’ve definitely talked 
to operators who would like this kind of thing.  This would be a low-risk, 
low-friction solution to the problem because nothing about OpenStack today 
would need to change.  We’d just add an extra service and tell people to use 
it—sort of a new UI for the policy.json files.  And we could add interesting 
functionality, e.g. hypothetical queries such as “if I were to add role X, what 
changes would that make in my rights?"

Perhaps some more context about why users want to know all of the actions they 
are permitted to execute might help.

Tim


 
On Oct 14, 2014, at 1:56 AM, David Chadwick  wrote:

> 
> 
> On 14/10/2014 01:25, Nathan Kinder wrote:
>> 
>> 
>> On 10/13/2014 01:17 PM, Morgan Fainberg wrote:
>>> Description of the problem: Without attempting an action on an
>>> endpoint with a current scoped token, it is impossible to know what
>>> actions are available to a user.
>>> 
> 
> This is not unusual in the physical world. If you think about all the
> authz tokens you carry around in your pocket (as plastic cards), very
> few of them (if any) list what you are entitled to do with them. This
> gives the issuers and SPs flexibility to dynamically change your
> accesses rights without changing your authorisation. What you can do, in
> general terms, may be written in policy documents that you can consult
> if you wish. So you may wish to introduce a service that is equivalent
> to this (i.e. user may optionally consult some policy advice service).
> 
> If you introduce a service to allow a user to dynamically determine his
> access rights (absolutely), you have to decide what to do about the
> dynamics of this service compared to the lifetime of the keystone token,
> as the rights may change more quickly than the token's lifetime.
> 
>>> 
>>> Horizon makes some attempts to solve this issue by sourcing all of
>>> the policy files from all of the services to determine what a user
>>> can accomplish with a given role. This is highly inefficient as it
>>> requires processing the various policy.json files for each request
>>> in multiple places and presents a mechanism that is not really
>>> scalable to understand what a user can do with the current
>>> authorization. Horizon may not be the only service that (in the
>>> long term) would want to know what actions a token can take.
>> 
>> This is also extremely useful for being able to actually support
>> more restricted tokens as well.  If I as an end user want to request
>> a token that only has the roles required to perform a particular
>> action, I'm going to need to have a way of knowing what those roles
>> are.  I think that is one of the main things missing to allow the
>> "role-filtered tokens" option that I wrote up after the last Summit
>> to be a viable approach:
>> 
>> https://urldefense.proofpoint.com/v1/url?u=https://blog-nkinder.rhcloud.com/?p%3D101&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=XcBszEjYqiYgkoy9iUk3baKeyYoE%2Bb20k6zm3jIXGAs%3D%0A&s=ff78352cef9982b47c9e6cca97aa001f38b13837387332a38b56ce30e1394b87
>> 
>>> 
>>> I would like to start a discussion on how we should improve our
>>> policy implementation (OpenStack wide) to help make it easier to
>>> know what is possible with a current authorization context
>>> (Keystone token). The key feature should be that whatever the
>>> implementation is, it doesn’t require another round-trip to a third
>>> party service to “enforce” the policy which avoids another scaling
>>> point like UUID Keystone token validation.
> 
> Presumably this does not rule out the user, at his option, calling
> another service to ask for advice "what can I do with this token",
> bearing in mind that the response will be advice and not a definite
> answer (since the PDP will always be the one to provide the definitive
> answer).
> 
> 
> 
>>> 
>>> Here are a couple of ideas that we’ve discussed over the last few
>>> development cycles (and none of this changes the requirements to
>>> manage sc

Re: [openstack-dev] [all][policy][keystone] Better Policy Model and Representing Capabilites

[Tim Hinrichs]

That was really helpful background.  Thanks!

I’d be happy to look into using Congress to implement what we’ve discussed: 
caching policy.json files, updating them periodically, and answering queries 
about the roles required to be granted access to a certain kind of action.  I 
think we have the right algorithms sitting around.

Let me know if that would help or if you’d prefer a different approach.

Tim



On Oct 14, 2014, at 10:31 AM, Morgan Fainberg 
mailto:morgan.fainb...@gmail.com>> wrote:



2) What role(s) do I need to perform a particular action?

For me, the second question is more interesting.  A user likely already
has an idea of a task that they want to perform.  With question number
1, what do I do as a user if the response says that I'm not allowed to
perform the task I'm trying to accomplish?  The answer really doesn't
give me a way to move forward and perform my task.

With question 2, I'm able to find out what exact roles are needed to
perform a specific action.  With this information, I could request a
Keystone token with a subset of my roles that is authorized to perform
the task while leaving out roles that might have a higher level of
authorization.  For instance, why should I need to send a token with the
'admin' role to Nova just to launch an instance if '_member_' is all
that's required?

Another real use case is determining what roles are needed when creating
a trust in Keystone.  If I want to use a trust to allow a service like
Heat or Neutron's LBaaS to perform an action on my behalf, I want to
minimize the authorization that I'm delegating to those services.
Keystone trusts already have the ability to explicitly define the roles
that will be present in the issues trust tokens, but I have no way of
knowing what roles are required to perform a particular action without
consulting the policy

This sums up the large(er) part of or starting this conversation.


-NGK

>
> Tim
>
>
>
> On Oct 14, 2014, at 1:56 AM, David Chadwick 
> > wrote:
>
>>
>>
>> On 14/10/2014 01:25, Nathan Kinder wrote:
>>>
>>>
>>> On 10/13/2014 01:17 PM, Morgan Fainberg wrote:
 Description of the problem: Without attempting an action on an
 endpoint with a current scoped token, it is impossible to know what
 actions are available to a user.

>>
>> This is not unusual in the physical world. If you think about all the
>> authz tokens you carry around in your pocket (as plastic cards), very
>> few of them (if any) list what you are entitled to do with them. This
>> gives the issuers and SPs flexibility to dynamically change your
>> accesses rights without changing your authorisation. What you can do, in
>> general terms, may be written in policy documents that you can consult
>> if you wish. So you may wish to introduce a service that is equivalent
>> to this (i.e. user may optionally consult some policy advice service).
>>
>> If you introduce a service to allow a user to dynamically determine his
>> access rights (absolutely), you have to decide what to do about the
>> dynamics of this service compared to the lifetime of the keystone token,
>> as the rights may change more quickly than the token's lifetime.
>>

 Horizon makes some attempts to solve this issue by sourcing all of
 the policy files from all of the services to determine what a user
 can accomplish with a given role. This is highly inefficient as it
 requires processing the various policy.json files for each request
 in multiple places and presents a mechanism that is not really
 scalable to understand what a user can do with the current
 authorization. Horizon may not be the only service that (in the
 long term) would want to know what actions a token can take.
>>>
>>> This is also extremely useful for being able to actually support
>>> more restricted tokens as well.  If I as an end user want to request
>>> a token that only has the roles required to perform a particular
>>> action, I'm going to need to have a way of knowing what those roles
>>> are.  I think that is one of the main things missing to allow the
>>> "role-filtered tokens" option that I wrote up after the last Summit
>>> to be a viable approach:
>>>
>>> https://urldefense.proofpoint.com/v1/url?u=https://blog-nkinder.rhcloud.com/?p%3D101&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=XcBszEjYqiYgkoy9iUk3baKeyYoE%2Bb20k6zm3jIXGAs%3D%0A&s=ff78352cef9982b47c9e6cca97aa001f38b13837387332a38b56ce30e1394b87
>>>

 I would like to start a discussion on how we should improve our
 policy implementation (OpenStack wide) to help make it easier to
 know what is possible with a current authorization context
 (Keystone token). The key feature should be that whatever the
 implementation is, it doesn’t require another round-trip to a third
 party service to “enforce” the policy which avoids another scaling
 point like UUID Keystone token validation.
>>
>> Presumably this doe

Re: [openstack-dev] [congress] kilo design session

[Tim Hinrichs]

Works for me. 

Tim

P. S. Pardon the brevity. Sent from my mobile. 

> On Oct 22, 2014, at 5:01 PM, "Sean Roberts"  wrote:
> 
> We are scheduled for Monday, 03 Nov, 14:30 - 16:00. I have a conflict with 
> the “Meet the Influencers” talk that runs from 14:30-18:30, plus the GBP 
> session is on Tuesday, 04 Nov, 12:05-12:45. I was thinking we would want to 
> co-located the Congress and GBP talks as much as possible.
> 
> The BOSH team has the Tuesday, 04 Nov, 16:40-18:10 slot and wants to switch. 
> 
> Does this switch work for everyone?
> 
> Maybe we can get some space in one of the pods or cross-project workshops on 
> Tuesday between the GBP and the potential Congress session to make it even 
> more better.
> 
> ~sean
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Use Cases

[Tim Hinrichs]

Hi all,

We’ve been working on developing a list of use cases for Congress, which we’ve 
also started voting on so as to prioritize our efforts.  Please feel free take 
a look, leave feedback, add to the list, vote, whatever.

https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit

If you add a use case, please use the one called “Public/private networks with 
group membership” as a guide.  The policy doesn’t need to be written in the 
policy language syntax, but it needs to be precise enough for us to figure out 
if it can be written in that language.

For those of you who have already contributed use cases, could you take a look 
at “Public/private networks with group membership” to see what info I believe 
we need for each use case?  And then if your use case is missing something, 
could you add it (or leave a note explaining why the info is 
unnecessary/unknown)?

Thanks,
Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] data-source renovation

[Tim Hinrichs]

Hi all,

As I mentioned in a previous IRC, when writing our first few policies I had 
trouble using the tables we currently use to represent external data sources 
like Nova/Neutron.  

The main problem is that wide tables (those with many columns) are hard to use. 
 (a) it is hard to remember what all the columns are, (b) it is easy to 
mistakenly use the same variable in two different tables in the body of the 
rule, i.e. to create an accidental join, (c) changes to the datasource drivers 
can require tedious/error-prone modifications to policy.

I see several options.  Once we choose something, I’ll write up a spec and 
include the other options as alternatives.


1) Add a preprocessor to the policy engine that makes it easier to deal with 
large tables via named-argument references.

Instead of writing a rule like

p(port_id, name) :-
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts, 
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif, 
device_owner, mac_address, fixed_ips, router_id, binding_host)

we would write

p(id, nme) :-
neutron:ports(port_id=id, name=nme)

The preprocessor would fill in all the missing variables and hand the original 
rule off to the Datalog engine.

Pros: (i) leveraging vanilla database technology under the hood
  (ii) policy is robust to changes in the fields of the original data b/c 
the Congress data model is different than the Nova/Neutron data models
Cons: (i) we will need to invert the preprocessor when showing 
rules/traces/etc. to the user
  (ii) a layer of translation makes debugging difficult

2) Be disciplined about writing narrow tables and write 
tutorials/recommendations demonstrating how.

Instead of a table like...
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts, 
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif, 
device_owner, mac_address, fixed_ips, router_id, binding_host)

we would have many tables...
neutron:ports(port_id)
neutron:ports.addr_pairs(port_id, addr_pairs)
neutron:ports.security_groups(port_id, security_groups)
neutron:ports.extra_dhcp_opts(port_id, extra_dhcp_opts)
neutron:ports.name(port_id, name)
...

People writing policy would write rules such as ...

p(x) :- neutron:ports.name(port, name), ...

[Here, the period e.g. in ports.name is not an operator--just a convenient way 
to spell the tablename.]

To do this, Congress would need to know which columns in a table are sufficient 
to uniquely identify a row, which in most cases is just the ID.

Pros: (i) this requires only changes in the datasource drivers; everything else 
remains the same
  (ii) still leveraging database technology under the hood
  (iii) policy is robust to changes in fields of original data
Cons: (i) datasource driver can force policy writer to use wide tables
  (ii) this data model is much different than the original data models
  (iii) we need primary-key information about tables

3) Enhance the Congress policy language to handle objects natively.

Instead of writing a rule like the following ...

p(port_id, name, group) :-
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts, 
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif, 
device_owner, mac_address, fixed_ips, router_id, binding_host),
neutron:ports.security_groups(security_group, group)

we would write a rule such as
p(port_id, name) :-
neutron:ports(port),
port.name(name),
port.id(port_id),
port.security_groups(group)

The big difference here is that the period (.) is an operator in the language, 
just as in C++/Java.

Pros:
(i) The data model we use in Congress is almost exactly the same as the data 
model we use in Neutron/Nova.

(ii) Policy is robust to changes in the Neutron/Nova data model as long as 
those changes only ADD fields.

(iii) Programmers may be slightly more comfortable with this language.

Cons:

(i) The obvious implementation (changing the engine to implement the (.) 
operator directly is quite a change from traditional database technology.  At 
this point, that seems risky.

(ii) It is unclear how to implement this via a preprocessor (thereby leveraging 
database technology).  The key problem I see is that we would need to translate 
port.name(...) into something like option (2) above.  The difficulty is that 
TABLE could sometimes be a port, sometimes be a network, sometimes be a subnet, 
etc.

(iii) Requires some extra syntactic restrictions to ensure we don't lose 
decidability.

(iv) Because the Congress and Nova/Neutron models are the same, changes to the 
Nova/Neutron model can require rewriting policy.



Thoughts?
Tim
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Integration Demo

[Tim Hinrichs]

Hi Madhu,

Sounds like you’re doing some good stuff!  We’d love to have you involved.

Here’s the status of the items you mentioned.

- We’re about to merge a keystone/devstack integration.  
- We have an HTTP API merged. 
- arosen was working on a python-client; not sure how far along that is.
- As you mentioned, the Tetris group, who just joined, also has plans for 
optimization.  No work has been done in this area yet.
- kudva is working on a ceilometer integration.
- Hopefully soon we’ll have an end-to-end example at least pushed to review, 
along with docs.

I’d suggest touching base with arosen discuss the python-client, since you’ve 
already made progress on that.

The other thing I’d suggest is touching base with gokul, one of the Tetris 
members, and discuss optimization.

I’ll let you know once I’ve put some basic docs together along with a working 
example so you can start getting a feel for the end-to-end flow.

Tim





On Jul 30, 2014, at 10:12 PM, Madhu Mohan Nelemane  wrote:

> Hi All,
> 
> I have been working on integrating developing a policy demo using Congress. I 
> have done following activities in this direction
> - Integrated Congress as a service with keystone 
> - Developed a trivial Python-Congressclient with support for basic commands 
> through command line
> - Integrated congress and python-congressclient into devstack
> 
> To verify the integration, I am using the example policy files (VM Grouping 
> policy)  as input and trying to see the policies being written to the 
> Congress tables and the Data source tables.
> 
> Could some one help me out understand what are the other missing parts to get 
> this integration working ? What needs to be implemented from the Congress 
> server side ?
> 
> I am particularly interested in Contributing to all areas of Congress 
> including policy definition, enforcement and also have special interests in 
> run-time optimization, high-availability use-cases.
> 
> Any guidance would be of utmost help.
> 
> Thanks and Regards,
> Madhu Mohan
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Congress: an open policy framework

[Tim Hinrichs]

Hi OpenStackers,

We've been working on an open policy framework for OpenStack that we're calling 
Congress.  We've been talking with OpenStack users and several of our partners 
to understand the kinds of rules and regulations they envision enforcing with a 
policy-based management framework.  Across the board they are interested in 
policies that span networking, compute, storage, etc.

The idea behind Congress is to have a single policy engine that integrates any 
collection of external authentication and data stores and allows cloud 
administrators to write policies over those data stores in a rich, declarative 
language.  The policy engine can either enforce the policy proactively (i.e. 
preventing policy violations before they occur) or reactively (identifying 
violations after they occur and taking corrective action) or a combination 
(proactively when possible and reactively when not).  The policy engine can 
also interact with the administrator, explaining the causes of violations, 
computing potential remediation plans, and simulating action executions to 
understand what violations those actions might cause.  

While the project is still in the early stages, we have identified a grammar 
for the policy language, implemented a policy engine, and written a proof of 
concept integration for ActiveDirectory.  We would love to get participation 
and feedback.  

Code (in the midst of moving to stackforge):
https://github.com/pballand/congress

Wiki:
https://wiki.openstack.org/wiki/Congress

We'll be in Hong Kong, so if you would like to meet up to discuss please e-mail 
Peter  and Pierre .

-- The Congress Team

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Congress: an open policy framework

[Tim Hinrichs]

We're just getting started with Congress and understanding how it will 
integrate with the OS ecosystem, but here's our current thinking about how 
Congress relates to Oslo's policy engine and to Keystone.  Comments and 
suggestions are welcome.


Congress and Oslo

Three dimensions for comparison: policy language, data sources, and policy 
engine.  

We've always planned to make Congress compatible with existing policy languages 
like the one in oslo.  The plan is to build a front-end for a number of policy 
languages/formats, e.g. oslo-policy language, XACML, JSON, YAML, SQL, etc.  The 
idea being that the syntax/language you use is irrelevant as long as it can be 
mapped into Congress's native policy language.  As of now, Congress is using 
Datalog, which is a variant of SQL and is at least as expressive as all of the 
policy languages we've run across in the cloud domain, including the 
oslo-policy language.

In terms of the data sources you can reference in the policy, Congress is 
designed to enable policies that reference arbitrary data sources in the cloud. 
 For example, we could write a Nova authorization policy that permits a new VM 
to be created if that VM is connected to a network owned by a tenant (info 
stored in Neutron) where the VM owner (info in the request) is in the same 
group as the network owner (info stored in Keystone/LDAP).  Oslo's handles some 
of these data sources with its terminal rules, but it's not involved in data 
integration to the same extent Congress is.

In terms of policy engines, Congress is intended to enforce policies in 2 
different ways: proactively (stopping policy violations before they occur) and 
reactively (acting to eliminate a violation after it occurs).  Ideally we 
wouldn't need reactive enforcement, but there will always be cases where 
proactive enforcement is not possible (e.g. a DOS attack brings app latencies 
out of compliance).  The oslo-engine does proactive enforcement only--stopping 
API calls before they violate the policy.

One concrete integration idea would be to treat Congress as a plugin for the 
oslo-policy engine.  This wouldn't enable say Nova to write policies that take 
advantage of the expressiveness of Datalog, but it would give us backwards 
compatibility.

Congress and Keystone
--
I see Keystone as providing two pieces of functionality: authentication and 
group membership.  Congress has nothing to do with authentication and never 
will.  Groups, on the other hand, are things we end up defining when writing 
policies in Congress, so conceptually there's some overlap with Keystone.  I 
guess Congress could serve as a plugin/data source for Keystone and provide it 
with the groups defined within the policy.  This would allow a group to be 
defined using data sources not available to Keystone, e.g. we could define a 
group as all users who own a VM (info from Nova) connected to a network owned 
by someone (info from Neutron) in the same group (info from LDAP).  I don't 
know how useful or efficient this would be, and it's certainly not something 
we've designed Congress for.

Thoughts?
Tim





- Original Message -
| From: "Dolph Mathews" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Monday, November 11, 2013 3:10:01 PM
| Subject: Re: [openstack-dev] Congress: an open policy framework
| 
| 
| 
| 
| 
| 
| On Mon, Nov 11, 2013 at 4:28 AM, Flavio Percoco < fla...@redhat.com >
| wrote:
| 
| 
| 
| On 02/11/13 21:31 -0700, Tim Hinrichs wrote:
| 
| 
| Hi OpenStackers,
| 
| We've been working on an open policy framework for OpenStack that
| we're calling Congress. We've been talking with OpenStack users and
| several of our partners to understand the kinds of rules and
| regulations they envision enforcing with a policy-based management
| framework. Across the board they are interested in policies that
| span networking, compute, storage, etc.
| 
| The idea behind Congress is to have a single policy engine that
| integrates any collection of external authentication and data stores
| and allows cloud administrators to write policies over those data
| stores in a rich, declarative language. The policy engine can either
| enforce the policy proactively (i.e. preventing policy violations
| before they occur) or reactively (identifying violations after they
| occur and taking corrective action) or a combination (proactively
| when possible and reactively when not). The policy engine can also
| interact with the administrator, explaining the causes of
| violations, computing potential remediation plans, and simulating
| action executions to understand what violations those actions might
| cause.
| 
| While the project is still in the early stages, we have identified a
| grammar for the policy language, implemented a policy engine, and
| written a proof 

Re: [openstack-dev] [neutron] Group-based Policy Sub-team Meetings

[Tim Hinrichs]

Are there plans for a concrete policy language (e.g. a grammar and semantics) 
to be part of the proposal, or does each plugin to Neutron supply its own 
policy language?

I'm trying to envision how Heat would utilize the policy API.  If there's a 
concrete policy language, then Heat can take an app template, extract the 
networking-relevant policy, and express that policy in the concrete language.  
Then whatever plugin we're using for Neutron can implement that policy in any 
way it sees fit as long as it obeys the policy's semantics (according to the 
language--the semantics Heat intended).

But if there's no concrete policy language, how does Heat know which policy 
statements to send?  It doesn't know which plugin is being used for Neutron.  
So it doesn't even know which strings are valid policy statements.  Or are we 
assuming that Heat knows which plugin is being used for Neutron?  Or am I 
missing something?

Thanks,
Tim

- Original Message -
| From: "Kyle Mestery (kmestery)" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Wednesday, November 13, 2013 9:57:54 AM
| Subject: Re: [openstack-dev] [neutron] Group-based Policy Sub-team Meetings
| 
| On Nov 13, 2013, at 10:36 AM, Stephen Wong 
|  wrote:
| 
| > Hi Kyle,
| > 
| >So no meeting this Thursday?
| > 
| I am inclined to skip this week's meeting due to the fact I haven't
| heard many
| replies yet. I think a good starting point for people would be to
| review the
| BP [1] and Design Document [2] and provide feedback where
| appropriate.
| We should start to formalize what the APIs will look like at next
| week's meeting,
| and the Design Document has a first pass at this.
| 
| Thanks,
| Kyle
| 
| [1]
| https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction
| [2]
| 
https://docs.google.com/document/d/1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit?usp=sharing
| 
| > Thanks,
| > - Stephen
| > 
| > On Wed, Nov 13, 2013 at 7:11 AM, Kyle Mestery (kmestery)
| >  wrote:
| >> On Nov 13, 2013, at 8:58 AM, "Stein, Manuel (Manuel)"
| >>  wrote:
| >> 
| >>> Kyle,
| >>> 
| >>> I'm afraid your meeting vanished from the Meetings page [2] when
| >>> user amotiki reworked neutron meetings ^.^
| >>> Is the meeting for Thu 1600 UTC still on?
| >>> 
| >> Ack, thanks for the heads up here! I have re-added the meeting. I
| >> only heard
| >> back from one other person other than yourself, so at this point
| >> I'm inclined
| >> to wait until next week to hold our first meeting unless I hear
| >> back from others.
| >> 
| >>> A few heads-up questions (couldn't attend the HK design summit
| >>> Friday meeting):
| >>> 
| >>> 1) In the summit session Etherpad [3], ML2 implementation
| >>> mentions insertion of arbitrary metadata to hint to underlying
| >>> implementation. Is that (a) the plug-ing reporting its
| >>> policy-bound realization? (b) the user further specifying what
| >>> should be used? (c) both? Or (d) none of that but just some
| >>> arbitrary message of the day?
| >>> 
| >> I believe that would be (a).
| >> 
| >>> 2) Would policies _always_ map to the old Neutron entities?
| >>> E.g. when I have policies in place, can I query related
| >>> network/port, subnet/address, router elements on the API or are
| >>> there no equivalents created? Would the logical topology created
| >>> under the policies be exposed otherwise? for e.g.
| >>> monitoring/wysiwyg/troubleshoot purposes.
| >>> 
| >> No, this is up to the plugin/MechanismDriver implementation.
| >> 
| >>> 3) Do the chain identifier in your policy rule actions match to
| >>> "Service Chain UUID" in Service Insertion, Chaining and API [4]
| >>> 
| >> That's one way to look at this, yes.
| >> 
| >>> 4) Are you going to describe L2 services the way group policies
| >>> work? I mean, why would I need a LoadBalancer or Firewall
| >>> instance before I can insert it between two groups when all that
| >>> load balancing/firewalling requires is nothing but a policy for
| >>> group communication itself? - regardless the service instance
| >>> used to carry out the service.
| >>> 
| >> These are things I'd like to discuss at the IRC meeting each week.
| >> The goal
| >> would be to try and come up with some actionable items we can
| >> drive towards
| >> in both Icehouse-1 and Icehouse-2. Given how close the closing of
| >> Icehouse-1
| >> is, we need to focus on this very fast if we want to have a
| >> measurable impact in
| >> Icehouse-1.
| >> 
| >> Thanks,
| >> Kyle
| >> 
| >> 
| >>> Best, Manuel
| >>> 
| >>> [2]
| >>> 
https://wiki.openstack.org/wiki/Meetings#Neutron_Group_Policy_Sub-Team_Meeting
| >>> [3]
| >>> 
https://etherpad.openstack.org/p/Group_Based_Policy_Abstraction_for_Neutron
| >>> [4]
| >>> 
https://docs.google.com/document/d/1fmCWpCxAN4g5txmCJVmBDt02GYew2kvyRsh0Wl3YF2U/edit#
| >>> 
|  -Original Message-
|  From: Kyle Mestery (kmestery) [mailto:kmest...@cisco.com]
|  Sent: Montag, 11. November 2013 19:41
| 

Re: [openstack-dev] Congress: an open policy framework

[Tim Hinrichs]

I completely agree that making Congress successful will rely crucially on 
addressing performance and scalability issues.  Some thoughts...

1. We're definitely intending to cache data locally to avoid repeated API 
calls.  In fact, a prototype cache is already in place.  We haven't yet hooked 
up API calls (other than to AD).  We envision some data sources pushing us data 
(updates) and some data sources requiring us to periodically pull.  

2. My main concern for scalability/performance is for proactive enforcement, 
where at least conceptually Congress is on the critical path for API calls.

One thought is that we could splice out, say, the network portion of the 
Congress policy and push it down into neutron, assuming neutron could enforce 
that policy.  This would at least eliminate cross-component communication.  It 
would require a policy engine on each of the OS components, but (a) there 
already is one on many components and (b) if there isn't, we can rely on 
reactive enforcement.

The downside with policy-caching on other OS components are the usual problems 
with staleness and data replication, e.g. maybe we'd end up copying all of 
nova's VM data into neutron so that neutron could enforce its policy.  But 
because we have reactive enforcement to rely on, we could always approximate 
the policy that we push down (conservatively) to catch the common mistakes and 
leave the remainder to reactive enforcement.  For example, we might be able to 
auto-generate the current policy.json files for each component from Congress's 
policy.

Keeping Congress out of the critical path for every API call is one of the 
reasons it was designed to do reactive enforcement as well as proactive 
enforcement.

3. Another option is to build high-performance optimizations for certain 
fragments of the policy language.  Then the cloud architect can decide whether 
she wants to utilize a more expressive language whose performance is worse or a 
less expressive language whose performance is better.

Tim




- Original Message -
| From: "Flavio Percoco" 
| To: "OpenStack Development Mailing List (not for usage questions)" 

| Sent: Thursday, November 14, 2013 6:05:46 AM
| Subject: Re: [openstack-dev] Congress: an open policy framework
| 
| On 14/11/13 04:40 -0800, Morgan Fainberg wrote:
| >On Wed, Nov 13, 2013 at 10:40 AM, Tim Hinrichs
| > wrote:
| >> We're just getting started with Congress and understanding how it
| >> will integrate with the OS ecosystem, but here's our current
| >> thinking about how Congress relates to Oslo's policy engine and
| >> to Keystone.  Comments and suggestions are welcome.
| >>
| >>
| >> Congress and Oslo
| >> 
| >> Three dimensions for comparison: policy language, data sources,
| >> and policy engine.
| >>
| >> We've always planned to make Congress compatible with existing
| >> policy languages like the one in oslo.  The plan is to build a
| >> front-end for a number of policy languages/formats, e.g.
| >> oslo-policy language, XACML, JSON, YAML, SQL, etc.  The idea
| >> being that the syntax/language you use is irrelevant as long as
| >> it can be mapped into Congress's native policy language.  As of
| >> now, Congress is using Datalog, which is a variant of SQL and is
| >> at least as expressive as all of the policy languages we've run
| >> across in the cloud domain, including the oslo-policy language.
| >>
| >> In terms of the data sources you can reference in the policy,
| >> Congress is designed to enable policies that reference arbitrary
| >> data sources in the cloud.  For example, we could write a Nova
| >> authorization policy that permits a new VM to be created if that
| >> VM is connected to a network owned by a tenant (info stored in
| >> Neutron) where the VM owner (info in the request) is in the same
| >> group as the network owner (info stored in Keystone/LDAP).
| >>  Oslo's handles some of these data sources with its terminal
| >> rules, but it's not involved in data integration to the same
| >> extent Congress is.
| >>
| >> In terms of policy engines, Congress is intended to enforce
| >> policies in 2 different ways: proactively (stopping policy
| >> violations before they occur) and reactively (acting to eliminate
| >> a violation after it occurs).  Ideally we wouldn't need reactive
| >> enforcement, but there will always be cases where proactive
| >> enforcement is not possible (e.g. a DOS attack brings app
| >> latencies out of compliance).  The oslo-engine does proactive
| >> enforcement only--stopping API calls before they violate the
| >> policy.
| >>
| >
| >Does this mean all poli

Re: [openstack-dev] Congress: an open policy framework

[Tim Hinrichs]

Inline.

- Original Message -
| From: "Adam Young" 
| To: openstack-dev@lists.openstack.org
| Sent: Friday, November 15, 2013 2:46:02 PM
| Subject: Re: [openstack-dev] Congress: an open policy framework
|
| On 11/14/2013 11:39 AM, Tim Hinrichs wrote:
| > I completely agree that making Congress successful will rely
| > crucially on addressing performance and scalability issues.  Some
| > thoughts...
| >
| > 1. We're definitely intending to cache data locally to avoid
| > repeated API calls.  In fact, a prototype cache is already in
| > place.  We haven't yet hooked up API calls (other than to AD).  We
| > envision some data sources pushing us data (updates) and some data
| > sources requiring us to periodically pull.
| So, I think that you should start with what we already have working.
| Policy distribution is a part of Keystone now.  Policy processing
| uses
| Oslo policy.py.  I am not sure why it would make sense to have a
| separate program.  I would be more than happy to house this work
| inside
| the existing Keystone architecture.
|
| >
| > 2. My main concern for scalability/performance is for proactive
| > enforcement, where at least conceptually Congress is on the
| > critical path for API calls.
| >
| > One thought is that we could splice out, say, the network portion
| > of the Congress policy and push it down into neutron, assuming
| > neutron could enforce that policy.  This would at least eliminate
| > cross-component communication.  It would require a policy engine
| > on each of the OS components, but (a) there already is one on many
| > components and (b) if there isn't, we can rely on reactive
| > enforcement.
| >
| > The downside with policy-caching on other OS components are the
| > usual problems with staleness and data replication, e.g. maybe
| > we'd end up copying all of nova's VM data into neutron so that
| > neutron could enforce its policy.  But because we have reactive
| > enforcement to rely on, we could always approximate the policy
| > that we push down (conservatively) to catch the common mistakes
| > and leave the remainder to reactive enforcement.  For example, we
| > might be able to auto-generate the current policy.json files for
| > each component from Congress's policy.
| >
| > Keeping Congress out of the critical path for every API call is one
| > of the reasons it was designed to do reactive enforcement as well
| > as proactive enforcement.
|
| I think all these questions have come up multiple times with
| Keystone.
| There was a signifcant discussion about Configuration management,
| which
| would encompass policy.  We were discussing the Key Distribution
| Service
| as well. It bascially comes down to "inside the undercloud" versus
| outside (end user facing).
|
| Termie's suggestion was that we needed to consider using something
| like
| Zookeeper.  I think he is right.
|

I've heard different people mean different things by "configuration mgmt".  
Some are talking about managing server configurations e.g. with Puppet.  Others 
are talking about managing the overall configuration of the cloud across 
components (e.g. every network connected to a VM must be owned by someone in 
the same group as the VMs owner).  Congress is intended to support whatever 
configuration management/policy you can write in terms of the cloud services 
you have available.  We don't differentiate between configuration management 
and policy.

| >
| > 3. Another option is to build high-performance optimizations for
| > certain fragments of the policy language.  Then the cloud
| > architect can decide whether she wants to utilize a more
| > expressive language whose performance is worse or a less
| > expressive language whose performance is better.
|
| So we should not be writing yet another policy language.  We have a
| simple one in Open Stack already, and there are many, many ones out
| there already.  XACML is the standard for authorization decisions, as
| an
| example.  Puppet  falls into this role as well.
|

Not all languages are created equal, which is especially true of 
declarative/policy languages.  The one we've prototyped is more expressive than 
the languages you mention (though is general-purpose, not domain-specific like 
Puppet).  We're not inventing yet another policy language--we're using a core 
fragment of SQL that's been studied and used for 50 years.  We're simply 
advocating its use.  We're also building a policy engine around it for both 
proactive enforcement (stopping violations before they occur) and reactive 
enforcement (correcting violations after they occur).  We're also adding 
algorithms that help an admin understand her current policy as well as 
hypothetical changes to that policy and its data sources.  Finally, we

[openstack-dev] [Congress] Monasca @10a on Thu in Tokyo

[Tim Hinrichs]

Hi all,

At the summit next week, we're planning on a 10a meeting on Thursday with
the Monasca team for a deep dive.  All are welcome.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] IRC canceled during summit

[Tim Hinrichs]

As you might expect, next week's IRC is canceled since many of us will be
attending the Tokyo summit.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Summit recap

[Tim Hinrichs]

Hi all,

It was great seeing so many Congress people in Tokyo last week!  Hopefully
you've all had a chance to recover by now.  Here's an overview of what
happened.  I was planning to go over this at this week's IRC meeting, but
forgot about the U.S. time change and missed the meeting--sorry about that.

1. Hands On Lab.   There were 40-50 people who attended, and all but 3-4 of
them got the VM we provided installed and worked through the lab.  1 of the
failures didn't have enough memory; 1 was something to do with VDX (?
Eric--is that right?); 1 was a version of Linux for which there wasn't a
VirtualBox installer.  The only weird problem was a glitch with the Horizon
interface that wouldn't show a table that we could show on the command
line.  Overall, people seemed to like Congress and what it had to offer.

2. Working session: distributed architecture
Base class is working with oslo-messaging, but unit tests are not working.
Peter is planning to debug and push to review in the next few weeks.

One thing we discussed was that the distributed architecture is only a
building block for an HA design.  But it does not deliver HA.  In
particular, for HA we will want to have multiple copies of the policy
engine, and these copies should be hidden from the user; the system should
take care of mapping an API call intended for the policy engine to one of
the copies.  The distributed architecture does not hide the existence of
multiple policy engines; rather, the user is responsible for spinning up
multiple policy engines, giving them different names, and directing API
requests to whichever one of the policy engines she wants to interact with.

3. Working session: infrastructure/testing
- We agreed to add Murano tests to our gate (as non-voting) to ensure that
we know when we add something to Congress that breaks Murano.  Should be
sufficient to simply copy their jenkins job into the Congress job-list and
make that job non-voting.

- We discussed the problem of datasource drivers, where to store them, and
how to test them.  Neutron has a similar issue with vendor-specific
plugins.  We thought it would be nice to have a separate requirements.txt
file for each driver; but then it is unclear how to test datasource drivers
in the gate because setup.py only installs the 1 requirements.txt in the
root directory.  So in the end, we decided the right thing was to have 1
requirements.txt file that includes all the dependencies for the OpenStack
drivers so that we can test those in the gate, and to have a separate
requirements.txt for each of the non-OpenStack drivers, since we can't test
those in the gate anyway.

4. Working session: Monasca and NFV.
- Fabio introduced us to Monasca, which is a monitoring project about to be
accepted into the BigTent.  It is an alternative to Ceilometer and focused
on high-performance.  They have alarms that can be set to inform the caller
any time a certain kind of event occurs.  Monasca is supposed to get a
superset of the data that Congress currently has drivers for.  They
suggested that Congress could automatically generate alarms based on the
data required by policy.  As a first step, we decided to write a simple
datasource driver to integrate with Monasca, as an easy way for the
Congress team to get familiar with Monasca.

- OPNFV Doctor project.  The Doctor project aims to detect and manage
faults in OPNFV platforms.  They hoped to use Congress to help identify
faults.  They wanted to connect Zabbix to Congress, which creates events
and have Congress push out config changes.  Concretely they asked for a
push-style datasource driver so that Zabbix could push data to Congress
through the API.  The blueprint for that work is here:
https://blueprints.launchpad.net/congress/+spec/push-type-datasource-driver

5. Discussion about Application-level Intent.

Outside the working sessions we talked with Ken Owens and his team about
application-level intent.  They are planning on building an
application-specific policy engine within the Congress framework.  For each
VM in an application, the user can rank the sensitivity of that VM as
low/medium/high for a handful of properties, e.g. latency, throughput.  The
provisioning system (which is external to Congress) then provisions the app
according to that policy, and the policy engine within Congress continually
monitors those properties and corrects violations.  The plan is to start
this as a completely standalone policy engine running a Congress node but
build it with an eye toward eventually delegating from the agnostic policy
engine to the application-intent engine.

6. Senlin project.  I heard about this project for the first time at the
summit.  It's policy-based cluster management.  Here's an email with more
details.

http://lists.openstack.org/pipermail/openstack-dev/2015-November/078498.html

It'd be great if those attended could respond with clarifications,
comments, and things I missed.

Let me know if anyone has questions/comments.
Tim
__

Re: [openstack-dev] [keystone] [Mistral] Autoprovisioning, per-user projects, and Federation

[Tim Hinrichs]

Congress allows users to write a policy that executes an action under
certain conditions.

The conditions can be based on any data Congress has access to, which
includes nova servers, neutron networks, cinder storage, keystone users,
etc.  We also have some Ceilometer statistics; I'm not sure about whether
it's easy to get the Keystone notifications that you're talking about
today, but notifications are on our roadmap.  If the user's login is
reflected in the Keystone API, we may already be getting that event.

The action could in theory be a mistral/heat API or an arbitrary script.
Right now we're set up to invoke any method on any of the python-clients
we've integrated with.  We've got an integration with heat but not
mistral.  New integrations are typically easy.

Happy to talk more.

Tim



On Fri, Nov 6, 2015 at 9:17 AM Doug Hellmann  wrote:

> Excerpts from Dolph Mathews's message of 2015-11-05 16:31:28 -0600:
> > On Thu, Nov 5, 2015 at 3:43 PM, Doug Hellmann 
> wrote:
> >
> > > Excerpts from Clint Byrum's message of 2015-11-05 10:09:49 -0800:
> > > > Excerpts from Doug Hellmann's message of 2015-11-05 09:51:41 -0800:
> > > > > Excerpts from Adam Young's message of 2015-11-05 12:34:12 -0500:
> > > > > > Can people help me work through the right set of tools for this
> use
> > > case
> > > > > > (has come up from several Operators) and map out a plan to
> implement
> > > it:
> > > > > >
> > > > > > Large cloud with many users coming from multiple Federation
> sources
> > > has
> > > > > > a policy of providing a minimal setup for each user upon first
> visit
> > > to
> > > > > > the cloud:  Create a project for the user with a minimal quota,
> and
> > > > > > provide them a role assignment.
> > > > > >
> > > > > > Here are the gaps, as I see it:
> > > > > >
> > > > > > 1.  Keystone provides a notification that a user has logged in,
> but
> > > > > > there is nothing capable of executing on this notification at the
> > > > > > moment.  Only Ceilometer listens to Keystone notifications.
> > > > > >
> > > > > > 2.  Keystone does not have a workflow engine, and should not be
> > > > > > auto-creating projects.  This is something that should be
> performed
> > > via
> > > > > > a Heat template, and Keystone does not know about Heat, nor
> should
> > > it.
> > > > > >
> > > > > > 3.  The Mapping code is pretty static; it assumes a user entry
> or a
> > > > > > group entry in identity when creating a role assignment, and
> neither
> > > > > > will exist.
> > > > > >
> > > > > > We can assume a special domain for Federated users to have
> per-user
> > > > > > projects.
> > > > > >
> > > > > > So; lets assume a Heat Template that does the following:
> > > > > >
> > > > > > 1. Creates a user in the per-user-projects domain
> > > > > > 2. Assigns a role to the Federated user in that project
> > > > > > 3. Sets the minimal quota for the user
> > > > > > 4. Somehow notifies the user that the project has been set up.
> > > > > >
> > > > > > This last probably assumes an email address from the Federated
> > > > > > assertion.  Otherwise, the user hits Horizon, gets a "not
> > > authenticated
> > > > > > for any projects" error, and is stumped.
> > > > > >
> > > > > > How is quota assignment done in the other projects now?  What
> happens
> > > > > > when a project is created in Keystone?  Does that information
> gets
> > > > > > transferred to the other services, and, if so, how?  Do most
> people
> > > use
> > > > > > a custom provisioning tool for this workflow?
> > > > > >
> > > > >
> > > > > I know at Dreamhost we built some custom integration that was
> triggered
> > > > > when someone turned on the Dreamcompute service in their account
> in our
> > > > > existing user management system. That integration created the
> account
> > > in
> > > > > keystone, set up a default network in neutron, etc. I've long
> thought
> > > we
> > > > > needed a "new tenant creation" service of some sort, that sits
> outside
> > > > > of our existing services and pokes them to do something when a new
> > > > > tenant is established. Using heat as the implementation makes
> sense,
> > > for
> > > > > things that heat can control, but we don't want keystone to depend
> on
> > > > > heat and we don't want to bake such a specialized feature into heat
> > > > > itself.
> > > > >
> > > >
> > > > I agree, an automation piece that is built-in and easy to add to
> > > > OpenStack would be great.
> > > >
> > > > I do not agree that it should be Heat. Heat is for managing stacks
> that
> > > > live on and change over time and thus need the complexity of the
> graph
> > > > model Heat presents.
> > > >
> > > > I'd actually say that Mistral or Ansible are better choices for
> this. A
> > > > service which listens to the notification bus and triggered a
> workflow
> > > > defined somewhere in either Ansible playbooks or Mistral's workflow
> > > > language would simply run through the "skel" workflow for each user.
> > > >
> > > > The actual wor

Re: [openstack-dev] [keystone] [Mistral] Autoprovisioning, per-user projects, and Federation

[Tim Hinrichs]

Congress happens to have the capability to run a script/API call under
arbitrary conditions on the state of other OpenStack projects, which
sounded like what you wanted.  Or did I misread your original question?

Congress and Mistral are definitely not competing.Congress lets people
declare which states of the other OpenStack projects are permitted using a
general purpose policy language, but it does not try to make complex
changes (often requiring a workflow) to eliminate prohibited states.
Mistral lets people create a workflow that makes complex changes to other
OpenStack projects, but it doesn't have a general purpose policy language
that describes which states are permitted.  Congress and Mistral are
complementary, and each can stand on its own.

Tim


On Mon, Nov 9, 2015 at 6:46 AM Adam Young  wrote:

> On 11/06/2015 06:28 PM, Tim Hinrichs wrote:
>
> Congress allows users to write a policy that executes an action under
> certain conditions.
>
> The conditions can be based on any data Congress has access to, which
> includes nova servers, neutron networks, cinder storage, keystone users,
> etc.  We also have some Ceilometer statistics; I'm not sure about whether
> it's easy to get the Keystone notifications that you're talking about
> today, but notifications are on our roadmap.  If the user's login is
> reflected in the Keystone API, we may already be getting that event.
>
> The action could in theory be a mistral/heat API or an arbitrary script.
> Right now we're set up to invoke any method on any of the python-clients
> we've integrated with.  We've got an integration with heat but not
> mistral.  New integrations are typically easy.
>
>
> Sounds like Mistral and Congress are competing here, then.  Maybe we
> should merge those efforts.
>
>
>
> Happy to talk more.
>
> Tim
>
>
>
> On Fri, Nov 6, 2015 at 9:17 AM Doug Hellmann 
> wrote:
>
>> Excerpts from Dolph Mathews's message of 2015-11-05 16:31:28 -0600:
>> > On Thu, Nov 5, 2015 at 3:43 PM, Doug Hellmann 
>> wrote:
>> >
>> > > Excerpts from Clint Byrum's message of 2015-11-05 10:09:49 -0800:
>> > > > Excerpts from Doug Hellmann's message of 2015-11-05 09:51:41 -0800:
>> > > > > Excerpts from Adam Young's message of 2015-11-05 12:34:12 -0500:
>> > > > > > Can people help me work through the right set of tools for this
>> use
>> > > case
>> > > > > > (has come up from several Operators) and map out a plan to
>> implement
>> > > it:
>> > > > > >
>> > > > > > Large cloud with many users coming from multiple Federation
>> sources
>> > > has
>> > > > > > a policy of providing a minimal setup for each user upon first
>> visit
>> > > to
>> > > > > > the cloud:  Create a project for the user with a minimal quota,
>> and
>> > > > > > provide them a role assignment.
>> > > > > >
>> > > > > > Here are the gaps, as I see it:
>> > > > > >
>> > > > > > 1.  Keystone provides a notification that a user has logged in,
>> but
>> > > > > > there is nothing capable of executing on this notification at
>> the
>> > > > > > moment.  Only Ceilometer listens to Keystone notifications.
>> > > > > >
>> > > > > > 2.  Keystone does not have a workflow engine, and should not be
>> > > > > > auto-creating projects.  This is something that should be
>> performed
>> > > via
>> > > > > > a Heat template, and Keystone does not know about Heat, nor
>> should
>> > > it.
>> > > > > >
>> > > > > > 3.  The Mapping code is pretty static; it assumes a user entry
>> or a
>> > > > > > group entry in identity when creating a role assignment, and
>> neither
>> > > > > > will exist.
>> > > > > >
>> > > > > > We can assume a special domain for Federated users to have
>> per-user
>> > > > > > projects.
>> > > > > >
>> > > > > > So; lets assume a Heat Template that does the following:
>> > > > > >
>> > > > > > 1. Creates a user in the per-user-projects domain
>> > > > > > 2. Assigns a role to the Federated user in that project
>> > > > > > 3. Sets the minimal quota for the user
>> > > > &

Re: [openstack-dev] [keystone] [Mistral] Autoprovisioning, per-user projects, and Federation

[Tim Hinrichs]

They shouldn't be combined because they can each be used without the
other.  That is, they each stand on their own.

Congress can be used for monitoring or delegating policy without attempting
to correct violations (i.e. without needing workflows).

Mistral can be used to make complex changes without writing a policy.

Tim





On Mon, Nov 9, 2015 at 8:57 AM Adam Young  wrote:

> On 11/09/2015 10:57 AM, Tim Hinrichs wrote:
>
> Congress happens to have the capability to run a script/API call under
> arbitrary conditions on the state of other OpenStack projects, which
> sounded like what you wanted.  Or did I misread your original question?
>
> Congress and Mistral are definitely not competing.Congress lets
> people declare which states of the other OpenStack projects are permitted
> using a general purpose policy language, but it does not try to make
> complex changes (often requiring a workflow) to eliminate prohibited
> states.  Mistral lets people create a workflow that makes complex changes
> to other OpenStack projects, but it doesn't have a general purpose policy
> language that describes which states are permitted.  Congress and Mistral
> are complementary, and each can stand on its own.
>
>
> And why should not these two things be in a single project?
>
>
>
>
> Tim
>
>
> On Mon, Nov 9, 2015 at 6:46 AM Adam Young  wrote:
>
>> On 11/06/2015 06:28 PM, Tim Hinrichs wrote:
>>
>> Congress allows users to write a policy that executes an action under
>> certain conditions.
>>
>> The conditions can be based on any data Congress has access to, which
>> includes nova servers, neutron networks, cinder storage, keystone users,
>> etc.  We also have some Ceilometer statistics; I'm not sure about whether
>> it's easy to get the Keystone notifications that you're talking about
>> today, but notifications are on our roadmap.  If the user's login is
>> reflected in the Keystone API, we may already be getting that event.
>>
>> The action could in theory be a mistral/heat API or an arbitrary script.
>> Right now we're set up to invoke any method on any of the python-clients
>> we've integrated with.  We've got an integration with heat but not
>> mistral.  New integrations are typically easy.
>>
>>
>> Sounds like Mistral and Congress are competing here, then.  Maybe we
>> should merge those efforts.
>>
>>
>>
>> Happy to talk more.
>>
>> Tim
>>
>>
>>
>> On Fri, Nov 6, 2015 at 9:17 AM Doug Hellmann 
>> wrote:
>>
>>> Excerpts from Dolph Mathews's message of 2015-11-05 16:31:28 -0600:
>>> > On Thu, Nov 5, 2015 at 3:43 PM, Doug Hellmann 
>>> wrote:
>>> >
>>> > > Excerpts from Clint Byrum's message of 2015-11-05 10:09:49 -0800:
>>> > > > Excerpts from Doug Hellmann's message of 2015-11-05 09:51:41 -0800:
>>> > > > > Excerpts from Adam Young's message of 2015-11-05 12:34:12 -0500:
>>> > > > > > Can people help me work through the right set of tools for
>>> this use
>>> > > case
>>> > > > > > (has come up from several Operators) and map out a plan to
>>> implement
>>> > > it:
>>> > > > > >
>>> > > > > > Large cloud with many users coming from multiple Federation
>>> sources
>>> > > has
>>> > > > > > a policy of providing a minimal setup for each user upon first
>>> visit
>>> > > to
>>> > > > > > the cloud:  Create a project for the user with a minimal
>>> quota, and
>>> > > > > > provide them a role assignment.
>>> > > > > >
>>> > > > > > Here are the gaps, as I see it:
>>> > > > > >
>>> > > > > > 1.  Keystone provides a notification that a user has logged
>>> in, but
>>> > > > > > there is nothing capable of executing on this notification at
>>> the
>>> > > > > > moment.  Only Ceilometer listens to Keystone notifications.
>>> > > > > >
>>> > > > > > 2.  Keystone does not have a workflow engine, and should not be
>>> > > > > > auto-creating projects.  This is something that should be
>>> performed
>>> > > via
>>> > > > > > a Heat template, and Keystone does not know about Heat, nor
>>> should
>>> > > it.
>>> > &

Re: [openstack-dev] [keystone] [Mistral] [Heat] Autoprovisioning, per-user projects, and Federation

[Tim Hinrichs]

Excerpts from Wed, Nov 11, 2015 at 10:14 AM Clint Byrum 
wrote:

> > But as Renat mentioned, the part about triggering Mistral workflows from
> > a message does not yet exist. As Tim pointed out, Congress could be a
> > solution to that (listening for a message and then starting the Mistral
> > workflow). That may be OK in the short term, but in the long term I'd
> > prefer that we implement the triggering thing in Mistral (since there
> > are *lots* of end-user use cases for this too), and have the workflow
> > optionally query Congress for the policy rather than having Congress in
> > the loop.
> >
>
> I agree 100% on the positioning of Congress vs. Mistral here.
>
>
One problem that I'd imagine Mistral would want to solve if it's picking up
events off the bus and executing workflows is how the operator configures
the event-to-workflow mapping logic.  In Adam's example, the operator would
want to say that every time the 'new-user-login-event' shows up on the bus
that Mistral should kick off the 'create-quota' workflow and the
'create-role' workflow.  In simple cases, this would just be a dictionary,
but what happens if the operator wants to condition workflow execution on
an AND/OR/NOT expression evaluated over state from different projects (e.g.
run 'create-quota' when a new user logs in and that user doesn't already
have a nova quota).

For the operator, the problem becomes more complicated when multiple
OpenStack projects are listening to the bus and kicking off
workflows/scripts/etc. The operator now has N projects to configure
(possibly in different ways) and needs to feel confident that there's not
some (rare) sequence of events that puts OpenStack as a whole into a bad
state because the events/workflows she configured have opposing goals.

The benefit of Congress is that there's one rich, declarative language that
operators can use to control the event-to-workflow mapping.  The operator
dictates which events/states (drawn from any collection of OpenStack
projects) should cause which workflows/templates/APIs (again from any
OpenStack project) to be executed.  And because the mapping is written
declaratively, it's feasible to do some conflict detection.

I'm not arguing that Mistral can't or shouldn't be adapted as was
suggested.  I'm just articulating what Congress brings to the table.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] kilo bug on datasource listing

[Tim Hinrichs]

Congress stable-maintenance team:

We seem to have a bug in kilo that is making it tough for Bryan Sullivan to
get things up and running.  The swift driver doesn't have a 'secret' field,
which is causing a 500 error when listing datasources.  If I remember
right, we fixed this bug later.

https://bugs.launchpad.net/congress/+bug/1518496

Could someone volunteer to fix it?  I think we should do a release once
that's in.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] tempest tests in gate not running

[Tim Hinrichs]

It looks like our tempest tests aren't running in the gate.  Here's a
recent patch that just merged this morning.

https://review.openstack.org/#/c/242305/

If you go to the testr results, you'll see a list of all the tests that
were run.

http://logs.openstack.org/05/242305/5/gate/gate-congress-dsvm-api/979bc9d/testr_results.html.gz

Two things: (i) there are many tests here that are not Congress-specific
and (ii) I don't see the Congress tests running at all.  During manual
testing both these issues were handled by the following lines in our gate
job config.

https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/congress.yaml#L33-L34

The good news, at least, is that congress is getting properly spun up.

http://logs.openstack.org/05/242305/5/gate/gate-congress-dsvm-api/979bc9d/logs/screen-congress.txt.gz

Would someone want to volunteer to dig into this?

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress]role in Congress

[Tim Hinrichs]

Hi Yali,

We didn't discuss role assignment for service users in Tokyo.  Could you
explain a bit more what you need?

Tim

On Sun, Nov 29, 2015 at 7:33 PM zhangyali (D) 
wrote:

> Hi Tim and All,
>
>
>
> I remember there is a topic named “role assignment for service users” in
> the Tokyo Summit. Since I have not heard any message of this topic. Does
> anyone could contribute some information for me? I think it is vital for my
> design of Congress UI in horizon. Thanks a lot!
>
>
>
>
>
> Best Regards,
>
> Yali
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 答复: [congress]role in Congress

[Tim Hinrichs]

Hi Yali,

To support the kind of collaboration you describe, we have multiple
policies.  (By "policy" here I mean a collection of rules.)  One reasonable
workflow is to designate some policy (e.g. "library") to be where people
define predicates and complicated relations that are useful for writing
policy in general, and then have everyone else use the tables defined in
"library".  I'd imagine we could do the same kind of workflow in the UI.

By the way, your original patch never got merged (it's got a -1 on review
and a -1 on tests), so before looking into adding new features it'd be
great if we could finish off your original patch.  Let us know if you want
help.

Tim


On Mon, Nov 30, 2015 at 2:20 AM Masahito MUROI 
wrote:

> Hi Yali,
>
> I'm not sure which your concern is about UI by Horizon or for a policy
> in Congress.
>
> On 2015/11/30 16:27, zhangyali (D) wrote:
> > Hi Tim,
> >
> > In the implementation of Congress UI, I have realized that we need to
> > add role assignment. In many systems and cases, RBAC (Role-Based Access
> > Control) is an vital function which are beneficial to the division of
> work.
> >
> > I think we should have “admin” and “tenant” user at least.  The admin
> > user could define predicates or complicated relations used in the
> > violations, and tenant user could just use the predicates defined by
> > admin user.
> >
> > A typical example is that, the owner of network connected to VM should
> > be in the same group with the owner of this VM. In this example,
> > same_group is a predicates or complicated relations. It should be
> I think this depends on how admin writes policy and what service works
> as Congress datasource. If admin want to adopt the policy to specific
> tenant, user or group, admin writes an additional policy to affect them
> by the policy.
>
>
> > defined by admin user. And this predicate could be used by any tenant
> > user. In this way, some typical or common predicates could be defined in
> > a individual admin board, and other tenant user could choose which one
> > to use.
> On the other hand, this mentions about which user has permission to edit
> and to see a policy by Horizon.
>
> best regard,
> Masahito
>
> >
> > Yali
> >
> > *发件人:*Tim Hinrichs [mailto:t...@styra.com]
> > *发送时间:*2015年11月30日10:37
> > *收件人:*zhangyali (D); OpenStack Development Mailing List (not for
> > usage questions)
> > *主题:*Re: [openstack-dev][congress]role in Congress
> >
> > Hi Yali,
> >
> > We didn't discuss role assignment for service users in Tokyo.  Could you
> > explain a bit more what you need?
> >
> > Tim
> >
> > On Sun, Nov 29, 2015 at 7:33 PM zhangyali (D)  > <mailto:zhangyali...@huawei.com>> wrote:
> >
> > Hi Tim and All,
> >
> > I remember there is a topic named “role assignment for service
> > users”in the Tokyo Summit. Since I have not heard any message of
> > this topic. Does anyone could contribute some information for me? I
> > think it is vital for my design of Congress UI in horizon. Thanks a
> lot!
> >
> > Best Regards,
> >
> > Yali
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> 室井 雅仁(Masahito MUROI)
> Software Innovation Center, NTT
> Tel: +81-422-59-4539，FAX: +81-422-59-2699
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Mid-cycle meet-up

[Tim Hinrichs]

Hi all,

We're having a mid-cycle meet-up January 26-28 in the Bay Area.

Here's a tentative schedule.  Comments, suggestions, questions?

Tue Jan 26: Discussion with Monasca team about integration.
- Intro to Congress (slides or whiteboard)
- Intro to Monasca (slides or whiteboard)
- Design discussion
- Prototyping

Wed Jan 27: Congress team code sprint for distributed arch

Thu Jan 28: Congress team design session for high availability and high
throughput

More logistics details will follow.

I hope everyone can make it!

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] IRC broken

[Tim Hinrichs]

It seems IRC is broken for the meeting we're supposed to be having right
now.  The symptom is that you can logon but may only see a fragment of the
users/messages that are being sent.  We even had a partitioning where 2
people could exchange messages, and a different 2 people could exchange
messages, but not all 4 people could exchange messages.

Unless it fixes itself in the next few minutes, we'll cancel and have the
discussion over email.  And given how late we're starting, we'll likely
have an email discussion anyway.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] [infra] IRC broken

[Tim Hinrichs]

The issue with #openstack-meeting seems to have resolved itself about 35
minutes into our scheduled time.  The only thing that might be missing from
the logs are messages where people were trying to figure out what was going
on.  So if anyone missed the meeting, start at the 35 minute mark.

I included [infra] and a few more details below.

- It seemed there was a partitioning in #openstack-meeting for a while.  I
could #start the meeting, but except for 1 other person (masahito), no one
could see my messages.  I could exchange messages with masahito, but
neither of us could send messages to ekcs or ramineni.  I could, however,
see ekcs and ramineni's messages to each other.

- We had the same problem with #congress.  I eventually called ekcs, and we
tried to debug.

- I tried logging back in a couple of times, which eventually worked.  My
nick changed.  At that point, I could exchange messages with ekcs and
ramineni, and I could send messages to masahito, but I couldn't see
messages from masahito.

- Just as we were about to cancel the meeting, #congress started working
again.  Then #openstack-meeting started working, so we resumed the meeting.
 (I'm not sure #congress started working first, or if we just all happened
to notice #congress first.)

If anyone has any other pertinent details, please include them.

Tim

On Wed, Dec 9, 2015 at 5:07 PM Anita Kuno  wrote:

> On 12/09/2015 07:33 PM, Tim Hinrichs wrote:
> > It seems IRC is broken for the meeting we're supposed to be having right
> > now.  The symptom is that you can logon but may only see a fragment of
> the
> > users/messages that are being sent.  We even had a partitioning where 2
> > people could exchange messages, and a different 2 people could exchange
> > messages, but not all 4 people could exchange messages.
> >
> > Unless it fixes itself in the next few minutes, we'll cancel and have the
> > discussion over email.  And given how late we're starting, we'll likely
> > have an email discussion anyway.
> >
> > Tim
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> It seems you were able to hold your meeting?
>
> http://eavesdrop.openstack.org/meetings/congressteammeeting/2015/congressteammeeting.2015-12-10-00.00.log.html
>
> I don't see anything wrong with the #openstack-meeting channel.
>
> Do let infra know if you have additional details.
>
> Thank you,
> Anita.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] with vm-placement and the third party module

[Tim Hinrichs]

Hi Yue,

Not sure I understood your question exactly.  You can definitely build your
own datasource and have it do whatever you want.  The one I created had all
the vm-placement logic that was contained in
policy_engines/vmplacement.py.  And in the commit it shows how to invoke
the different functions on that datasource from the command line using curl.

BTW I added the openstack-dev mailing list to this thread.  It's good to
send this kind of note to openstack-dev and start the subject line with
[Congress] so the Congress team sees it.

Tim



On Thu, May 12, 2016 at 7:47 PM Yue Xin  wrote:

> Hi Tim and all,
>
> Thank you so much. It helps a lot.
> But I still have some confusion here. The vmplace driver trigger the vm
> placement by combine the driver and vm together。is there any way to have
> congress communicate with a separate one?(seperate driver and module)
>
> Thank you very much
>
> *Regards,*
> *Yue*
>
> On 11 May 2016 at 21:22, Tim Hinrichs  wrote:
>
>> Hi Yue,
>>
>> I got the vm-placement code added to a datasource that seems to spin up
>> and is working.  You can get the vm-placement datasource driver from here:
>>
>> https://review.openstack.org/#/c/315303/
>>
>> The commit message explains how to use it.  (You already know how to
>> configure Congress to use a new datasource driver.  So you do the same
>> thing with this new file.)
>>
>> Tim
>>
>> On Tue, May 10, 2016 at 6:50 AM Yue Xin  wrote:
>>
>>> Hi all,
>>>
>>> Thank you so much for your kindness feedback.
>>>
>>> I will try to put vm_placement.py in the same directory with
>>> others.(pervious I didn't find the location of other drivers, now I figured
>>> out they are in /opt/stack/congress/congress/datasources/).
>>>
>>> I am not quite sure about the congress version, I download the "Tokyo
>>> Hands On Lab virtual machine <https://goo.gl/o062Kc>" directly to work
>>> on virtualbox.
>>>
>>> My vm_placement_driver.py is quite simple right now(not sure about how
>>> it works). I have attached it below. I will work on it this week, if any
>>> progress, I will let you know.
>>>
>>> Thanks
>>>
>>>
>>> *Regards,*
>>> *Yue*
>>>
>>> On 9 May 2016 at 18:27, Tim Hinrichs  wrote:
>>>
>>>> Hi all,
>>>>
>>>> Yue wants to use the vm-placement module.  It's intended to be an
>>>> additional policy-engine, but maybe we can load it as if it were a new
>>>> datasource.  Anyone know if that would work?
>>>>
>>>> Yue: I CCed the rest of the team.  When you configure Congress, you are
>>>> giving it the path to the datasource:
>>>> congress.datasources.vm_placement_driver says to grab
>>>> congress/datasources/vm_placement_driver.py.  So put your file at that
>>>> location within the Congress repo.  (Make sure it's in the same directory
>>>> as all the other drivers.)  If that doesn't work, could you tell us which
>>>> version of Congress you are using (the Hands-on-lab from Vancouver?)?
>>>> Could you attach the vm_placement_driver.py file you are using?
>>>>
>>>> Tim
>>>>
>>>>
>>>>
>>>> On Mon, May 9, 2016 at 3:13 PM Yue Xin  wrote:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> Thank you very much for your reply.
>>>>>
>>>>>  I have written an demo vm-placement driver, and tried to put it into
>>>>> the demo. But I have no clue where to put it.
>>>>>
>>>>> I added an line in "devstack/lib/congress" to tell congress there is a
>>>>> new driver added
>>>>> [image: image.png]
>>>>>
>>>>> but when I use the command "opeopenstack congress datasource create
>>>>> --config username=admin --config tenant_name=admin  --config auth_url=
>>>>> http://127.0.0.1:5000/v2.0 --config password=password -ope-config
>>>>> poll_time=5 vm_placement vm_placement"
>>>>>
>>>>> the response is"*driver not found(Http 404)"*
>>>>>
>>>>> I guess it is because I didn't put the vm_placement_driver.py in the
>>>>> right place.
>>>>>
>>>>> Can you give me some suggestion where to put the added driver?
>>>>>
>>>>> Thank you very much
>>>>>
>>>>>

Re: [openstack-dev] [Congress] driver (authentication issue)

[Tim Hinrichs]

Hi Yue,

Added the mailing list to this one too.  Responses inline.

If you're comfortable with IRC, you can go to the #congress channel on
chat.freenode.net to ask question in real time.  Would likely be quicker
than debugging over email.


On Fri, May 13, 2016 at 2:03 PM Yue Xin  wrote:

>  Hi Tim,
>
> Thank you so much for your warm help previously.
> I have written a driver as a demo to create a table on congress. it
> succeed. But when I use the command to list the table, the error is:
> [image: image.png]
> "internal server error (http 500)"
>

That looks like a bug--you shouldn't see 500 errors.  We'd need more info
to help you avoid the problem: (i) the command you used to create the
'test' datasource and (ii) the datasource driver code that you created.


> [image: image.png]
>
> when I want to use curl to access localhost:1789 the error is
> authentication required. I tried to add -u congress:password, still get
> this error.
>
>
To use curl, the easiest thing to do is go into /etc/congress/congress.conf
and change the auth_strategy from keystone to noauth, and restart the
congress server.  Then Congress won't ask for authentication credentials.

auth_strategy = noauth

Tim


> My working environment is "Tokyo hands on lab virtual machine"
>
> Thank you so much for your help.
>
> *Regards,*
> *Yue*
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Nominating Anusha Ramineni and Eric Kao for core reviewer

[Tim Hinrichs]

Hi all,

I'm writing to nominate Anusha Ramineni and Eric Kao as Congress core
reviewers.  Both Anusha and Eric have been active and consistent
contributors in terms of code, reviewing, and interacting with the
community since September--for all of Mitaka and a few months before that.

Anusha was so active in Mitaka that she committed more code than the other
core reviewers, and wrote the 2nd most reviews over all.  She took on
stable-maintenance, is the first person to fix gate breakages, and manages
to keep Congress synchronized with the rest of the OpenStack projects we
depend on.  She's taken on numerous tasks in migrating to our new
distributed architecture, especially around the API.  She manages to write
honest yet kind reviews, and has discussions at the same level as the rest
of the cores.

Eric also committed more code in Mitaka than the other core reviewers.  He
has demonstrated his ability to design and implement solutions and work
well with the community through the review process.  In particular, he
completed the Congress migration to Python3 (including dealing with the
antlr grammar), worked through difficult problems with the new distributed
architecture (e.g. message sequencing, test-nondeterminism), and is now
designing an HA deployment architecture.  His reviews and responses are
both thoughtful and thorough and engages in discussions at the same level
as the rest of the core team.

Anusha and Eric: it's great working with you both!

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] driver (authentication issue)

[Tim Hinrichs]

When you're doing testing like this, the easiest solution is to disable
authentication.  Here are the instructions from the previous email:

To use curl, the easiest thing to do is go into /etc/congress/congress.conf
and change the auth_strategy from keystone to noauth, and restart the
congress server.  Then Congress won't ask for authentication credentials.

auth_strategy = noauth

Tim

On Tue, May 17, 2016 at 8:36 PM Yue Xin  wrote:

> Hi Tim and all,
>
> Thank you very much. I put congress in a tag so I missed your email and
> relay late. Sorry about that.
>
> My problem here is that i want to use the command line to put the data
> into the congress datasource.
> I use the command openstack congress datasource create test test(i have a
> test_driver), and successed in creating a table.
> Then I want to check the table use the command(openstack congress
> datasource list table test) the errer is (internal server error 501).
> Then I try to push the data to the test table, use
>
> curl -g -i -X PUThttp://localhost:1789/v1/data-sources/id/tables/ 
> 
>
> the response is "authentication required" which means I can't push data into 
> congress datasource. I have no idea how to fix it. Can you give me some hints?
>
> Thank you very much.
>
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] update congress

[Tim Hinrichs]

Hi Yue,

That version of Congress definitely doesn't have the Push driver.  The Push
driver code was implemented only in the latest release (Mitaka).

Here are the upgrade instructions.  They SHOULD work, but let us know if
you run into problems, both so we can help you and so we can correct the
instructions.

https://review.openstack.org/#/c/320652/2/README.rst

Tim

On Tue, May 24, 2016 at 3:40 AM Yue Xin  wrote:

> Hi Tim and all,
>
> May I ask how to update the congress version with the Tokyo Hands on Lab
> environment? cause I use the command ＂openstack congress list version＂ it
> show the congress is 2013's version.
>
> The reason why I want to update it is that I write a demo driver, which I
> want to push data into the datasource table, but when I use 'curl -i -g -X
> PUT ' command , the error is "501 not implemented" , same error comes with
> 'curl -X PUSH', I am not sure whether it comes from the version of congress
> or not(I thought maybe congress is too old so it doesn't support push data).
>
> Thank you very much for your response.
>
> *Regards,*
> *Yue*
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress] Spec for congress.conf

[Tim Hinrichs]

We should add a section to our docs that details the config option names,
their descriptions, and which ones are required.  We should backport that
to mitaka and maybe liberty.

Tim

On Mon, May 30, 2016 at 12:49 AM Masahito MUROI <
muroi.masah...@lab.ntt.co.jp> wrote:

> Hi Bryan,
>
>
> On 2016/05/28 2:52, Bryan Sullivan wrote:
> > Masahito,
> >
> > Sorry, I'm not quite clear on the guidance. Sounds like you're saying
> > all options will be defaulted by Oslo.config if not set in the
> > congress.conf file. That's OK, if I understood.
> you're right.
>
> >
> > It's clear to me that some will be deployment-specific.
> >
> > But what I am asking is where is the spec for:
> > - what congress.conf fields are supported i.e. defined for possible
> > setting in a release
> Your generated congress.conf has a list of all supported config fields.
>
> > - which fields are mandatory to be set (or Congress will simply not work)
> > - which fields are not mandatory, but must be set for some specific
> > purpose, which right now is unclear
> Without deployment-specific configs, IIRC what you need to change from
> default only is "drivers" fields to run Congress with default setting.
>
> >
> > I'm hoping the answer isn't "go look at the code"! That won't work for
> > end-users, who are looking to use Congress but not decipher the
> > meaning/importance of specific fields from the code.
> I guess your generated config has the purpose of each config fields.
>
> If you expect the spec means documents like [1], unfortunately Congress
> doesn't have these kind of document now.
>
> [1] http://docs.openstack.org/mitaka/config-reference/
>
> best regards,
> Masahito
>
> >
> > Thanks,
> > Bryan Sullivan
> >
> >> From: muroi.masah...@lab.ntt.co.jp
> >> Date: Fri, 27 May 2016 15:40:31 +0900
> >> To: openstack-dev@lists.openstack.org
> >> Subject: Re: [openstack-dev] [congress] Spec for congress.conf
> >>
> >> Hi Bryan,
> >>
> >> Oslo.config that Congress uses to manage config sets each fields to
> >> default value if you don't specify your configured values in
> >> congress.conf. In that meaning, all config is option/required.
> >>
> >> In my experience, config values differing from each deployment, like ip
> >> address and so on, have to be configured, but others might be configured
> >> when you want Congress to run with different behaviors.
> >>
> >> best regard,
> >> Masahito
> >>
> >> On 2016/05/27 3:36, SULLIVAN, BRYAN L wrote:
> >> > Hi Congress team,
> >> >
> >> >
> >> >
> >> > Quick question for anyone. Is there a spec for fields in congress.conf
> >> > file? As of Liberty this has to be tox-generated but I need to know
> >> > which conf values are required vs optional. The generated sample
> output
> >> > doesn't clarify that. This is for the Puppet Module and JuJu Charm I
> am
> >> > developing with the help of RedHat and Canonical in OPNFV. I should
> have
> >> > Congress installed by default (for the RDO and JuJu installers) in the
> >> > OPNFV Colorado release in the next couple of weeks, and the
> >> > congress.conf file settings are an open question. The Puppet module
> will
> >> > also be used to create a Fuel plugin for installation.
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Bryan Sullivan | AT&T
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >
> __
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >>
> >> --
> >> 室井 雅仁(Masahito MUROI)
> >> Software Innovation Center, NTT
> >> Tel: +81-422-59-4539
> >>
> >>
> >>
> >>
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> 室井 雅仁(Masahito MUROI)
> Software Innovation Center, NTT
> Tel: +81-422-59-4539
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/opensta

Re: [openstack-dev] [congress] Spec for congress.conf

[Tim Hinrichs]

Bryan,

I spent some time looking into the proper way to document configuration
options in OpenStack.  There's a configuration reference that's part of the
openstack-manuals project.  It'll take some time to figure out the best way
of contributing to that.

But for now I added a section to our docs detailing the most important
options for Congress.  All the rest of the config options are common to
most OpenStack projects.  We should document those too, but for the time
being I think we've got the most important part in place.

Here's the change in review, if you want to comment.
https://review.openstack.org/#/c/324732/

If you'd rather just have the Congress-specific config options, here's the
text from that change.
As Masahito said, everything has a default value, so all of these are
optional for you to include in the configuration file.  In practice,
though, you should always provide the ``drivers`` option, since otherwise
you won't be able to create any datasources.

Quick request: Could you give us feedback around whether an empty default
value for ``drivers`` makes sense, or whether it would be (much) better for
the default to be all of the drivers that exist in Congress?



The options most important to Congress are described below, all of which
appear under the [DEFAULT] section of the configuration file.

``drivers``
The list of permitted datasource drivers.  Default is the empty list.
The list is a comma separated list of Python class paths. For example:
drivers =
congress.datasources.neutronv2_driver.NeutronV2Driver,congress.datasources.glancev2_driver.GlanceV2Driver

``datasource_sync_period``
The number of seconds to wait between synchronizing datasource config
from the database.  Default is 0.

``enable_execute_action``
Whether or not congress will execute actions.  If false, Congress will
never execute any actions to do manual reactive enforcement, even if
there
are policy statements that say actions should be executed and the
conditions of those actions become true.  Default is True.

One of Congress's new experimental features is distributing its services
across multiple services and even hosts.  Here are the options for using
that feature.

``distributed_architecture``
Whether to enable the distributed architecture.  Don't set it to true in
before Newton release since the new architecture is still under
development as of Newton.  Default is false.

``node_id``
Unique ID of this Congress instance.  Can be any string.  Useful if
you want to create multiple, distributed instances of Congress.

Here are the most often-used, but standard OpenStack options.  These
are specified in the [DEFAULT] section of the configuration file.

``auth_strategy``
Method for authenticating Congress users.
Can be assigned to either 'keystone' meaning that the user must provide
Keystone credentials or to 'noauth' meaning that no authentication is
required.  Default is 'keystone'.

``verbose``
Controls whether the INFO-level of logging is enabled.  If false,
logging
level will be set to WARNING.  Default is false.  Deprecated.

``debug``
Whether or not the DEBUG-level of logging is enabled. Default is false.




Tim

On Tue, May 31, 2016 at 10:18 AM Tim Hinrichs  wrote:

> We should add a section to our docs that details the config option names,
> their descriptions, and which ones are required.  We should backport that
> to mitaka and maybe liberty.
>
> Tim
>
> On Mon, May 30, 2016 at 12:49 AM Masahito MUROI <
> muroi.masah...@lab.ntt.co.jp> wrote:
>
>> Hi Bryan,
>>
>>
>> On 2016/05/28 2:52, Bryan Sullivan wrote:
>> > Masahito,
>> >
>> > Sorry, I'm not quite clear on the guidance. Sounds like you're saying
>> > all options will be defaulted by Oslo.config if not set in the
>> > congress.conf file. That's OK, if I understood.
>> you're right.
>>
>> >
>> > It's clear to me that some will be deployment-specific.
>> >
>> > But what I am asking is where is the spec for:
>> > - what congress.conf fields are supported i.e. defined for possible
>> > setting in a release
>> Your generated congress.conf has a list of all supported config fields.
>>
>> > - which fields are mandatory to be set (or Congress will simply not
>> work)
>> > - which fields are not mandatory, but must be set for some specific
>> > purpose, which right now is unclear
>> Without deployment-specific configs, IIRC what you need to change from
>> default only is "drivers" fields to run Congress with default setting.
>>
>> >
>> > I'm hoping the answer isn't "go look at the code"! That w

Re: [openstack-dev] [puppet] [Congress] Request to create puppet-congress

[Tim Hinrichs]

Hi Dan,

As far as the Congress team is concerned, that'd be great!  Let us know how
we can help.

Tim

On Tue, Jun 7, 2016 at 12:54 PM Dan Radez  wrote:

> I'd like to get puppet-congress started.
>
> I've written some code based on the cookie cutter structure but I've not
> gone through proper channels yet to get it into openstack-puppet.
>
> I'd like to get the project establish so that the code can be run
> through the review process.
>
> Dan Radez
> freenode: radez
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress][release] missing build artifacts

[Tim Hinrichs]

Hi Doug,

Thanks for letting us know.  Here's what we're intending...
- We'd like to release the server code for Mitaka.
- We release the client to Pypi, so that's already taken care of.
- We haven't moved our docs off of readthedocs yet, so we're taking care of
that as well.

I gave a +1 to your patch for adding openstack-server-release-jobs to zuul.
I also pushed a congress-only patch for the same, thinking that's probably
what you actually wanted us to do.
https://review.openstack.org/#/c/300682/

I gave a -1 to your patch that removes all the Congress deliverables from
openstack/releases, thinking that we can have this sorted out quickly.  The
job we're missing just produces a tarball and uploads it, right?  We've
been manually doing releases up to this point, which is why we didn't have
the job in place.
https://review.openstack.org/300644

I didn't touch your change on the artifact-link-mode, since it seems like a
short-term solution that may go in before we get everything sorted.
https://review.openstack.org/300457

Tim

On Fri, Apr 1, 2016 at 1:23 PM Doug Hellmann  wrote:

> Congress team,
>
> We noticed in our audit of the links on
> http://releases.openstack.org/mitaka/index.html that the links to
> the build artifacts for congress point to missing files. The
> repository doesn't seem to have any real build jobs configured in
> openstack-infra/project-config/zuul/layout.yaml, so it's not clear
> how tagging is producing a release for you.
>
> For now, we are disabling links to the artifacts for those repos
> via https://review.openstack.org/300457 but we're also planning to
> remove them from the official Mitaka page since there don't
> appear to be any actual related deliverables
> (https://review.openstack.org/300644).
>
> It would be good to understand what your intent is for builds. Can
> you follow up here on this thread with some details?
>
> Thanks,
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress][release] missing build artifacts

[Tim Hinrichs]

Looks like the congress job is working now.  We have a tarball for our
latest RC:

http://tarballs.openstack.org/congress/congress-3.0.0.0rc3.tar.gz

Thierry added the links back for the tarball.

https://github.com/openstack/releases/commit/beee35379b3b52ed7d444d93d7afd8b6603f69b6

So it looks like everything is back on track for the Congress Mitaka
release.  Let me know if there's anything else I need to do.

Thanks for all the help release team!
Tim


On Sat, Apr 2, 2016 at 5:39 AM Doug Hellmann  wrote:

> Excerpts from Tim Hinrichs's message of 2016-04-01 22:46:59 +:
> > Hi Doug,
> >
> > Thanks for letting us know.  Here's what we're intending...
> > - We'd like to release the server code for Mitaka.
> > - We release the client to Pypi, so that's already taken care of.
> > - We haven't moved our docs off of readthedocs yet, so we're taking care
> of
> > that as well.
> >
> > I gave a +1 to your patch for adding openstack-server-release-jobs to
> zuul.
> > I also pushed a congress-only patch for the same, thinking that's
> probably
> > what you actually wanted us to do.
> > https://review.openstack.org/#/c/300682/
>
> Now that the jobs are in place, we should tag new release candidates
> using new versions and the same SHAs as the last candidate to get the
> artifacts built properly. You can submit the request for that using the
> releases repo, or you can tag yourself and submit the info after the
> fact. We will decide what to do about the old artifacts you have on
> launchpad separately.
>
> >
> > I gave a -1 to your patch that removes all the Congress deliverables from
> > openstack/releases, thinking that we can have this sorted out quickly.
> The
> > job we're missing just produces a tarball and uploads it, right?  We've
> > been manually doing releases up to this point, which is why we didn't
> have
> > the job in place.
> > https://review.openstack.org/300644
> >
> > I didn't touch your change on the artifact-link-mode, since it seems
> like a
> > short-term solution that may go in before we get everything sorted.
> > https://review.openstack.org/300457
>
> Yes, we'll merge that as a short-term fix until we have the rest of it
> worked out.
>
> Doug
>
> >
> > Tim
> >
> > On Fri, Apr 1, 2016 at 1:23 PM Doug Hellmann 
> wrote:
> >
> > > Congress team,
> > >
> > > We noticed in our audit of the links on
> > > http://releases.openstack.org/mitaka/index.html that the links to
> > > the build artifacts for congress point to missing files. The
> > > repository doesn't seem to have any real build jobs configured in
> > > openstack-infra/project-config/zuul/layout.yaml, so it's not clear
> > > how tagging is producing a release for you.
> > >
> > > For now, we are disabling links to the artifacts for those repos
> > > via https://review.openstack.org/300457 but we're also planning to
> > > remove them from the official Mitaka page since there don't
> > > appear to be any actual related deliverables
> > > (https://review.openstack.org/300644).
> > >
> > > It would be good to understand what your intent is for builds. Can
> > > you follow up here on this thread with some details?
> > >
> > > Thanks,
> > > Doug
> > >
> > >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress]Authorization mechanisms for each user

[Tim Hinrichs]

Hi Yuki,

As Masahito mentioned, the usual way to authorize API calls in OpenStack is
through policy.json.  If I remember right, you can make a decision about
whether an API call is permitted using (i) all the values in the API call
and (ii) the Keystone role of the user making the request.  I'm not sure if
you can also use the actual userID of the person making the request or not,
but as soon as you get to 10 users, you'll end up grouping individual users
into roles anyway.

That said, Congress would be valuable for authorization decisions IF those
decisions require information other than (i) values in the API call and
(ii) the role of the user making the request.  For example, if you wanted
to stop someone from deleting a Neutron network whenever there is a Nova VM
attached to that network with status ACTIVE, then policy.json isn't
adequate, and Congress makes sense.

So as Masahito mentioned, if you provide more details about your use case
(in particular what kinds of information you need for making authorization
decisions), we can help you pick the right tool.

Tim




On Fri, Apr 15, 2016 at 1:06 AM Masahito MUROI 
wrote:

> Hi Yuki,
>
> This sounds interesting. AFAIK, there is no similar use-case you mentioned.
>
> On 2016/04/15 10:13, Yuki Nisiwaki wrote:
> > Hi openstacker working on congress.
> >
> > I want to implement the authorization mechanisms for each user, not role
> > base.
> > For example, User A can change security group, But User B can’t change
> > security group like IAM feature of AWS.
> >
> > In order to achieve it,
> > I’m considering whether can I utilize Congress feature.
> > I am thinking somehow that I can achieve it by following step.
> > 1. create policy for each user with datalog in congress
> > 2. prepare the wsgi filter for each project that works confirming
> > authorization of each user to Congress.
> Could you clarify your usecase? I think it can be done by roles and
> modifying policy.json. If you assume A and B are under some conditions,
> what kind of condition do you want to use?
>
> btw, I added [Congress] prefix in the subject.
>
> >
> > I think this use-case is very popular and there is someone who think
> > same thing.
> > But There is no information about it in any website (blog, presentation
> > in summit).
> > So why is there anyone who achieve it?
> > or does this approach have anxious point?
> > If you are interested in this approach or think same thing, I want to
> > know it.
> >
> >
> > Best regards
> >
> > Yuki Nishiwaki
> > NTT Communitions
> > Technology development
> > Cloud Core Technology Unit
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> best regards,
> Masahito
>
>
> --
> 室井 雅仁(Masahito MUROI)
> Software Innovation Center, NTT
> Tel: +81-422-59-4539
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress]Authorization mechanisms for each user

[Tim Hinrichs]

Hi Yuki,

That description was very helpful.  In short, policy.json doesn't work
because the person setting policy is not permitted to change policy.json
(which happens in part because policy.json has no API for controlling it).

In that case, using Congress makes sense.  I know someone was doing what
you are proposing to implement a sophisticated notion of quotas for Nova.
As far as I know, Congress is the best option for your use case today
 (though there's been talk of a more dynamic policy.json in the past).  You
can find the plugin for Nova in the repo.

https://github.com/openstack/congress/tree/master/contrib/nova

One thing to think about is what kind of performance and availability needs
you have.  Making Congress say Yes or No to every API call puts it on the
critical path.  How many requests per second do you expect?  If each one of
those requests needs to talk to Congress, is the latency/throughput of a
round-trip to another service going to be a problem?  What happens if the
network is partitioned and the other services cannot reach Congress for
some period of time?  Can you deny or allow all access until the network
reconnects?

The Congress team is getting close to finishing a new architecture aimed at
high-availability and high-throughput, so we'd love to hear more details
about what you think you need.

Also, longer term we could look into more exotic options, such as the cloud
admin managing policy via Congress, and Congress distributing that policy
to the services that need it.  Think of this as a mini-Congress running on
each service.  This way, instead of each service contacting Congress on
every API call, the service could make the Yes/No decision itself, and
Congress would update each mini-Congress as needed.  That would give you
low-latency and high-availability, even in the presence of network
disconnections.

Tim


On Sat, Apr 16, 2016 at 9:41 AM Yuki Nisiwaki  wrote:

> Hi Masahito, Tim.
>
> Thanks for your messages.
>
> >> btw, I added [Congress] prefix in the subject.
> Firstly thanks for your point, I'm beginner openstacker. So this
> information is very useful.
>
> Move to main topic.
>
> >So as Masahito mentioned, if you provide more details about your use case
> >(in particular what kinds of information you need for making authorization
> >decisions),
>
> I supposed that we use OpenStack as Public Cloud which have multiple
> tenants.
> So The cloud administrator on OpenStack who have the admin role in
> OpenStack
> and The deployer who is in charge of creating OpenStack environment and
> providing service are not same.
> So we desire some mechanism to enable the admin on OpenStack in tenant (or
> domain) to specify which service(or function) can the user to use.
>
> >the usual way to authorize API calls in OpenStack is
> >through policy.json.  If I remember right, you can make a decision about
> >whether an API call is permitted using (i) all the values in the API call
> >and (ii) the Keystone role of the user making the request.  I'm not sure
> if
> >you can also use the actual userID of the person making the request or
> not,
> >but as soon as you get to 10 users, you'll end up grouping individual
> users
> >into roles anyway.
>
> Surely, The policy.json have potential to cover above our needs.
> But as I mentioned before, the deployer and administrator of OpenStack
> isn't same.
> And the deployer want to restrict the admin from login to the server which
> start up api processes.
> So the admin on OpenStack can't login to the servers of working api
> process.
>
> But it can't definitely realized by policy.json.
> If we prepare the api server that works changing policy.json  and  provide
> the users.
> But I think this solution isn't very good.
> There are many reason of making this solution bad.
> first reason is to make the rule of policy.json complex.
> second reason is that policy file is distributed in many server and many
> directory.
>
>
> The reasons I'm interested in congress are followings.
>  - Congress can centrally manage policy rule.
>  - Congress can change policy or add policy via API.
> So honestly I don't concern about dynamical policy (e.g. depending on the
> policy, which network is connected to) in our use-case.
>
>
> if you'll allow a slight digression,
> If we integrate authorization mechanism to actual environment,
> we will prepare the translator from authorization specific policy to
> datalog and
> we don't expose the congress in order to improve usability in the user
> perspective.
>
> Can this mail give the image of what I want to do to you?
>
> Yuki
> 
> >Hi Yuki,
> >
> >As Masahito mentioned, the usual way to authorize API calls in OpenStack
> is
> >through policy.json.  If I remember right, you can make a decision about
> >whether an API call is permitted using (i) all the values in the API call
> >and (ii) the Keystone role of the user making the request.  I'm not sure
> if
> >you can also use the actual userID of th

Re: [openstack-dev] [Congress]Authorization mechanisms for each user

[Tim Hinrichs]

Hi Yuki,

Sounds good.  The Congress team will be at Austin next week at the summit.
Most of the team will be at most of the events that you find by searching
for "Congress".  Let us know if you want to meet face-to-face to discuss
further.

https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=congress

Tim


On Mon, Apr 18, 2016 at 6:31 PM Yuki Nisiwaki  wrote:

> Hi Tim,
>
> Thanks.
> Your information is very useful for me.
>
> >I know someone was doing what you are proposing to implement a
> sophisticated notion of quotas for Nova
> I'll read later and make use of it as sample program.
>
>
> >though there's been talk of a more dynamic policy.json in the past
> I didn't know there's been talk of a more dynamic policy.json.
> So I tried to search "dinamic policy.json" and
> I found the keystone blueprint (
> https://blueprints.launchpad.net/keystone/+spec/dynamic-policies-delivery
> ).
> And this dynamical policy blueprint might  adopt the pluggable policy
> backend.
> So I thought we can consider to use congress as this backend.
>
> I'll consider how to implement authorization system with above information
> as well.
>
>
> >One thing to think about is what kind of performance and availability
> needs you have.  Making Congress say Yes or No to every API call puts it on
> the critical path.
> Your point is very useful for me.
> Surely, I have to consider the performance problem.
>
>
> > longer term we could look into more exotic options, such as the cloud
> admin managing policy via Congress, and Congress distributing that policy
> to the services that need it.  Think of this as a mini-Congress running on
> each service.  This way, instead of each service contacting Congress on
> every API call, the service could make the Yes/No decision itself.
> This future plan is very interesting. I think this architecture is also
> useful other than this time use-case.
>
>
> Thanks your information,
> and I'll put these information in order by myself, and consider how to
> achieve it.
> I'm interested in Congress project, so I'll watch the activity of Congress.
>
> Best regards.
>
> Yuki
>
>
>
> 2016-04-19 7:35 GMT+09:00 Tim Hinrichs :
>
>> Hi Yuki,
>>
>> That description was very helpful.  In short, policy.json doesn't work
>> because the person setting policy is not permitted to change policy.json
>> (which happens in part because policy.json has no API for controlling it).
>>
>> In that case, using Congress makes sense.  I know someone was doing what
>> you are proposing to implement a sophisticated notion of quotas for Nova.
>> As far as I know, Congress is the best option for your use case today
>>  (though there's been talk of a more dynamic policy.json in the past).  You
>> can find the plugin for Nova in the repo.
>>
>> https://github.com/openstack/congress/tree/master/contrib/nova
>>
>> One thing to think about is what kind of performance and availability
>> needs you have.  Making Congress say Yes or No to every API call puts it on
>> the critical path.  How many requests per second do you expect?  If each
>> one of those requests needs to talk to Congress, is the latency/throughput
>> of a round-trip to another service going to be a problem?  What happens if
>> the network is partitioned and the other services cannot reach Congress for
>> some period of time?  Can you deny or allow all access until the network
>> reconnects?
>>
>> The Congress team is getting close to finishing a new architecture aimed
>> at high-availability and high-throughput, so we'd love to hear more details
>> about what you think you need.
>>
>> Also, longer term we could look into more exotic options, such as the
>> cloud admin managing policy via Congress, and Congress distributing that
>> policy to the services that need it.  Think of this as a mini-Congress
>> running on each service.  This way, instead of each service contacting
>> Congress on every API call, the service could make the Yes/No decision
>> itself, and Congress would update each mini-Congress as needed.  That would
>> give you low-latency and high-availability, even in the presence of network
>> disconnections.
>>
>> Tim
>>
>>
>> On Sat, Apr 16, 2016 at 9:41 AM Yuki Nisiwaki 
>> wrote:
>>
>>> Hi Masahito, Tim.
>>>
>>> Thanks for your messages.
>>>
>>> >> btw, I added [Congress] prefix in the subject.
>>> Firstly thanks for your point, I'm beginner openstacker. So this
>>> info

[openstack-dev] [Congress] No meeting on April 27

[Tim Hinrichs]

No meeting for next week (April 27), since we'll be in Austin at the
summit.  We will send out an update about what happened during the summit
next week.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Austin recap

[Tim Hinrichs]

Hi all,

Here’s a quick summary of the Congress activities in Austin.  Everyone
should feel free to chime in with corrections and things I missed.

1. Talks

Masahito gave a talk on applying Congress for fault recovery in the context
of NFV.

https://www.openstack.org/summit/austin-2016/summit-schedule/events/7199

Fabio gave a talk on applying Congress + Monasca to enforce
application-level SLAs.

https://www.openstack.org/summit/austin-2016/summit-schedule/events/7363

2. Integrations

We had discussions, both within the Congress Integrations fishbowl session,
and outside of that session on potential integrations with other OpenStack
projects.  Here's a quick overview.

- Monasca (fabiog). The proposed integration: Monasca pushes data to
Congress using the push driver to let Congress know about the alarms
Monasca configured.  Can use multiple alarms using a single table.
Eventually we talked about having Congress analyze the policy to configure
the alarms that Monasca uses, completing the loop.

- Watcher (acabot). Watcher aims to optimize the placement of VMs by
pulling data from Ceilometer/Monasca and Nova (including
affinity/anti-affinity info), computing necessary migrations for whichever
strategy is configured, and migrates the VMs.  Want to use Congress as a
source of policies that they take into account when computing the necessary
migrations.

- Nova scheduler.  There’s interest in policy-enabling the Nova scheduler,
and then integrating that with Congress in the context of delegation, both
to give Congress the ability to pull in the scheduling policy and to push
the scheduling policy.

- Mistral.  The use case for this integration is to help people create an
HA solution for VMs.  So have Congress monitor VMs, identify when they have
failed, and kick off a Mistral workflow to resurrect them.

- Vintrage.  Vintrage does root-cause-analysis.  It provides a graph-based
model for the structure of the datacenter (switches attached to
hypervisors, servers attached to hypervisors, etc.) and a templating
language for defining how to create new alarms from existing alarms.  The
action item that we left is that the Vintrage team will initiate a mailing
list thread where we discuss which Vintrage data might be valuable for
Congress policies.

3. Working sessions

- The new distributed architecture is nearing completion.  There seems to
be 1 blocker for having the basic functionality ready to test: at boot,
Congress doesn’t properly spin up datasources that have already been
configured in the database.  As an experiment to see how close we were to
completion, we started up the Congress server with just the API and policy
engine and saw the basics actually working!  When we added the datasources,
we found a bug where the API was assuming the datasources could be
referenced by UUID, when in fact they can only be referenced by Name on the
message-bus.   So while there’s still quite a bit to do, we’re getting
close to having all the basics working.

- We made progress on the high-availability and high-throughput design.
This is still very much open to design and discussion, so continuing the
design on the mailing list would be great.  Here are the highlights.

   o  Policy engine: split into (i) active-active for queries to deal with
high-throughput (ii) active-passive for action-execution (requiring
leader-election, etc.).  Policy CRUD modifies DB; undecided whether API
also informs all policy-engines, or whether they all sync from the DB.

   o  Pull datasources: no obvious need for replication, since they restart
really fast and will just re-pull the latest data anyhow

   o  Push datasources: Need HA for ensuring the pusher can always push,
e.g. the pusher drops the message onto oslo-messaging.  Still up for debate
is whether we also need HA for storing the data since there is no way to
ask for it after a restart; one suggestion is that every datasource must
allow us to ask for the state.  HT does not require replication, since
syncing the state between several instances would be required and would be
less performant than a single instance.

   o  API (didn’t really discuss this, so here’s my take).  No obvious need
for replication for HT, since if the API is a bottleneck, the backend will
be an even bigger bottleneck.  For HA, could do active-active since the API
is just a front-end to the message bus + database, though we would need to
look into locking now that there is no GIL.

It was great seeing everyone in Austin!
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [vitrage] [congress] Vitrage-Congress Collaboration

[Tim Hinrichs]

Hi Alexey,

Thanks for the overview of how you see a Congress-Vitrage integration being
valuable.

I'd imagine that the right first step in this integration would be creating
a new datasource driver within Congress to pull data from Vitrage.  It
doesn't need to pull all the data in your list to start, but enough so that
we can try writing policy over that data.  It's helpful to have a policy in
mind that you want to write and then set up the datasource driver to grab
enough of the Vitrage data to write that policy.  Here are the relevant
docs.

Datasource drivers
http://docs.openstack.org/developer/congress/cloudservices.html

Writing policy
http://docs.openstack.org/developer/congress/policy.html

Let me know if you have any questions,
Tim



On Wed, May 4, 2016 at 11:51 PM Weyl, Alexey (Nokia - IL) <
alexey.w...@nokia.com> wrote:

> Hi to all Vitrage and Congress contributors,
>
> We had a good introduction meeting in Austin and we (Vitrage) think that
> we can have a good collaboration between the projects.
>
> Vitrage, as an Openstack Root Cause Analysis (RCA) Engine, builds a
> topology graph of all the entities in the system (physical, virtual and
> application) from different datasources. It thus can enrich Congress by
> providing more data about what is happening in the system. Additionally,
> the Vitrage RCA and deduce alarms & states mechanism can enhance the
> visibility of faults and how they inter-relate.  By using this information
> Congress could then execute different policies and perform more accurate
> actions.
>
> Another good property of Vitrage is that it can receive data also from
> non-openstack sources, like Nagios, which monitor the physical resources,
> including Switches (which are not modeled today in OpenStack).
>
> There are many ways in which Congress-Vitrage combination would be
> helpful. To take just one example:
> a. If a physical Switch is down, Vitrage can raise deduced alarms on the
> connected hosts and on the virtual machines affected by this change in
> switch state.
> b. Congress will then be notified by Vitrage about these alarms, which can
> set off Congress policies of migration.
> c. Furthermore, due to the RCA functionality, Congress will be aware that
> the Switch error is the source of the problem, and can determine the best
> place to create new instances of the VMs so that this  switch fault will
> not impact the new instances.
>
> As you can see, for each fault, we can use Vitrage to link it to other
> faults, and create alarms to reflect them. This is all done via Vitrage
> Templates, so the system is configurable to the needs of the user. Thus
> many more cases such as the example above could be thought of.
>
> To summarize, Vitrage can enrich Congress with the following four features:
> a. RCA
> b. Deduced alarms
> c. Physical, virtual and application layers
> d. Graph structure and topology of the system that defines the connections
> and relationships between all entities on which we can run quick graph
> algorithms to decide different actions to perform
>
> If you can think of additional use cases that can be used here, please
> share ☺
>
> For more data about Vitrage and its insights please take a look here:
> https://wiki.openstack.org/wiki/Vitrage
>
> Best Regards,
> Alexey Weyl
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [vitrage] [congress] Vitrage-Congress Collaboration

[Tim Hinrichs]

uestion.
> >>>
> >>> On 2016/05/10 0:17, Weyl, Alexey (Nokia - IL) wrote:
> >>>> Hi Tim,
> >>>>
> >>>> I agree – creating a datasource from Vitrage to Congress is the
> >>>> first step, and we should have some concrete use case in mind to
> >>>> help guide this process.
> >>>>
> >>>> The most straightforward use case I would suggest is when there is
> >> a
> >>>> problem on an instance that is caused by some problem on the
> >>>> physical host. Then:
> >>>>
> >>>> ·Vitrage will notify about an alarm on the instance, which Congress
> >>>> will receive
> >>>>
> >>> Why does Congress need to receive the alarm?  DataSouce Driver pulls
> >>> data from Vitrage, so it looks like Congress should only pull the
> >>> cause of the failure from Vitrage.
> >>>
> >>> Best regards,
> >>> Masahito
> >>>
> >>>> ·Congress can then call the Vitrage RCA API. The response will
> >> state
> >>>> that the cause of the instance alarm is the host alarm.
> >>>>
> >>>> ·Congress policy can define that in such a case, the instance
> >> should
> >>>> be migrated to (or healed on) a different physical host
> >>>>
> >>>> Does this seem like a good first step for you?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Alexey
> >>>>
> >>>> *From:*Tim Hinrichs [mailto:t...@styra.com]
> >>>> *Sent:* Saturday, May 07, 2016 2:43 AM
> >>>> *To:* OpenStack Development Mailing List (not for usage questions)
> >>>> *Subject:* Re: [openstack-dev] [vitrage] [congress] Vitrage-
> >> Congress
> >>>> Collaboration
> >>>>
> >>>> Hi Alexey,
> >>>>
> >>>> Thanks for the overview of how you see a Congress-Vitrage
> >>>> integration being valuable.
> >>>>
> >>>> I'd imagine that the right first step in this integration would be
> >>>> creating a new datasource driver within Congress to pull data from
> >>>> Vitrage.  It doesn't need to pull all the data in your list to
> >>>> start, but enough so that we can try writing policy over that data.
> >>>> It's helpful to have a policy in mind that you want to write and
> >>>> then set up the datasource driver to grab enough of the Vitrage
> >> data
> >>>> to write that policy.  Here are the relevant docs.
> >>>>
> >>>> Datasource drivers
> >>>>
> >>>> http://docs.openstack.org/developer/congress/cloudservices.html
> >>>>
> >>>> Writing policy
> >>>>
> >>>> http://docs.openstack.org/developer/congress/policy.html
> >>>>
> >>>> Let me know if you have any questions,
> >>>>
> >>>> Tim
> >>>>
> >>>> On Wed, May 4, 2016 at 11:51 PM Weyl, Alexey (Nokia - IL)
> >>>> mailto:alexey.w...@nokia.com>> wrote:
> >>>>
> >>>> Hi to all Vitrage and Congress contributors,
> >>>>
> >>>> We had a good introduction meeting in Austin and we (Vitrage)
> >>> think
> >>>> that we can have a good collaboration between the projects.
> >>>>
> >>>> Vitrage, as an Openstack Root Cause Analysis (RCA) Engine,
> >>>> builds
> >>> a
> >>>> topology graph of all the entities in the system (physical,
> >>> virtual
> >>>> and application) from different datasources. It thus can enrich
> >>>> Congress by providing more data about what is happening in the
> >>>> system. Additionally, the Vitrage RCA and deduce alarms &
> >> states
> >>>> mechanism can enhance the visibility of faults and how they
> >>>> inter-relate.  By using this information Congress could then
> >>> execute
> >>>> different policies and perform more accurate actions.
> >>>>
> >>>> Another good property of Vitrage is that it can receive data
> >> also
> >>>> from non-openstack sources, like Nagios, which monitor the
> >>> physical
> >>>> resources, includi

[openstack-dev] [Congress] Mid-cycle sprint and next week's IRC

[Tim Hinrichs]

Just a reminder that next week Jan 26-Jan 28 is the Congress mid-cycle
Sprint in the Bay Area.  It's not too late to attend.  It'd be best to
register, but we'd love to have you in any case.

More details here:
https://wiki.openstack.org/wiki/Sprints/CongressMitakaSprint

Also, since we're at the Sprint next week, there will be NO IRC meeting Jan
27.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Mid-cycle sprint debrief

[Tim Hinrichs]

Hi all,

TL;DR.  We had a great midcycle sprint.  If you want to help us move
Congress toward its new distributed architecture, there are a list of items
you can help with below.


1. We had a productive mid-cycle sprint last week!  Here were the topics we
covered...

- Design discussions about implementing the distributed architecture using
oslo-messaging.

- Discussion about integrating Monasca so that when people write policy
Congress can use Monasca alarms to push information to us.

- Discussion with Su Zhang about his experiences using Congress at Symantec.

- Code sprint aimed at migrating off of our current, in-process message bus
to oslo-messaging.

For details about the discussions, check out the etherpad...
https://etherpad.openstack.org/p/congress-mitaka-sprint

2. Moving forward in the short-term, we are focusing on migrating from our
existing in-process message bus (DSE) to a small wrapper around
oslo-messaging (DSE2).

- As we migrate from DSE to DSE2, we're leaving the mainline code in place
and minimizing the changes so that it runs on both the old arch and the new
arch.  We use the flag 'distributed_architecture' to signal which version
we are running.

- Ideally the tests will all continue to function without modification in
both the old and new architectures.  But for those test files that don't
pass, we are copying them to congress/tests2 and modifying them there.

- We have disabled a few tests temporarily and marked them with TODO(dse2)
and an explanation as to why they are disabled so we can easily grep for
them later.

- tox -enew_arch will (soon) run all the tests in congress/tests2.

3. For those of you looking to help out, here are a few items you can sign
up for.

3.1. Work on porting congress/tests/test_congress.py to the new distributed
arch.  See congress/tests2/test_dse2.py for an example.  In fact, there may
be tests commented out in tests2/dse2/test_dse2.py that we should
re-enable/port.
https://bugs.launchpad.net/congress/+bug/1541008

3.2. Work on porting the remaining API models to the new arch.  See my
recent changesets for the basic idea.
https://review.openstack.org/#/c/274957/

https://bugs.launchpad.net/congress/+bug/1541001
https://bugs.launchpad.net/congress/+bug/1541002
https://bugs.launchpad.net/congress/+bug/1541003
https://bugs.launchpad.net/congress/+bug/1541004

3.3. Create a non-voting gate job for tox -enew_arch
https://bugs.launchpad.net/congress/+bug/1540990

3.4. Try out the scripts/start_process.py script to see how we would use it
for the new arch.  Eventually we'll want a non-voting job in the gate that
runs all the tempest tests on the new architecture.
https://bugs.launchpad.net/congress/+bug/1541019

Questions/comments?
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] review for UI

[Tim Hinrichs]

Yali has been working hard on a new GUI for writing Congress policies.  The
goal was to focus on the common case and make it super-easy (and accept
that the uncommon cases may be hard or impossible).  So this GUI is
complementary to the one we have: infrequent users who want to write simple
policies would use the new GUI, and power users would use the existing GUI.

As part of the review process for the new GUI, Yali is giving a demo.  B/c
of time-zones the choices are either 6-7a or 5-6p Pacific.  For those of
you who are interested, drop me a note with your availability this week and
Mon-Tue of next week.

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [murano] [congress] Congress needs to fetch environments from all tenants.

[Tim Hinrichs]

There are two things to remember here.

1) When you configure the Congress datasource driver to talk to Murano, you
choose which user rights Congress should use.  If you need to get all of
the tenants data, you want to choose an admin user for the Murano driver.
Personally I always use admin users so that I can write policy over
everything.  Typically we think of Congress as an admin tool.

2) As you point out, if the Murano driver doesn't provide all_tenants=true
argument when it makes the API call into Murano, it won't get all the data
for all the tenants; it'll only get the data for the user you provided in
(1).  Ideally whether all_tenants=true would be a datasource configuration
option, but it's not today.  The datasource drivers I've looked at all use
all_tenants=true.

Tim




On Wed, Jul 8, 2015 at 5:16 AM Kirill Zaitsev  wrote:

> 1) This does raise a security concern. We can however cover it with a
> separate policy-based permission, that would check if a user can view all
> tenants. nova seem to do so, see:
> https://github.com/openstack/nova/blob/4209d0140774adf3e162b7bde3cbd6b417065dd5/etc/nova/policy.json#L13
>
> 2) Will give it some thought, but it does seem like an ok practice.
>
> --
> Kirill Zaitsev
> Murano team
> Software Engineer
> Mirantis, Inc
>
> On 8 Jul 2015 at 14:44:51, Filip Blaha (filip.bl...@hp.com) wrote:
>
> Hi all,
>
> I started implement bp [1]. Problem is that congress needs data about
> environments from all tenants but murano API lists only environments of
> user's current tenant. We decided to ipmplement it similarly like
> listing servers in nova where is query parameter all_tenants=true for
> that (user must be admin) I have 2 questions about that:
>
> 1) Are there any security concerns about this approach?
> 2) Has someone better idea how to implement this?
>
> [1]
> https://blueprints.launchpad.net/murano/+spec/murano-api-all-tenants-search
>
> Regards
> Filip
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [murano] [congress] Congress needs to fetch environments from all tenants.

[Tim Hinrichs]

We sometimes want the ability to write policy across tenants, e.g. VMs from
Coke and Pepsi must always be deployed on different hosts.

I didn't think there were any roles that could see everything without
all_tenants=true.  If there are such roles, I'd be happy to remove the
all_tenants=true from the datasource drivers.

Tim


On Fri, Jul 10, 2015 at 8:00 AM Dolph Mathews 
wrote:

> How about using domain-based role assignments in keystone and requiring
> domain-level authorization in policy, and then only returning data about
> the collection of tenants that belong to the authorized domain? That way
> you don't have an API that violates multi-tenant isolation, consumable only
> by cloud operators.
>
> On Wed, Jul 8, 2015 at 6:27 AM, Filip Blaha  wrote:
>
>> Hi all,
>>
>> I started implement bp [1]. Problem is that congress needs data about
>> environments from all tenants but murano API lists only environments of
>> user's current tenant. We decided to ipmplement it similarly like listing
>> servers in nova where is query parameter all_tenants=true for that (user
>> must be admin) I have 2 questions about that:
>>
>> 1) Are there any security concerns about this approach?
>> 2) Has someone better idea how to implement this?
>>
>> [1]
>> https://blueprints.launchpad.net/murano/+spec/murano-api-all-tenants-search
>>
>> Regards
>> Filip
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] New IRC meeting time

[Tim Hinrichs]

To better accommodate the active contributors, we're moving our IRC meeting
to

2300 UTC = 4p Pacific = 7p Eastern
#openstack-meeting-3

Hope to see you there!
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [murano] [congress] murano congress integration test - trusts

[Tim Hinrichs]

Hi Filip,

Did you get this resolved?  If not, could you point me to the gate
failure?  I'm on #congress for higher-bandwidth communication.

Tim



On Wed, Jul 15, 2015 at 6:39 AM Filip Blaha  wrote:

> Hi all
>
> our congress integration tests were  broken by the change [1] (trusts
> enabled by default).  However I suspect problem could be with
> initialization congress client [2] or in python-congressclient. Any
> ideas about that? Thanks
>
> [1] https://review.openstack.org/#/c/194615/
> [2]
>
> https://github.com/openstack/murano/blob/6ac473fabbc2d2e1f3ed4c3d36be6439c1d6c2cd/murano/engine/client_manager.py#L102
>
> Regards
> Filip
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] New IRC meeting time

[Tim Hinrichs]

Yes the day is still Tuesday.  Sorry that wasn't clear.

Tim

On Tue, Jul 14, 2015 at 7:30 PM Masahito MUROI 
wrote:

> I'm happy to see that.
>
> btw, is the day on Tuesday?
>
> best regard,
> masa
>
> On 2015/07/15 9:52, Zhou, Zhenzan wrote:
> > Glad to see this change.
> > Thanks for the supporting for developers in Asia☺
> >
> > BR
> > Zhou Zhenzan
> >
> > From: Tim Hinrichs [mailto:t...@styra.com]
> > Sent: Wednesday, July 15, 2015 02:14
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: [openstack-dev] [Congress] New IRC meeting time
> >
> > To better accommodate the active contributors, we're moving our IRC
> meeting to
> >
> > 2300 UTC = 4p Pacific = 7p Eastern
> > #openstack-meeting-3
> >
> > Hope to see you there!
> > Tim
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> 室井 雅仁(Masahito MUROI)
> Software Innovation Center, NTT
> Tel: +81-422-59-4539，FAX: +81-422-59-2699
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] meeting time (again)

[Tim Hinrichs]

Hi all,

Our new meeting time seemed to work well for almost everyone.
Unfortunately, there's one person for which the meeting time is 4:30a.  So
before our new meeting time becomes part of our routine, it's worth
figuring out if we can find a time where everyone is awake.

Here's a doodle poll.  Ignore the date on it.  I'm mainly interested in
when the non-US folks are available (meaning awake).  Sorry all the times
are Pacific--I couldn't find a way to include multiple time zones and
figured I'd be least likely to make mistakes this way.

http://doodle.com/intkzvgizusb7b4t

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Need reviews of bp rpc-for-dse

[Tim Hinrichs]

Hi Yingxin,

+1 to joining the sprint.  If you can't make the mid-cycle sprint in
person, we're planning to have a remote option.  Keep in mind though that
it will be harder to engage with everyone if you're remote.

We'll review your spec and get you feedback, but the purpose of the sprint
(Aug 6-7) is to decide and hopefully build the next generation of
communication between the policy engines and the datasources.

Tim

On Thu, Jul 23, 2015 at 1:22 AM Masahito MUROI 
wrote:

> Hi Yingxin,
>
> I think moving Congress from dse to rpc is a good idea.
>
> Congress team will discuss its scalability in mid cycle sprint[1], [2].
> I guess using rpc is one of key item for congress to get the ability.
> Why don't you join the meetup?
>
> [1] https://wiki.openstack.org/wiki/Sprints/CongressLibertySprint
> [2]
>
> http://www.eventbrite.com/e/congress-liberty-midcycle-sprint-tickets-17654731778
>
> best regard,
> masa
>
> On 2015/07/23 16:07, Cheng, Yingxin wrote:
> > Hi all,
> >
> >
> > I have some thoughts about congress dse improvement after having read
> the code for several days.
> >
> > Please refer to bp/rpc-for-dse<
> https://blueprints.launchpad.net/congress/+spec/rpc-for-dse>, its idea is
> to implement RPC in deepsix. Congress can benefit from lower-coupling
> between dse services. And the steps towards oslo-messaging integration can
> be milder too.
> >
> > Eager to learn everything from Community. Anything wrong, please kindly
> point it out.
> >
> >
> > Thank you
> > Yingxin
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> 室井 雅仁(Masahito MUROI)
> Software Innovation Center, NTT
> Tel: +81-422-59-4539，FAX: +81-422-59-2699
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Need reviews of bp rpc-for-dse

[Tim Hinrichs]

Hi Yingxin,

We definitely want details for what you have in mind.  In case you haven't
done this before, we write a 'spec' to explain the details.  Here's some
info on how to do that.

https://wiki.openstack.org/wiki/Congress#How_To_Propose_a_New_Feature

Tim



On Thu, Jul 23, 2015 at 12:13 AM Cheng, Yingxin 
wrote:

>  Hi all,
>
>
>
>
>
> I have some thoughts about congress dse improvement after having read the
> code for several days.
>
>
>
> Please refer to bp/rpc-for-dse
> , its idea
> is to implement RPC in deepsix. Congress can benefit from lower-coupling
> between dse services. And the steps towards oslo-messaging integration can
> be milder too.
>
>
>
> Eager to learn everything from Community. Anything wrong, please kindly
> point it out.
>
>
>
>
>
> Thank you
>
> Yingxin
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] How to start a replica ?

[Tim Hinrichs]

Could you show us the contents of /tmp/congress.conf?

Tim


On Mon, Jul 27, 2015 at 3:09 PM Wong, Hong  wrote:

>  Hi Tim and Alex,
>
>
>
> I see congress recently added the HA functionality, and I was looking at
> the tempest test code to understand how to start a replica.  I created a
> new congress.conf file with the different "bind_port" and set the
> "datasource_sync_period" value to 5.  However, I got the errors below when
> I try to bring up the replica:
>
>
>
> to start the replica: cd /opt/stack/congress && python
> /usr/local/bin/congress-server --config-file /tmp/congress.conf & echo $!
> >/tmp/congress.pid; fg || echo "congress failed to start" | tee
> "/tmp/congress.failure"
>
>
>
> 2015-07-27 14:56:33.592 TRACE congress.service Traceback (most recent call
> last):
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/opt/stack/congress/congress/service.py", line 32, in wrapper
>
> 2015-07-27 14:56:33.592 TRACE congress.service return f(*args, **kw)
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/opt/stack/congress/congress/service.py", line 50, in congress_app_factory
>
> 2015-07-27 14:56:33.592 TRACE congress.service cage =
> harness.create(root_path, data_path)
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/opt/stack/congress/congress/harness.py", line 151, in create
>
> 2015-07-27 14:56:33.592 TRACE congress.service for policy in
> db_policy_rules.get_policies():
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/opt/stack/congress/congress/db/db_policy_rules.py", line 84, in
> get_policies
>
> 2015-07-27 14:56:33.592 TRACE congress.service session = session or
> db.get_session()
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/opt/stack/congress/congress/db/api.py", line 40, in get_session
>
> 2015-07-27 14:56:33.592 TRACE congress.service facade =
> _create_facade_lazily()
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/opt/stack/congress/congress/db/api.py", line 27, in _create_facade_lazily
>
> 2015-07-27 14:56:33.592 TRACE congress.service _FACADE =
> session.EngineFacade.from_config(cfg.CONF, sqlite_fk=True)
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py",
> lin
>
> .
>
> 2015-07-27 14:56:33.592 TRACE congress.service   File
> "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in
> _check_mysql_exception
>
> 2015-07-27 14:56:33.592 TRACE congress.service raise errorclass(errno,
> errorvalue)
>
> 2015-07-27 14:56:33.592 TRACE congress.service OperationalError:
> (pymysql.err.OperationalError) (1040, u'Too many connections')
>
>
>
> I got the same error when running the tempest test as well.  Any idea ?
>
>
>
> Thanks,
>
> Hong
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] meeting time change

[Tim Hinrichs]

Hi all,

We managed to find a day/time where all the active contributors can attend
(without being up too early/late).  The room, day, and time have all
changed.

Room: #openstack-meeting-2
Time: Wednesday 5p Pacific = Thursday midnight UTC

Next week we begin with this new schedule.

And don't forget that next week Thu/Fri is our Mid-cycle sprint.  Hope to
see you there!

Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] meeting time change

[Tim Hinrichs]

Peter pointed out that no one uses #openstack-meeting-2.  So we'll go with
#openstack-meeting.  Here are the updated meeting details.

Room: #openstack-meeting
Time: Wednesday 5p Pacific = Thursday midnight UTC

There's a change out for review that will update the meeting website once
it merges.
http://eavesdrop.openstack.org/#Congress_Team_Meeting
https://review.openstack.org/#/c/207981/

Tim

On Fri, Jul 31, 2015 at 9:24 AM Tim Hinrichs  wrote:

> Hi all,
>
> We managed to find a day/time where all the active contributors can attend
> (without being up too early/late).  The room, day, and time have all
> changed.
>
> Room: #openstack-meeting-2
> Time: Wednesday 5p Pacific = Thursday midnight UTC
>
> Next week we begin with this new schedule.
>
> And don't forget that next week Thu/Fri is our Mid-cycle sprint.  Hope to
> see you there!
>
> Tim
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Mid-cycle sprint summary

[Tim Hinrichs]

Hi all,

We just finished up a great 2 day sprint focusing on a new distributed
architecture for Congress.  Details can be found in the etherpad:

https://etherpad.openstack.org/p/congress-liberty-sprint

Here's the summary.

1. Architecture.  Each datasource driver will run in its own process; each
policy engine will run in its own process; the API will run in its own
process.  All processes will communicate using oslo-messaging.  We decided
to remove the functionality for creating/deleting datasources, since that
functionality will be handled by the operating system.

2. Blueprints.  The blueprints we created all start with "dist-".  Please
sign up if you're interested.  If you attended the sprint and volunteered
for a blueprint but did not sign up, please do so.

https://blueprints.launchpad.net/congress

3. Code commits.  We're making changes in place on master.  The plan is to
make most of the changes without breaking existing functionality.  Then
there will be one or two changes at the end that swap out the current,
single-process architecture for the new, multi-process architecture.

4. Timelines.  We're hoping to have the basic functionality in place by
Tokyo.  We will release the current architecture for liberty and the new
architecture for M.

Thanks for a great sprint, everyone!

Let me know if you have questions.
Tim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Confused syntax error when inserting rule.

[Tim Hinrichs]

Hi Rui,

The problem with the following rule is that there are a bunch of hidden
variables in the "not cinder:volumes(...)" literal.  The error message
shows the hidden variables.  The syntax restriction is that every variable
in a negative literal must appear in a positive literal in the body.  Those
hidden variables fail to satisfy that restriction, hence the error.

 error(id) :- cinder:volumes(id=id), not cinder:volumes(id=id,
status="available")

The reason the other rule worked is that you made the hidden variables
equivalent to the ones in the positive literal, e.g. x_0_1 shows up in both
the positive and negative literals.

error(x) :- cinder:volumes(x, _x_0_1, _x_0_2, _x_0_3, _x_0_4, _x_0_5,
_x_0_6, _x_0_7, _x_0_8),not cinder:volumes(x, _x_0_1, _x_0_2,
\"available\",_x_0_4, _x_0_5, _x_0_6, _x_0_7, _x_0_8)

But in the auto-generated one, the variables in the two literals are
different e.g. _x_0_1 and _x_1_1

error(id) :- cinder:volumes(id, _x_0_1, _x_0_2, _x_0_3, _x_0_4, _x_0_5,
_x_0_6, _x_0_7, _x_0_8), not cinder:volumes(id, _x_1_1, _x_1_2,
"available", _x_1_4, _x_1_5, _x_1_6, _x_1_7, _x_1_8).  Unsafe lits: not
cinder:volumes(id, _x_1_1, _x_1_2, "available", _x_1_4, _x_1_5, _x_1_6,
_x_1_7, _x_1_8)

Probably the solution you want is to write 2 rules:

error(id) :- cinder:volumes(id=id), not avail_cinder_vol(id)
avail_cinder_vol(id) :- cinder:volumes(id=id, status="available")

Tim

On Thu, Aug 13, 2015 at 8:07 PM Rui Chen  wrote:

> Sorry, send the same mail again, please comments at here, the other mail
> lack title.
>
> 2015-08-14 11:03 GMT+08:00 Rui Chen :
>
>> Hi folks:
>>
>> I face a problem when I insert a rule into Congress. I want to find
>> out all of the volumes that are not available status, so I draft a rule
>> like this:
>>
>> error(id) :- cinder:volumes(id=id), not cinder:volumes(id=id,
>> status="available")
>>
>> But when I create the rule, a error is raised:
>>
>> (openstack) congress policy rule create chenrui_p "error(id) :-
>> cinder:volumes(id=id),not cinder:volumes(id=id, status=\"available\")"
>> ERROR: openstack Syntax error for rule::Errors: Could not reorder rule
>> error(id) :- cinder:volumes(id, _x_0_1, _x_0_2, _x_0_3, _x_0_4, _x_0_5,
>> _x_0_6, _x_0_7, _x_0_8), not cinder:volumes(id, _x_1_1, _x_1_2,
>> "available", _x_1_4, _x_1_5, _x_1_6, _x_1_7, _x_1_8).  Unsafe lits: not
>> cinder:volumes(id, _x_1_1, _x_1_2, "available", _x_1_4, _x_1_5, _x_1_6,
>> _x_1_7, _x_1_8) (vars set(['_x_1_2', '_x_1_1', '_x_1_6', '_x_1_7',
>> '_x_1_4', '_x_1_5', '_x_1_8'])) (HTTP 400) (Request-ID:
>> req-1f4432d6-f869-472b-aa7d-4cf78dd96fa1)
>>
>> I check the Congress policy docs [1], looks like that the rule don't
>> break any syntax restrictions.
>>
>> If I modify the rule like this, it works:
>>
>> (openstack) congress policy rule create chenrui_p "error(x) :-
>> cinder:volumes(x, _x_0_1, _x_0_2, _x_0_3, _x_0_4, _x_0_5, _x_0_6, _x_0_7,
>> _x_0_8),not cinder:volumes(x, _x_0_1, _x_0_2, \"available\",_x_0_4, _x_0_5,
>> _x_0_6, _x_0_7, _x_0_8)"
>>
>> +-++
>> | Field   | Value
>>  |
>>
>> +-++
>> | comment | None
>>   |
>> | id  | ad121e09-ba0a-45d6-bd18-487d975d5bf5
>>   |
>> | name| None
>>   |
>> | rule| error(x) :-
>>  |
>> | | cinder:volumes(x, _x_0_1, _x_0_2, _x_0_3, _x_0_4, _x_0_5,
>> _x_0_6, _x_0_7, _x_0_8), |
>> | | not cinder:volumes(x, _x_0_1, _x_0_2, "available",
>> _x_0_4, _x_0_5, _x_0_6, _x_0_7, _x_0_8) |
>>
>> +-++
>>
>> I'm not sure this is a bug or I miss something from docs, so I need
>> some feedback from mail list.
>> Feel free to discuss about it.
>>
>>
>> [1]:
>> http://congress.readthedocs.org/en/latest/policy.html#datalog-syntax-restrictions
>>
>>
>> Best Regards.
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress] [murano] Congress jenkins job is failing

[Tim Hinrichs]

Looks like we merged a database schema change without the migration
script.  I'm on it.  (We'll get our tempest tests running in gate again
ASAP.)

Tim


On Tue, Aug 18, 2015 at 7:36 AM Filip Blaha  wrote:

> Hi Congress team.
>
> Our jenkins job testing integration congress and murano is failing.
> Congress fails to start on some DB error [1] . Does anyone know what
> could be the problem?
>
> [1]
>
> http://logs.openstack.org/08/211608/3/check/gate-murano-congress-devstack-dsvm/db1b682/logs/screen-congress.txt.gz#_2015-08-18_10_17_05_033
>
> Thanks
> Filip
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev