Re: [openstack-dev] [Congress] Policy types
Hi Tim, Here is a small change I wanted to try in runtime.py It may already exist in MaterializedViewTheory, but wasn't clear to me. Checking to see if this is something that:1. makes sense 2. already exists 3. worth implementing? in that order. Let's take the example from private_public_network.classify error(vm) :- nova:virtual_machine(vm), nova:network(vm, network), not neutron:public_network(network), neutron:owner(network, netowner), nova:owner(vm, vmowner), not same_group(netowner, vmowner) same_group(user1, user2) :- cms:group(user1, group), cms:group(user2, group) nova:virtual_machine(vm1) nova:virtual_machine(vm2) nova:virtual_machine(vm3) nova:network(vm1, net_private) nova:network(vm2, net_public) neutron:public_network(net_public) nova:owner(vm1, tim) nova:owner(vm2, pete) nova:owner(vm3, pierre) neutron:owner(net_private, martin) In this example, if as in Scenario 1: Cloud services at our disposal: nova:virtual_machine(vm) nova:network(vm, network) nova:owner(vm, owner) neutron:public_network(network) neutron:owner(network, owner) cms:group(user, group) are all python functions called through some nova/neutron api, then, we just execute them to get a true/false value in runtime.py They should be first checked to make sure they are python functions and not condition primitives using 'callable' and os.dir or some such combination. If not, and they are assertions made in the file, not directly related to OS state, then in Scenario 2 nova:owner(vm1, tim) nova:owner(vm2, pete) nova:owner(vm3, pierre) Are assertions made in the file. In a dynamic environment, a python function could query an OS client to actually find the current owner, since some other OS command could have been used to change the owner without an entry being made in this file, i.e., without explicitly informing Congres. This may not occur currently with vms, but may be implemented in a future release. Similar other examples are possible https://ask.openstack.org/en/question/5582/how-to-change-ownership-between-tenants-of-volume/ https://blueprints.launchpad.net/cinder/+spec/volume-transfer So, I was thinking that python_nova_owner(vm1), is first checked as a python function which calls the appropriate OS client to check the current owner. As in nova:owner(vm1, python_nova_owner(vm1)) So, in runtime.py, in either scenario, condition primitives are first checked to see if it they are callable python functions (by using python 'callable' with os.dir). In which case, it is executed to get the name of the owner. All non-callable primitives are assumed assumed to be congress and/or datalog primitives, and unified through the Materializedviewtheory. Thanks, Date: Thu, 13 Mar 2014 08:55:24 -0700 From: thinri...@vmware.com To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Congress] Policy types Hi Prabhakar, I'm not sure the functionality is split between 'policy' and 'server' as cleanly as you describe. The 'policy' directory contains the Policy Engine. At its core, the policy engine has a generic Datalog implementation that could feasibly be used by other OS components. (I don't want to think about pulling it out into Oslo though. There are just too many other things going on and no demand yet.) But there are also Congress-specific things in that directory, e.g. the class Runtime in policy/runtime.py will be the one that we hook up external API calls to. The 'server' directory contains the code for the API web server that calls into the Runtime class. So if you're digging through code, I'd suggest focusing on the 'policy' directory and looking at compile.py (responsible for converting Datalog rules written as strings into an internal representation) and runtime.py (responsible for everything else). The docs I mentioned in the IRC should have a decent explanation of the functions in Runtime that the API web server will hook into. Be warned though that unless someone raises some serious objections to the proposal that started this thread, we'll be removing some of the more complicated functions from Runtime. The compile.py code won't change (much). All of the 3 new theories will be instances of MaterializedViewTheory. That's also the code that must change to add in the Python functions we talked about (more specifically see MaterializedViewTheory::propagate_rule(), which calls TopDownTheory::top_down_evaluation(), which is what will need modification). Tim - Original Message - | From: prabhakar Kudva nandava...@hotmail.com | To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org | Sent: Wednesday, March 12, 2014 1:38:55 PM | Subject: Re: [openstack-dev] [Congress] Policy types | | | | | Hi Tim, | | Thanks for your comments. | Would be happy to contribute to the propsal and code. | | The existing code
Re: [openstack-dev] [Congress] Policy types
Hi Tim, All, I was in the discussion yesterday (kudva), and would like to start gradually contributing to the code base. So, this discussion below is based on my limited exploration of Congress code, running it. I am trying some small pieces to implement to familiarize. Please view it as such. As I start adding code, I am sure, my thoughts will be more evolved. I agree with the three types you outline. I also agree that these will grow. We are already thinking of expanding congress for various other types of policies. But those would be a manageable start. Regarding the comment below. I was wondering if all conditions, and actions could be both: 1. python functions (for conditions they eval 2. policy primitives. The advantage of 1, is that it is just executed and a True or False returned by Python for conditions. For actions, python functions are executed to respond to conditions. This controls the growth of policies and adding more primitives, and makes it flexible (say to use alarms, monitors, os clients, nova actions etc). The advantage of 2, is the ability to use unification (as in unify.py) and do some logic reduction. This gives us the full strength of extensive and mature logic reasoning and reduction methods. One possibility is that it checks which one the two it is and does the appropriate evaluation for condition and action. There are drawbacks to this proposal as well. - We will have 3 separate policies that are conceptually very similar. As the policies grow larger, it will become increasingly difficult to keep the policies synchronized. This problem can be mitigated to some extent by having all 3 share a library of policy statements that they all apply in different ways (and such a library mechanism is already implemented). - As cloud services change their behavior, policies may need to be re-written. For example, right now Nova does not consult Congress before creating a VM; thus, to enforce policy surrounding VMs, the best we could do is write a Condition-Action policy that adjusts VM configuration when it learns about new VMs being created. If we later make Nova consult with Congress before creating a VM, we need to write an Access-control policy that puts the proper controls in place. Thanks, Prabhakar Kudva Date: Wed, 12 Mar 2014 10:05:23 -0700 From: thinri...@vmware.com To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Congress] Policy types Hi all, We started a discussion on IRC yesterday that I'd like to continue. The main question is what kind of policy does a Congress user actually write? I can see three options. The first two focus on actions (API calls that make changes to the state of the cloud) and the last focuses on just the cloud state. (By state of the cloud I mean all the information Congress can see about all the cloud services it is managing, e.g. all the information we can get through API calls to Nova, Neutron, Cinder, Heat, ...). 1) Access Control (e.g. Linux, XACML, AD): which *actions* can be performed by other cloud services (for each state of the cloud)2) Condition Action: which *actions* Congress should execute (for each state of the cloud)3) Classification (currently supported in Congress): which *states* violate real-world policy. [For those of you who have read docs/white-papers/etc. I'm using Classification in this note to mean the combination of the current Classification and Action Description policies.] The important observation is that each of these policies could contain different information from each of the others. - Access Control vs Condition Action. The Access Control policy tells *other cloud services* which actions they are *allowed* to execute. The Condition Action policy tells *Congress* which actions it *must* execute. These policies differ because they constrain different sets of cloud services. - Access Control vs. Classification. The Access Control policy might permit some users to violate the Classification policy in some situations (e.g. to fix violation A, we might need to cause violation B before eliminating both). These policies differ because a violation in one policy might be be a violation in the other. - Classification vs. Condition Action. The Classification policy might imply which actions *could* eliminate a given violation, but the Condition Action policy would dictate which of those actions *should* be executed (e.g. the Classification policy might tell us that disconnecting a network and deleting a VM would both eliminate a particular violation, but the Condition Action policy would tell us which to choose). And the Condition Action policy need not eliminate all the violations present in the Classification policy. Again these policies differ because a violation in one policy might not be a violation in the other. I'm proposing that for the first release of Congress we support all 3 of these policies. When a user inserts
Re: [openstack-dev] [Congress] Policy types
Hi Tim, Thanks for your comments. Would be happy to contribute to the propsal and code. The existing code already reflects the thoughts below, and got me in the line of ideas. Please orrect me if I am wrong as I am learning with these discussions: One part (reflected by code in policy directory is the generic condition- action engine which could take logic primitives and (in the future) python functions, evaluate the conditions and execute the action. This portable core engine be used for any kind of policy enforcement (as by other OS projects), such as for data center monitoring and repair, service level enforcement, compliance policies, optimization (energy, performance) etc... at any level of the stack. This core engine seems possibly a combination of logic reasoning/unification and python function evaluation, and python code actions. Second part (reflected by code in server) are the applications for various purposes. These could be project specific, task specific. We could add a diverse set of examples. The example I have worked with seems closer to compliance (as in net owner, vm owner check), and we will add more. Prabhakar Date: Wed, 12 Mar 2014 12:33:35 -0700 From: thinri...@vmware.com To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Congress] Policy types Hi Prabhakar, Thanks for the feedback. I'd be interested to hear what other policy types you have in mind. To answer your questions... We're planning on extending our policy language in such a way that you can use Python functions as conditions (atom in the grammar) in rules. That's on my todo-list but didn't mention it yesterday as we were short on time. There will be some syntactic restrictions so that we can properly execute those Python functions (i.e. we need to always be able to compute the inputs to the function). I had thought it was just an implementation detail I hadn't gotten around to (all Datalog implementations I've seen have such things), but it sounds like it's worth writing up a proposal and sending it around before implementing. If that's a pressing concern for you, let me know and I'll bump it up the stack (a little). If you'd like, feel free to draft a proposal (or remind me to do it once in a while). As for actions, I typically think of them as API calls to other OS components like Nova. But they could just as easily be Python functions. But I would want to avoid an action that changes Congress's internal data structures directly (e.g. adding a new policy statement). Such actions have caused trouble in the past for policy languages (though for declarative programming languages like Prolog they are less problematic). I don't think there's anyway we can stop people from creating such actions, but I think we should advocate against them. Tim From: prabhakar Kudva nandava...@hotmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Wednesday, March 12, 2014 11:34:04 AM Subject: Re: [openstack-dev] [Congress] Policy types Hi Tim, All, I was in the discussion yesterday (kudva), and would like to start gradually contributing to the code base. So, this discussion below is based on my limited exploration of Congress code, running it. I am trying some small pieces to implement to familiarize. Please view it as such. As I start adding code, I am sure, my thoughts will be more evolved. I agree with the three types you outline. I also agree that these will grow. We are already thinking of expanding congress for various other types of policies. But those would be a manageable start. Regarding the comment below. I was wondering if all conditions, and actions could be both: 1. python functions (for conditions they eval 2. policy primitives. The advantage of 1, is that it is just executed and a True or False returned by Python for conditions. For actions, python functions are executed to respond to conditions. This controls the growth of policies and adding more primitives, and makes it flexible (say to use alarms, monitors, os clients, nova actions etc). The advantage of 2, is the ability to use unification (as in unify.py) and do some logic reduction. This gives us the full strength of extensive and mature logic reasoning and reduction methods. One possibility is that it checks which one the two it is and does the appropriate evaluation for condition and action. There are drawbacks to this proposal as well. - We will have 3 separate policies that are conceptually very similar. As the policies grow larger, it will become increasingly difficult to keep the policies synchronized. This problem can be mitigated to some extent by having all 3 share a library of policy statements that they all apply in different ways (and such a library mechanism is already implemented). - As cloud services change their behavior, policies may need to be re-written. For example, right now Nova
