Re: [openstack-dev] [TripleO] TripleO core reviewer update - november
Excerpts from Robert Collins's message of 2013-10-30 02:06:38 -0700: Hi, like most OpenStack projects we need to keep the core team up to date: folk who are not regularly reviewing will lose context over time, and new folk who have been reviewing regularly should be trusted with -core responsibilities. In this months review: - James Slagle for -core +1 - Arata Notsu to be removed from -core +1 - Devananda van der veen to be removed from -core +1 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
I looked at it once, but noticed its piggy backing on DNS infrastructure and then wondered who has actually used this in production and didn't find much info (except that it's a cornell project). I never quite did get around to setting it up. Maybe u will have better luck :) Sent from my really tiny device... On Oct 30, 2013, at 12:32 PM, Jay Pipes jaypi...@gmail.com wrote: Has anyone looked into using concoord for distributed locking? https://pypi.python.org/pypi/concoord Best, -jay On 10/30/2013 02:39 PM, Joshua Harlow wrote: So my idea here was that to break the abstraction for heat into 3 parts. Pardon my lack of heat terminology/knowledge if I miss something. 1. The thing that receives the API request (I would assume an api server here). I would expect #1 to parse something into a known internal format. Whether this is tasks or jobs or something is up to heat, so this might of been my lack of understanding heat concepts here, but usually an API request translates into some internal format. Maybe this is the parser or something else (not sure really). Lets assume for now that it parses the API request into some tasks + flow (what taskflow provides). So then it becomes a question of how what do u do with those tasks flows (what I call stage #2). - https://wiki.openstack.org/wiki/TaskFlow#Two_thousand_foot_view To me this is where taskflow 'shines' in that it has an engine concept which can run in various manners (the tasks and flow are not strongly associated with a engine). One of these engines is planned to be a distributed one (but its not the only one) and with that engine type it would have to interact with some type of job management system (or it would have to provide that job management system - or a simple version itself), but the difference is that the about tasks and flows (and the links/structure between them) is still disconnected from the actual engine that runs those tasks flows. So this to mean means that there is plugabbility with regard to execution, which I think is pretty great. If that requires rework of the heat model, way of running, maybe its for the better? Idk. As taskflow is still newish, and most projects in openstack have there own distributed model (conductors, rpc process separation), we wanted to focus on having the basic principles down, and the review https://review.openstack.org/#/c/47609/ I am very grateful for jessica working her hardest to get that in a nearly there state. So yes, taskflow will continue on the path/spirit of 47609, and contributions are welcome of course :-) Feel free to also jump on #openstack-state-management since it might be easier to just chat there in the end with other interested parties. -Josh On 10/30/13 11:10 AM, Steven Dake sd...@redhat.com wrote: On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? I will -2 any patch that adds zookeeper as a dependency to Heat. The rest of the idea
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
Lock free solutions usually involve very predictable and very well understood (and typically provably correct) state machines. In openstack u have state machines but I think there is work to be done to increase there understandability and predictability. Lock free algorithms are also extremely hard to get right (this is why the above state machine typically undergoes very thought out and extensive peer review). In openstack we all are that peer review :) Hopefully this discussion here (and on irc and on code reviews) can help jump start more thinking around this whole area. Sent from my really tiny device... On Oct 30, 2013, at 12:33 PM, Qing He qing...@radisys.com wrote: Has anyone looked at any lock-free solution? -Original Message- From: Sandy Walsh [mailto:sandy.wa...@rackspace.com] Sent: Wednesday, October 30, 2013 12:20 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey On 10/30/2013 03:10 PM, Steven Dake wrote: I will -2 any patch that adds zookeeper as a dependency to Heat. Certainly any distributed locking solution should be plugin based and optional. Just as a database-oriented solution could be the default plugin. Re: the Java issue, we already have optional components in other languages. I know Java is a different league of pain, but if it's an optional component and left as a choice of the deployer, should we care? -S PS As an aside, what are your issues with ZK? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] An early peek into the Solum.io website
Just wondering, Will said infrastructure be made available to all other openstack projects? It'd be nice to have that for other projects to (a nova blog, a taskflow blog...). It seems/feels a little awkward to me to have solum be a special snowflake here. Sent from my really tiny device... On Oct 30, 2013, at 9:55 PM, Adrian Otto adrian.o...@rackspace.com wrote: Noorul, Yes, it uses WordPress. The community website will be used primarily as a rally point for attracting new contributors. It will have a blog feature which is currently being configured so the project contributors can publish related content there. We expect to use this site to attract new developers who are not already part of the OpenStack ecosystem. Adrian On Oct 30, 2013, at 9:25 PM, Noorul Islam K M noo...@noorul.com wrote: Roshan Agrawal roshan.agra...@rackspace.com writes: The Solum community website is very close to be launched publicly. If you want to take an early peek into how it is coming along, here are the access details: www.Solum.iohttp://www.Solum.io User name: Solum, password: OpenStack The Solum logo is still in works, what we have now is meant to be a placeholder till we finalize on an awesome looking logo. Comments/suggestions welcome. (Cc Solum list for now, till everyone have had a chance to migrate to the openstack list) Is there any dynamic content in this site? Why are we not utilizing github pages or something that can be generated from git repository? Thanks and Regards Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Comments on Steve Baker's Proposal on HOT Software Config
Zane Bitter zbit...@redhat.com wrote on 30.10.2013 22:33:31: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org, Date: 30.10.2013 22:36 Subject: Re: [openstack-dev] [Heat] Comments on Steve Baker's Proposal on HOT Software Config On 30/10/13 20:35, Lakshminaraya Renganarayana wrote: I'd like to see some more detail about how inputs/outputs would be exposed in the configuration management systems - or, more specifically, how the user can extend this to arbitrary configuration management systems. The way inputs/outputs are exposed in a CM system would depend on its conventions. In our use with Chef, we expose these inputs and outputs as a Chef's node attributes, i.e., via the node[][] hash. I could imagine a similar scheme for Puppet. For a shell type of CM provider the inputs/outputs can be exposed as Shell environment variables. To avoid name conflicts, these inputs/outputs can be prefixed by a namespace, say Heat. Right, so who writes the code that exposes the inputs/outputs to the CM system in that way? If it is the user, where does that code go and how does it work? And if it's not the user, how would the user accommodate a CM system that has not been envisioned by their provider? That's what I'm trying to get at with this question. I think it is not the user who writes this binding code / glue code, but I envision a handful of component providers for the most common CM systems that implement the logic to (1) take data from Heat, (2) pass them to the CM system invocation, (3) wait for the CM system to report completion, (4) parse return data and pass it to Heat. So this is basically like an adapter, and for each such adapter it would have to be documented like the automation must be writen to work with it. E.g. in bash scripts the user can access all input as environment variables etc, or for Chef there mapping is like Lakshmi described it above. We have something like this implemented internally and I think such an implementation could be added to Heat. For any kind of custom CM system where we do not provide such an adapter, the user would have to write a plugin to handle the same logic. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Hi Adrian Sounds like there's a need for a true workflow engine here. I suspect that for any use case beyond trivial, each user will have her own flavor of CD (TripleO is a good example for this) - some will prefer to use a canary server and test it manually before deploying to other servers, some will deploy to specific servers based on the subset of users they're serving, some will do a red/black type of deployment (Which maps well into environments below), etc. But I think that there should be an user-configurable mechanism to define and tweak these workflows. There will also be additional related workflows - e.g. rolling back a deployment, or updating the infrastructure components of a running application (e.g. database version upgrade). I might be raising a topic that's already been discussed (sorry if I am), but I was wondering how this would fit into the architecture. Thanks! Uri On Oct 31, 2013, at 6:42 AM, Adrian Otto adrian.o...@rackspace.com wrote: Robert, We would be interested in exploring your use case for CD, and help you judge what would be the best fit. I know that Monty Taylor has identified some parts of openstack-infra that we could potentially leverage in Solum to address our CI/CD goals. The design of that feature is still pending, so we are looking forward to sourcing its of input on that. I don't anticipate any fundament gaps that would prevent you from using a future version of Solum to help narrow your scope. The whole point of our focus is to try to make these goals easier to achieve, and try to get maximum leverage of what the OpenStack ecosystem already offers. The high level concept behind the GIt Integration feature currently conceived for Solum is to offer application developers a way to commit code to a Git repo and upon a git push command, have a remote system trigger a modular CI/CD pipeline that ends with that application running on an OpenStack cloud in one or more environments (dev/test/stage/prod/etc.). In a simple case that may be (in the case of a dynamic language like Python) the testing of the code, arrangement of the commit into a deployable bundle, triggering of the Solum API to trigger a deployment, automatic generation of a HOT artifact, and stimulation of the Heat API and any other API calls needed. In a more complex scenario, you may want additional features and capabilities, such as a system like Gerrit to trigger the commit/push, integrate an artifact repository for storing the builds, etc. Using Zuul for that would be appropriate for sure. There are also dev shops who already have a lot invested into their Jenkins setups, and will want to bypass portions of what I described above, so there will be a use case supported where you can just integrate with the solum API, and pass in a pre-built bundle artifact and bypass part or all of Solum's CI features, and just use it for deployment and re-deployment and promotion of a deployment between various environments (dev/test/pros/etc.). You would expect features like canary deploy in that case that might make it more attractive than just generating your own HOT artifact in your Jenkins workflow and passing that in directly for orchestration. You should expect a higher level of control for management of the deployed system with abstracts that let you easily extend the system for monitoring and management reasons. Note that not all this functionality belongs in Solum. Some of it may actually come from Heat, and other downstream systems. The idea is that we address the key gaps. Adrian On Oct 30, 2013, at 9:16 PM, Robert Collins robe...@robertcollins.net wrote: Hi Adrian, I'm very interested in Solum, particularly as it relates to TripleO : as you know the TripleO story is to treat OpenStack as just another application to install via Heat/Nova etc - and we've got a major story around pulling from git - CI + CD pipeline - automated deployment to production. So - where we can say 'hey, this should be part of Solum', we can hive stuff out of the TripleO program into Solum - but only if we've got compatible stories. Right now, our design for CD is basically Gerrit + Zuul - exactly the same as gating checks for OpenStack itself. When I look at https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries I'm having a little trouble mapping that into your design. I'm wondering if you can expand on how that hangs together? Thanks! Rob On 31 October 2013 14:01, Adrian Otto adrian.o...@rackspace.com wrote: Clayton, Thanks for adding the diagram illustrating the flow. I expect that respond to client may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
On Wed, Oct 30 2013, Clint Byrum wrote: In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I didn't check extensively the patch and what it tries to address, so pardon me if I'm wrong, but I think there might be some overlap with what we need in Ceilometer. We scheduled a session in Oslo to talk about this: http://icehousedesignsummit.sched.org/event/ac126a677c0c8a7960b9ff75ccc8d4d2#.UnIeVHUsJRQ We want to solve this in a agnostic manner. There also might be some overlap or complementary with TaskFlow, I hope we'll discuss this during this session. -- Julien Danjou /* Free Software hacker * independent consultant http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Version numbering of TripleO releases
Hi On 31 October 2013 05:09, Roman Podoliaka rpodoly...@mirantis.com wrote: 0.MAJOR.MINOR versioning totally makes sense to me until we get to 1.0.0. +1 Your examples are both excellent. Thanks very much for taking the release baton for now :) -- Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] Openstack XCP with OVS Quantum
On 10/31/2013 2:53 PM, Rajshree Thorat wrote: Hi All, I have successfully configured Openstack Havana with xen hypervisor(XCP). Initially creating/deleting instances from OpenStack works as expected but networking part(neutron with OpenvSwitch) was not working. The steps I performed to make it work are as below: Normal flow to get DHCP IP: - VM boots and asks for an IP through DHCP - The Nova Compute has a GRE tunnel to the OpenStack Networking node where the neutron/openvSwitch agent provides an IP to the VM. VM --- Nova Compute Node --- GRE tunnel --- OpenStack Networking node --- DHCP agent In case of XCP when guest VM boots it sends a DHCP request to dom0 through xenapi but dom0 unable to communicate with OpenStack Networking node over GRE tunnel. To allow VM's to communicate with Network node over GRE tunnel, we can assign one more nic(eth2) which is part of xapi1 of dom0 to nova-compute and add eth2 to br-int on nova-compute. xapi1 is a openstack network bridge in dom0. Now the packet will traverse as VM -- xapi1(dom0) -- eth2(compute) -- br-tun(compute) -- Network-node(over GRE tunnel) VM -- xapi1(dom0) -- eth2(compute) -- br-tun(compute) -- Network-node(over GRE tunnel) Inbuilt Openvswitch-controller configures the v-switches to allow only specific flows which matches the rules installed on them. Even if we add eth2 to br-int, we will also need to add generic rules to br-tun such that they are able to pass the packets received from eth2 to br-int, then to br-tun and then to network node over GRE tunnel. That's it you are done ! dump-flows before adding rules: root@compute:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=3.248s, table=0, n_packets=0, n_bytes=0, idle_age=3, priority=1,in_port=1 actions=resubmit(,1) cookie=0x0, duration=2.069s, table=0, n_packets=0, n_bytes=0, idle_age=2, priority=1,in_port=2 actions=resubmit(,2) cookie=0x0, duration=3.187s, table=0, n_packets=1, n_bytes=70, idle_age=2, priority=0 actions=drop cookie=0x0, duration=3.066s, table=1, n_packets=0, n_bytes=0, idle_age=3, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,21) cookie=0x0, duration=3.126s, table=1, n_packets=0, n_bytes=0, idle_age=3, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0x0, duration=3.006s, table=2, n_packets=0, n_bytes=0, idle_age=3, priority=0 actions=drop cookie=0x0, duration=2.946s, table=3, n_packets=0, n_bytes=0, idle_age=2, priority=0 actions=drop cookie=0x0, duration=2.886s, table=10, n_packets=0, n_bytes=0, idle_age=2, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0-NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]-NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 cookie=0x0, duration=2.825s, table=20, n_packets=0, n_bytes=0, idle_age=2, priority=0 actions=resubmit(,21) cookie=0x0, duration=2.766s, table=21, n_packets=0, n_bytes=0, idle_age=2, priority=0 actions=drop Add flows: root@compute:~# ovs-vsctl add-port br-int eth2 tag=1 Here the neutron-plugin-openvswitch-agent has put port eth2 into VLAN 1 (tag 1) on br-int. root@compute:~# ovs-ofctl add-flow br-tun priority=3,in_port=1,dl_vlan=1,actions=set_tunnel:0x1,NORMAL It is for outgoing traffic from br-int VLAN 1 - it sets the GRE key to 0x1 and gives it the NORMAL action. root@compute:~# ovs-ofctl add-flow br-tun priority=2,tun_id=0x1,actions=mod_vlan_vid:1,NORMAL This flow accepts incoming traffic. dump-flows after adding rules: root@compute:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=115.467s, table=0, n_packets=5, n_bytes=958, idle_age=58, priority=2,tun_id=0x1 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=133.203s, table=0, n_packets=5, n_bytes=830, idle_age=61, priority=3,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=343.011s, table=0, n_packets=7, n_bytes=1230, idle_age=186, priority=1,in_port=1 actions=resubmit(,1) cookie=0x0, duration=341.832s, table=0, n_packets=0, n_bytes=0, idle_age=341, priority=1,in_port=2 actions=resubmit(,2) cookie=0x0, duration=342.95s, table=0, n_packets=4, n_bytes=300, idle_age=334, priority=0 actions=drop cookie=0x0, duration=342.829s, table=1, n_packets=7, n_bytes=1230, idle_age=186, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,21) cookie=0x0, duration=342.889s, table=1, n_packets=0, n_bytes=0, idle_age=342, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0x0, duration=342.769s, table=2, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop cookie=0x0, duration=342.709s, table=3, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop cookie=0x0, duration=342.649s, table=10, n_packets=0, n_bytes=0, idle_age=342, priority=1
[openstack-dev] [Ceilometer] Meeting agenda for Thu Oct 31st at 1500 UTC
The Ceilometer project team holds a meeting in #openstack-meeting, see https://wiki.openstack.org/wiki/Meetings/MeteringAgenda for more details. Next meeting is on Thu Oct 31st at 1500 UTC Please add your name with the agenda item, so we know who to call on during the meeting. * Release python-ceilometerclient? * Open discussion If you are not able to attend or have additional topic(s) you would like to add, please update the agenda on the wiki. Cheers, -- Julien Danjou /* Free Software hacker * independent consultant http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][LBaaS] Object status and admin_state_up
Hi, I think that the current implementation is fine. This are two different aspects. The status describes whether the last a-sync activity is active or whether it is not. The admin status describes what the user wishes for the object status to be. Follows an example: If I update the VIP with admin status down, the status should be moved to PENDING_UPDATE, when the driver implements this than the status with be back to being ACTIVE. The Term ACTIVE, might be wrong in that it might be renamed to something like APPLIED. Regards, -Sam. From: Eugene Nikanorov [mailto:enikano...@mirantis.com] Sent: Tuesday, October 29, 2013 11:19 AM To: OpenStack Development Mailing List Subject: [openstack-dev] [Neutron][LBaaS] Object status and admin_state_up Hi folks, Currently there are two attributes of vips/pools/members that represent a status: 'status' and 'admin_state_up'. The first one is used to represent deployment status and can be PENDING_CREATE, ACTIVE, PENDING_DELETE, ERROR. We also have admin_state_up which could be True or False. I'd like to know your opinion on how to change 'status' attribute based on admin_state_up changes. For instance. If admin_state_up is updated to be False, how do you think 'status' should change? Also, speaking of reference implementation (HAProxy), changing vip or pool admin_state_up to False effectively destroys the balancer (undeploys it), while the objects remain in ACTIVE state. There are two options to fix this discrepancy: 1) Change status of vip/pool to PENDING_CREATE if admin_state_up changes to False 2) Don't destroy the loadbalancer and use HAProxy capability to disable frontend and backend while leave vip/pool in ACTIVE state Please share your opinion. Thanks, Eugene. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] TripleO core reviewer update - november
On 30.10.2013 10:06, Robert Collins wrote: Hi, like most OpenStack projects we need to keep the core team up to date: folk who are not regularly reviewing will lose context over time, and new folk who have been reviewing regularly should be trusted with -core responsibilities. In this months review: - James Slagle for -core +1 - Arata Notsu to be removed from -core +1 - Devananda van der veen to be removed from -core +1 Existing -core members are eligible to vote - please indicate your opinion on each of the three changes above in reply to this email. James, please let me know if you're willing to be in tripleo-core. Arata, Devananda, if you are planning on becoming substantially more active in TripleO reviews in the short term, please let us know. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Version numbering of TripleO releases
On 31.10.2013 06:09, Roman Podoliaka wrote: Hi all, 0.MAJOR.MINOR versioning totally makes sense to me until we get to 1.0.0. Just a couple of examples of releases we are doing this week: 1) tripleo-image-elements is bumped from 0.0.8 to 0.1.0 (we introduced a kind of incompatible change by switching Seed VM to neutron-dhcp-agent) 2) os-collect-config is bumped from 0.1.4 to 0.1.5 (we had Clint's patch reducing the default polling interval, which is not a bug-fix, but doesn't break backwards compatibility either) Sounds good to me. J. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Blueprint review process
Stefano Maffulli wrote: I have the feeling we keep going back to communicating expectations to new participants to the community. Are we putting too much emphasis on new commits and too little on new reviews? What do you think if from now on the weekly newsletter would mention the new first time reviewers? We have that report ready: http://activity.openstack.org/data/display/OPNSTK2/New+Contributors+First+Review+-+Last+30+Days What other sort of other incentive do you think we can give to +1ers, the reviewers that without being core, can make life so much easier and shorten the time to get the +2s? Mentioning reviews in our dev documentation (as suggested by Dan Berrange in another thread) should be the first step. You could mention first-time reviewers on an equal footing with first-time authors on the weekly newsletter, but it could quickly get noisy. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for the new service: MagnetoDB, DynamoDB API implementation for OpenStack
Ilya Sviridov wrote: I would like to propose a new initiative to implement AWS DynamoDB API for OpenStack. [...] There is a parallel effort by Soren to create an equivalent of SimpleDB: http://blog.linux2go.dk/tag/basicdb/ I'm not familiar enough with SimpleDB/DynamoDB, but I wonder if we shouldn't have a single service to handle both cases, as the backend side looks reusable to me. You guys might want to talk together. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack's OpenPGP Web of Trust
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeremy Stanley wrote: I'd love for people already using OpenPGP within our community to sign each other's keys as opportunities present themselves, since this activity strengthens and grows a healthy web of trust. A summit is a great opportunity for exchanging identification and collecting key fingerprints as many of us will be in the same rooms anyway. I just wanted to take this as an opportunity to remind everyone who's planning on being in Hong Kong next week, if you have a key and are open to some opportunistic signing then don't hesitate to approach others whom you know also use theirs and inquire whether they have a moment for you to show each other identification and exchange key fingerprints. I hope to see many of you at the summit--and of course feel free to ask me to sign your key! The summit being a bit huge it can be difficult for the PGP-minded to cross paths. We could informally designate one of the Design Summit breaks (Thursday 10:30am ?) as a keysigning break and encourage people interested to spend some time in the dev lounge during that break. - -- Thierry Carrez (ttx) -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSciu2AAoJEFB6+JAlsQQjy3AP/jrM1KVPC3yoBzvQQASvnWLr IF8ilKMYs7cbGhLGdXNoP8Dr4RcjoyDfaVwxNNpqxzQpirTSXTYeT23Mx2sbDOR+ 0dOBjECxCC3Y7JiwbNOebRB1mMY+4gUqpDJPBq9a9P/qYL3PJWrWiWeiZ4/otw1c qzuLqYaEb+I+GrbPAVzRyLs8C1Pqq65OD9FE/GEevIcqfwNcUMq8ZgMDr6fxrm1Y GF6jbAP7gSwSXT30SSPDPdESTrKhR6ir7wzWyhIr/FhXRB4uwqmjcWSSUhT8LFju 5QT/qb16/Di0IdhIuW/X9A6IAF+5pEgbKiJcQCEx6P+f3iEuGAEU/siUiABxZ0P+ Z2TmdEoMLLZk2X+nNAL5fD8Ti9QdceEcgXgHnMqc1IlbYyFHlAcrG5CFQthKh4OF eQ++TFSADUHHuUwo1490qWA4yzETUh1DRM8KzGXV0of2kcyl0znvObicJZNgYHrP 0l6Nh8vBncQimUfSvifKE0B1ywtVPLSHMv5gF8CQHCsE9Y0L4ZecKXCrALNyI+jg 3FXTP/W6qYXPInM9BAp+uos25LBC6WZAo6pqsaw/Gw1SNPR7I0B2COaT98nVPLlX uHHtOSVNnW1Um2n2XbEqhrxPepKeeU2QEA3MQWAJJSBBVNUsculu+bmD12mLX50b bw/8QTi3NPpClRDNkCBt =VZFP -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][LBaaS] Object status and admin_state_up
Hi Sam, The question was based on the bug https://bugs.launchpad.net/neutron/+bug/1242351 Where it is a real discrepancy: you bring the balancer down (and it's really undeployed) but still remains active. I guess I need to put it to PENDING_CREATE for haproxy, not PENDING_UPDATE since this represents deployment status. In fact i think this could be driver-specific and differ from driver to driver. Thanks, Eugene. On Thu, Oct 31, 2013 at 1:36 PM, Samuel Bercovici samu...@radware.comwrote: Hi, ** ** I think that the current implementation is fine. This are two different aspects. The status describes whether the last a-sync activity is active or whether it is not. The admin status describes what the user wishes for the object status to be. ** ** Follows an example: If I *update* the VIP with admin status down, the status should be moved to PENDING_UPDATE, when the driver implements this than the status with be back to being ACTIVE. The Term ACTIVE, might be wrong in that it might be renamed to something like APPLIED. ** ** Regards, -Sam. ** ** ** ** ** ** *From:* Eugene Nikanorov [mailto:enikano...@mirantis.com] *Sent:* Tuesday, October 29, 2013 11:19 AM *To:* OpenStack Development Mailing List *Subject:* [openstack-dev] [Neutron][LBaaS] Object status and admin_state_up ** ** Hi folks, ** ** Currently there are two attributes of vips/pools/members that represent a status: 'status' and 'admin_state_up'. ** ** The first one is used to represent deployment status and can be PENDING_CREATE, ACTIVE, PENDING_DELETE, ERROR. We also have admin_state_up which could be True or False. ** ** I'd like to know your opinion on how to change 'status' attribute based on admin_state_up changes. For instance. If admin_state_up is updated to be False, how do you think 'status' should change? ** ** Also, speaking of reference implementation (HAProxy), changing vip or pool admin_state_up to False effectively destroys the balancer (undeploys it), while the objects remain in ACTIVE state. There are two options to fix this discrepancy: 1) Change status of vip/pool to PENDING_CREATE if admin_state_up changes to False 2) Don't destroy the loadbalancer and use HAProxy capability to disable frontend and backend while leave vip/pool in ACTIVE state ** ** Please share your opinion. ** ** Thanks, Eugene. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack's OpenPGP Web of Trust
On Thu, Oct 31, 2013 at 8:36 PM, Thierry Carrez thie...@openstack.orgwrote: The summit being a bit huge it can be difficult for the PGP-minded to cross paths. We could informally designate one of the Design Summit breaks (Thursday 10:30am ?) as a keysigning break and encourage people interested to spend some time in the dev lounge during that break. That sounds great. Having a specific time set aside would help as people are more likely to bring along stronger ID such as a passport which they wouldn't otherwise normally carry with them. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack's OpenPGP Web of Trust
Thu 10:30 AM - Design Summit Lounge - sounds like a great plan! -- dims On Thu, Oct 31, 2013 at 6:39 AM, Christopher Yeoh cbky...@gmail.com wrote: On Thu, Oct 31, 2013 at 8:36 PM, Thierry Carrez thie...@openstack.org wrote: The summit being a bit huge it can be difficult for the PGP-minded to cross paths. We could informally designate one of the Design Summit breaks (Thursday 10:30am ?) as a keysigning break and encourage people interested to spend some time in the dev lounge during that break. That sounds great. Having a specific time set aside would help as people are more likely to bring along stronger ID such as a passport which they wouldn't otherwise normally carry with them. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack's OpenPGP Web of Trust
On 31/10/13 11:06 +0100, Thierry Carrez wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeremy Stanley wrote: I'd love for people already using OpenPGP within our community to sign each other's keys as opportunities present themselves, since this activity strengthens and grows a healthy web of trust. A summit is a great opportunity for exchanging identification and collecting key fingerprints as many of us will be in the same rooms anyway. I just wanted to take this as an opportunity to remind everyone who's planning on being in Hong Kong next week, if you have a key and are open to some opportunistic signing then don't hesitate to approach others whom you know also use theirs and inquire whether they have a moment for you to show each other identification and exchange key fingerprints. I hope to see many of you at the summit--and of course feel free to ask me to sign your key! The summit being a bit huge it can be difficult for the PGP-minded to cross paths. We could informally designate one of the Design Summit breaks (Thursday 10:30am ?) as a keysigning break and encourage people interested to spend some time in the dev lounge during that break. +1 -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Use cases: Cloud Environment Deployment
Hi, Another very important Mistral use case is published on the project wiki: https://wiki.openstack.org/wiki/Mistral/Cloud_Environment_Deployment_details For convenience, below is the description of it. Please let us know your thoughts on additional things that should be taken into account. We invite everyone to the discussion. “ Cloud Environment Deployment Problem Statement Cloud environment deployment procedure typically involves many steps that include infrastructure provision (instances, networks etc) and software installation and configuration. Those actions often depend on each other and thus need to be performed in some specific order. Also this process often involves more than one tool. This is a perfect fit for Mistral tasks-dependencies model. There are several approaches how cloud environment deployment can be addressed in OpenStack with Heat as a software orchestration tool. Solution A. External approach With this approach each task in tasks graph corresponds to some deployment action. Those may be infrastructure related tasks (like create instance) and software related task (like execute shell script on VM via SSH). Because we have Heat in OpenStack all infrastructure tasks can be done via addition (merging) of additional parts to Heat HOT templates. This can be achieved by built-in Mistral’s data flow capabilities. Besides HOT template building there must be also explicit task to create/update Heat stack with built template. Software configuration actions can be performed by execution of shell scripts, puppet manifests etc. on VM. Mistral’s data flow can help with passing data between software configuration tasks. Because Mistral is going to be integrated with Software Configuration Management (SCM) tools and because it can natively access various OpenStack REST APIs all of the tasks mentioned above can be performed by Mistral alone without a need for additional software to handle callback actions. The graph itself can be created either manually by a user of auto-generated using some third party tool. This can be useful for cloud deployment/management tools like PaaSes, software catalogs and other similar tools that wish to take advantage of Heat and SCM tools integration. Solution B. Internal approach Heat itself can use Mistral for processing HOT templates. With this approach components and resources directly map to Mistral tasks and Fn::GetAtt/hosten_on map to task dependencies. Handling of resource/component deployment remains in Heat but the order and context passing between those activities are delegated to Mistral. SCM integration may be done using Heat components but users also have ability to hook custom workflows to Heat-generated task graph. Using Mistral for task-dependency tracking and context passing as well as task distribution may be a basis for Heat engine HA and further scalability. This also opens new possibilities for integration of Heat and 3rd party tools. It is also possible to have both approaches simultaneously or some mix of them working together. Renat Akhmerov @ Mirantis Inc.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack's OpenPGP Web of Trust
Flavio Percoco wrote: On 31/10/13 11:06 +0100, Thierry Carrez wrote: The summit being a bit huge it can be difficult for the PGP-minded to cross paths. We could informally designate one of the Design Summit breaks (Thursday 10:30am ?) as a keysigning break and encourage people interested to spend some time in the dev lounge during that break. +1 OK, made it official: http://icehousedesignsummit.sched.org/event/435822fb3258e31075c8b57d2d57dbf0 -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] changing old migrations is verboten
So there is a series of patches starting with - https://review.openstack.org/#/c/53417/ that go back and radically change existing migration files. This is really a no-no, unless there is a critical bug fix that absolutely requires it. Changing past migrations should be considered with the same level of weight as an N-2 backport, only done when there is huge upside to the change. I've -2ed the first 2 patches in the series, though that review applies to all of them (I figured a mailing list thread was probably more useful than -2ing everything in the series). There needs to be really solid discussion about the trade offs here before contemplating something as dangerous as this. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
Hi all, This is a bit of a social norms thread I've been consistently asking for tests in reviews for a while now, and I get the occasional push-back. I think this falls into a few broad camps: A - there is no test suite at all, adding one in unreasonable B - this thing cannot be tested in this context (e.g. functional tests are defined in a different tree) C - this particular thing is very hard to test D - testing this won't offer benefit E - other things like this in the project don't have tests F - submitter doesn't know how to write tests G - submitter doesn't have time to write tests Now, of these, I think it's fine not add tests in cases A, B, C in combination with D, and D. In my opinion C instead of being an acceptable reason for not having tests is a symptom of one of the two things: 1) F = Submitter doesn't know how to write tests, in this case someone else can help with suggestions 2) The code we are trying to test is too complicated so it's time to refactor it And about D, In my opinion tests always offer benefits, like code coverage or helping in understanding the code. What do you think? Regards -- Andrea Rosa ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Duplicated test effort development
a huge +1! thanks, chris, noticed the spreadsheet before, and have add some contents I have been working on. I think more people should know that, avoid conflicting. 2013/10/31 Zhu Bo bo...@linux.vnet.ibm.com hi, Chris thanks for your work. It's a good way. And how about creating a blue-print and putting these link on it? Some guys have been familiar with looking for work items from blue-print, and writing something in hacking guide may be better. On 2013年10月30日 21:10, Christopher Yeoh wrote: On Wed, Oct 30, 2013 at 11:18 PM, Steven Hardy sha...@redhat.com wrote: On Wed, Oct 30, 2013 at 11:00:22PM +1030, Christopher Yeoh wrote: snip I don't think blueprints/bugs work so well at this, and I don't think we have anything else setup at the moment, so as a temporary measure I've created an etherpad here: Can I ask why? Surely blueprints for new features (in this case the feature is test coverage for $api) are exactly what the normal openstack process dictates, and is what most folks are familiar with? Just to be clear - Its not that I think that we shouldn't have blueprints which covers the work being done (we should!), but they don't work so well at allowing people to see a good summary of what test coverage for an API we have (some of which may have been done a long time ago), what needs to be done and the quite fine grained allocation of what people are working on. For example, see the tempest coverage for the Nova v2 API spreadsheet: https://docs.google.com/spreadsheet/ccc?key=0AmYuZ6T4IJETdEVNTWlYVUVOWURmOERSZ0VGc1BBQWc#gid=0 Anyway, I added the keystone test I'm working on (which has a BP) to the etherpad, and definitely +1 on not duplicating effort, by whatever means ;) Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- *---* *Lingxian Kong* Huawei Technologies Co.,LTD. IT Product Line CloudOS PDU China, Xi'an Mobile: +86-18602962792 Email: konglingx...@huawei.com; anlin.k...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they might
be optional.
I totally agree with the sentiment but, especially when it's a newcomer
to the project, I try to put myself in the shoes of the patch submitter
and double-check whether what we're asking is reasonable.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug - think
typo like raise NotFund('foo') - and testing this code patch requires
more than adding a simple new scenario to existing tests ...
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project doesn't
have full test coverage to begin with really makes us seem unwelcoming.
In cases like that, I'd be of a mind to go +2 Awesome! Thanks for
catching this! It would be great to have a unit test for this, but it's
clear the current code is broken so I'm fine with merging the fix
without a test. You could say it's now the reviewers responsibility to
merge a test, but if that requirement then turns off reviewers even
reviewing such a patch, then that doesn't help either.
So, with all of this, let's make sure we don't forget to first
appreciate the effort that went into submitting the patch that lacks
tests.
Mark.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] An early peek into the Solum.io website
Joshua, Good question. It's a project specific initiative, rather than an OpenStack Foundation effort. One of the interesting aspects of the OpenStack ecosystem is that each OpenStack Related project can work individually on growing its community, which in turn feeds the OpenStack ecosystem. If this approach is successful, it could be replicated for other projects as well. We feel this is an important thing for Solum because of its open from the beginning approach. This is less important for projects that come to openstack after they have taken comprehensive form. Adrian On Oct 31, 2013, at 12:04 AM, Joshua Harlow harlo...@yahoo-inc.com wrote: Just wondering, Will said infrastructure be made available to all other openstack projects? It'd be nice to have that for other projects to (a nova blog, a taskflow blog...). It seems/feels a little awkward to me to have solum be a special snowflake here. Sent from my really tiny device... On Oct 30, 2013, at 9:55 PM, Adrian Otto adrian.o...@rackspace.com wrote: Noorul, Yes, it uses WordPress. The community website will be used primarily as a rally point for attracting new contributors. It will have a blog feature which is currently being configured so the project contributors can publish related content there. We expect to use this site to attract new developers who are not already part of the OpenStack ecosystem. Adrian On Oct 30, 2013, at 9:25 PM, Noorul Islam K M noo...@noorul.com wrote: Roshan Agrawal roshan.agra...@rackspace.com writes: The Solum community website is very close to be launched publicly. If you want to take an early peek into how it is coming along, here are the access details: www.Solum.iohttp://www.Solum.io User name: Solum, password: OpenStack The Solum logo is still in works, what we have now is meant to be a placeholder till we finalize on an awesome looking logo. Comments/suggestions welcome. (Cc Solum list for now, till everyone have had a chance to migrate to the openstack list) Is there any dynamic content in this site? Why are we not utilizing github pages or something that can be generated from git repository? Thanks and Regards Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [horizon / keystone] Marker could not be found?
Hello Folks.
I have a problem after grizzly-havana migration where i’m unable to rescue
myself.
When I open the Admin - Resource-Usage View i get no results – only a red
error box with the message Error: Unable to retrieve tenant list.“.
Horizon log:
[Thu Oct 31 11:39:44 2013] [error] Creating a new keystoneclient connection to
http://$controller:35357/v2.0.
[Thu Oct 31 11:39:44 2013] [error] REQ: curl -i -X GET
http://$controller:35357/v2.0/tenants?marker=tenant_markerlimit=21 -H
User-Agent: python-keystoneclient -H Forwarded:
for=131.234.5.178;by=python-keystoneclient -H X-Auth-Token: 82[…]f46
[Thu Oct 31 11:39:44 2013] [error] REQ: curl -i -X GET
http://$controller:35357/v2.0/tenants?marker=tenant_markerlimit=21 -H
User-Agent: python-keystoneclient -H Forwarded:
for=131.234.5.178;by=python-keystoneclient -H X-Auth-Token: 82[…]46
[Thu Oct 31 11:39:44 2013] [error] INFO:urllib3.connectionpool:Starting new
HTTP connection (1): $controller
[Thu Oct 31 11:39:44 2013] [error] DEBUG:urllib3.connectionpool:GET
/v2.0/tenants?marker=tenant_markerlimit=21 HTTP/1.1 400 88
[Thu Oct 31 11:39:44 2013] [error] RESP: [400] CaseInsensitiveDict({'date':
'Thu, 31 Oct 2013 11:39:47 GMT', 'vary': 'X-Auth-Token', 'content-length':
'88', 'content-type': 'application/json'})
[Thu Oct 31 11:39:44 2013] [error] RESP BODY: {error: {message: Marker
could not be found, code: 400, title: Bad Request}}
[Thu Oct 31 11:39:44 2013] [error]
[Thu Oct 31 11:39:44 2013] [error] RESP: [400] CaseInsensitiveDict({'date':
'Thu, 31 Oct 2013 11:39:47 GMT', 'vary': 'X-Auth-Token', 'content-length':
'88', 'content-type': 'application/json'})
[Thu Oct 31 11:39:44 2013] [error] RESP BODY: {error: {message: Marker
could not be found, code: 400, title: Bad Request}}
[Thu Oct 31 11:39:44 2013] [error]
[Thu Oct 31 11:39:44 2013] [error] Request returned failure status: 400
[Thu Oct 31 11:39:44 2013] [error] Request returned failure status: 400
[Thu Oct 31 11:39:44 2013] [error] Recoverable error: Marker could not be found
(HTTP 400)
Keystone Log:
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Matched GET /tenants
__call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.contrib.ec2.routers.Ec2Extension object at 0x4156f10} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.contrib.ec2.routers.Ec2Extension object at 0x4156f10,
'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Matched GET /tenants
__call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.contrib.s3.core.S3Extension object at 0x4156cd0} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.contrib.s3.core.S3Extension object at 0x4156cd0,
'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.354 17187 DEBUG routes.middleware [-] Matched GET /tenants
__call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.354 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.contrib.admin_crud.core.CrudExtension object at 0x41517d0} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.contrib.admin_crud.core.CrudExtension object at
0x41517d0, 'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Matched GET /tenants
__call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.common.wsgi.ComposingRouter object at 0x4151e50} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.356 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.common.wsgi.ComposingRouter object at 0x4151e50,
'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.356 17187 DEBUG routes.middleware [-] Matched GET /tenants
__call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.357 17187 DEBUG routes.middleware [-] Route path:
'/tenants', defaults: {'action': u'get_all_projects', 'controller':
keystone.identity.controllers.Tenant object at 0x4151ed0} __call__
Re: [openstack-dev] Keystone Concurrency Races in SQL Assignment Backend
Peter - We discussed better use of transactions in irc, but I don't think anyone has had a chance to look at it. This would be a very useful thing to have someone look at. I'm fine with holding off on the oslo.db sessions work until we're sure the code is correct w/r/t multi-processing so that tempest is going to pass consistently. - Brant On Wed, Oct 30, 2013 at 5:08 PM, Peter Feiner pe...@gridcentric.ca wrote: Hi Brant, In addition to the race you've fixed in https://review.openstack.org/#/c/50767/, it looks like there are quite a few more races in the SQL backend of keystone.assignment. I filed a bug to this effect: https://bugs.launchpad.net/keystone/+bug/1246489. The general problem is that transactions are used somewhat indiscriminately. The fix (i.e., using transactions judiciously) is straightforward and should be mostly independent of your ongoing oslo.db sessions port in https://review.openstack.org/#/c/49460/. So, unless you already have something in the works, I'll get started on that tomorrow. I'm eager to fix these races so https://review.openstack.org/#/c/42967/ can reliably pass tempest :-) Peter ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
Excerpts from Mark McLoughlin's message of 2013-10-31 06:30:32 -0700:
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they might
be optional.
I totally agree with the sentiment but, especially when it's a newcomer
to the project, I try to put myself in the shoes of the patch submitter
and double-check whether what we're asking is reasonable.
Even with a long time contributor, empathy is an important part of
constructing reviews. We could make more robotic things that review for
test coverage, but we haven't, because this is a gray area. The role of
a reviewer isn't just get patches merged and stop defects. It is also to
grow the other developers.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug - think
typo like raise NotFund('foo') - and testing this code patch requires
more than adding a simple new scenario to existing tests ...
This goes back to my recent suggestion to help the person not with a -1
or a +2, but with an additional patch that fixes it.
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project doesn't
have full test coverage to begin with really makes us seem unwelcoming.
In cases like that, I'd be of a mind to go +2 Awesome! Thanks for
catching this! It would be great to have a unit test for this, but it's
clear the current code is broken so I'm fine with merging the fix
without a test. You could say it's now the reviewers responsibility to
merge a test, but if that requirement then turns off reviewers even
reviewing such a patch, then that doesn't help either.
I understand entirely why you choose this, and I think that is fine. I,
however, see this as a massive opportunity to teach. That code was only
broken because it was allowed it to be merged without tests. By letting
that situation continue, we only fix it today. The next major refactoring
has a high chance now of breaking that part of the code again.
So, rather than +2, I suggest -1 with compassion. Engage with the
submitter. If you don't know them, take a look at how hard it would be
to write a test for the behavior and give pointers to the exact test
suite that would need to be changed, or suggest a new test suite and
point at a good example to copy.
So, with all of this, let's make sure we don't forget to first
appreciate the effort that went into submitting the patch that lacks
tests.
I'm not going to claim that I've always practiced -1 with compassion,
so thanks for reminding us all that we're not just reviewing code, we
are having a dialog with real live people.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Uri, Yes. We will need to offer a facility for customizing the CD pipeline(s). We have not contemplated all of the possibilities for that yet, so this is something that we will certainly discuss further. We will need to strike the right balance between options and simplicity. We have thought through the possibility of generating a HOT[1] artifact in the case that one is not provided, and allowing a user-supplied one in the event that one is supplied with the code at the time of the deployment action. Of course if no HOT is provided, we can generate one (or a simple hierarchy of them). That option would allow you to cause anything that Heat knows how to do. We will continue to contribute to Heat (and other downstream projects) to help it achieve its mission. From my view, Heat should allow us to connect various different imperative systems downstream. For example, if you want something to happen on a given node type, you might use Chef or Puppet to accomplish that. Features related to orchestration (the things offered in the HOT as declarations) should be provided by heat, or one of its resource types. That's where I see most features like rollback fitting in. The actual design of the logic that allows the promotion of a deployment through different Environments[2] will likely be triggered by Solum, but implemented downstream in Heat. A system level rollback (user requested, rather than initiated by a failure) may actually be achieved by a request to swap Environments, or modify one. Adrian References: [1]: https://wiki.openstack.org/wiki/Heat/DSL#Heat_Orchestration_Template_.28HOT.29 [2]: https://wiki.openstack.org/wiki/Heat/DSL#Environment On Oct 31, 2013, at 2:04 AM, Uri Cohen u...@gigaspaces.com wrote: Hi Adrian Sounds like there's a need for a true workflow engine here. I suspect that for any use case beyond trivial, each user will have her own flavor of CD (TripleO is a good example for this) - some will prefer to use a canary server and test it manually before deploying to other servers, some will deploy to specific servers based on the subset of users they're serving, some will do a red/black type of deployment (Which maps well into environments below), etc. But I think that there should be an user-configurable mechanism to define and tweak these workflows. There will also be additional related workflows - e.g. rolling back a deployment, or updating the infrastructure components of a running application (e.g. database version upgrade). I might be raising a topic that's already been discussed (sorry if I am), but I was wondering how this would fit into the architecture. Thanks! Uri On Oct 31, 2013, at 6:42 AM, Adrian Otto adrian.o...@rackspace.com wrote: Robert, We would be interested in exploring your use case for CD, and help you judge what would be the best fit. I know that Monty Taylor has identified some parts of openstack-infra that we could potentially leverage in Solum to address our CI/CD goals. The design of that feature is still pending, so we are looking forward to sourcing its of input on that. I don't anticipate any fundament gaps that would prevent you from using a future version of Solum to help narrow your scope. The whole point of our focus is to try to make these goals easier to achieve, and try to get maximum leverage of what the OpenStack ecosystem already offers. The high level concept behind the GIt Integration feature currently conceived for Solum is to offer application developers a way to commit code to a Git repo and upon a git push command, have a remote system trigger a modular CI/CD pipeline that ends with that application running on an OpenStack cloud in one or more environments (dev/test/stage/prod/etc.). In a simple case that may be (in the case of a dynamic language like Python) the testing of the code, arrangement of the commit into a deployable bundle, triggering of the Solum API to trigger a deployment, automatic generation of a HOT artifact, and stimulation of the Heat API and any other API calls needed. In a more complex scenario, you may want additional features and capabilities, such as a system like Gerrit to trigger the commit/push, integrate an artifact repository for storing the builds, etc. Using Zuul for that would be appropriate for sure. There are also dev shops who already have a lot invested into their Jenkins setups, and will want to bypass portions of what I described above, so there will be a use case supported where you can just integrate with the solum API, and pass in a pre-built bundle artifact and bypass part or all of Solum's CI features, and just use it for deployment and re-deployment and promotion of a deployment between various environments (dev/test/pros/etc.). You would expect features like canary deploy in that case that might make it more attractive than just generating your own HOT artifact in your
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
On 31 окт. 2013 г., at 2:37, Clint Byrum cl...@fewbar.com wrote: My point is really that we should not care how serialization happens, we should just express the work-flow, and let the underlying mechanisms distribute and manage it as it is completed. Sounds reasonable. In this context, you may want to look at Mistral project we recently started. We just published a kind of a high-level description of the use case that is relevant for the problem that is being discussed here. It’s called Cloud Environment Deployment and you can find it at https://wiki.openstack.org/wiki/Mistral/Cloud_Environment_Deployment_details. We think it can be a very important application for Mistral. Any kinds of locking mechanism, imho, should be avoiding at all system levels unless it’s absolutely required for algorithm complexity reasons (when there’s no other way). So If we can represent what Heat does in its internals as a set of related tasks we can offload dependencies resolution to a system like Mistral that would do everything in a distributed manner. Another interesting feature we’re planning to implement is data flow. That is, some state (sort of a context) associated with a workflow travels between nodes in a task graph so there’s no need to worry about a shared state in many cases. Data transfer itself is supposed to work on top of some HA transport itself (like rabbit) so it shouldn’t be a challenge to implement it. Not 100% sure that it’s all applicable for solving this Heat task (pardon me), but it definitely could be considered as a possibility. Unfortunately, we’ve not described this feature very well yet (only some pictures and sketches not published anywhere), but we’ll do soon. I don’t really want it to look like an ad, sorry if it does ( :) ). It would be cool if we could collaborate on this. Once you look at our ideas, you could provide your input on what else should be taken into account in Mistral design in order to address your problem well. Renat ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On Oct 31, 2013, at 8:56 AM, Clint Byrum cl...@fewbar.com wrote:
Excerpts from Mark McLoughlin's message of 2013-10-31 06:30:32 -0700:
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they might
be optional.
I totally agree with the sentiment but, especially when it's a newcomer
to the project, I try to put myself in the shoes of the patch submitter
and double-check whether what we're asking is reasonable.
Even with a long time contributor, empathy is an important part of
constructing reviews. We could make more robotic things that review for
test coverage, but we haven't, because this is a gray area. The role of
a reviewer isn't just get patches merged and stop defects. It is also to
grow the other developers.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug - think
typo like raise NotFund('foo') - and testing this code patch requires
more than adding a simple new scenario to existing tests ...
This goes back to my recent suggestion to help the person not with a -1
or a +2, but with an additional patch that fixes it.
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project doesn't
have full test coverage to begin with really makes us seem unwelcoming.
In cases like that, I'd be of a mind to go +2 Awesome! Thanks for
catching this! It would be great to have a unit test for this, but it's
clear the current code is broken so I'm fine with merging the fix
without a test. You could say it's now the reviewers responsibility to
merge a test, but if that requirement then turns off reviewers even
reviewing such a patch, then that doesn't help either.
I understand entirely why you choose this, and I think that is fine. I,
however, see this as a massive opportunity to teach. That code was only
broken because it was allowed it to be merged without tests. By letting
that situation continue, we only fix it today. The next major refactoring
has a high chance now of breaking that part of the code again.
So, rather than +2, I suggest -1 with compassion. Engage with the
submitter. If you don't know them, take a look at how hard it would be
to write a test for the behavior and give pointers to the exact test
suite that would need to be changed, or suggest a new test suite and
point at a good example to copy.
So, with all of this, let's make sure we don't forget to first
appreciate the effort that went into submitting the patch that lacks
tests.
I'm not going to claim that I've always practiced -1 with compassion,
so thanks for reminding us all that we're not just reviewing code, we
are having a dialog with real live people.
I think this is the key thing here, thanks for bringing this up Clint. At the
end of the day, patches are submitted by real people. If we want to grow
the committer base and help people to become better reviewers, taking
the time to show them the ropes is part of the game. I'd argue that is in
fact part of what being a core is about in fact.
Thanks,
Kyle
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Full schedule for HongKong summit?
On 10/29/2013 06:29 AM, Thierry Carrez wrote: Dina Belova wrote: There is no such mixed calendar. It was made, as far as I remember, because design and summit talks are very different and should be separated. But still, it is not really comfortable. Yes, it's a bit of a trade-off... In the past we had a lot of people, attracted by catchy session titles, wandering in the design summit area and be really surprised by the absence of presentations and the chaotic nature of the discussions there. For this time around we decided that further separating the two events (while keeping compatible time slots) was less confusing, and keeping two separate schedules was the simplest way of making that happen. I agree it's inconvenient for people who want to juggle between the two parts of the event. The best solution for those is probably to look at both schedules and use the iCal/ics exports to build your personal calendar: http://icehousedesignsummit.sched.org/mobile-site http://openstacksummitnovember2013.sched.org/mobile-site Hope this helps, I would think that this is a common enough occurance for the sched.org people that they would have a nested conference mechanism. However, I think that this might have a secondary effect of messing up people that really do want to participate in the design sessions, but are only aware of the main summit schedule. While it is true that we don't want the design sessions to be flooded, we do want participation, and balancing the two is going to allways err one way or the other. Security is particularly impacted by this, however, as developers really should be at the security sessions, but they are not part of the developers schedule. At a minimum, we should probably have a recurring theme among the early day presenters if you are interested in the design sessions, remember that they are on a separate schedule or some such. Dealing with this really feels like dealing with design issues in Keystone. What user should see which project... ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] An early peek into the Solum.io website
Adrian ++. I too believe its a very good initiative started with #Solum to follow up. Other projects (existing/incubating/future) will definitely have a fruitful benefit following similar approach in the long run, thus benefiting the OpenStack ecosystem :) Best Regards, Swapnil On Thu, Oct 31, 2013 at 7:07 PM, Adrian Otto adrian.o...@rackspace.comwrote: Joshua, Good question. It's a project specific initiative, rather than an OpenStack Foundation effort. One of the interesting aspects of the OpenStack ecosystem is that each OpenStack Related project can work individually on growing its community, which in turn feeds the OpenStack ecosystem. If this approach is successful, it could be replicated for other projects as well. We feel this is an important thing for Solum because of its open from the beginning approach. This is less important for projects that come to openstack after they have taken comprehensive form. Adrian On Oct 31, 2013, at 12:04 AM, Joshua Harlow harlo...@yahoo-inc.com wrote: Just wondering, Will said infrastructure be made available to all other openstack projects? It'd be nice to have that for other projects to (a nova blog, a taskflow blog...). It seems/feels a little awkward to me to have solum be a special snowflake here. Sent from my really tiny device... On Oct 30, 2013, at 9:55 PM, Adrian Otto adrian.o...@rackspace.com wrote: Noorul, Yes, it uses WordPress. The community website will be used primarily as a rally point for attracting new contributors. It will have a blog feature which is currently being configured so the project contributors can publish related content there. We expect to use this site to attract new developers who are not already part of the OpenStack ecosystem. Adrian On Oct 30, 2013, at 9:25 PM, Noorul Islam K M noo...@noorul.com wrote: Roshan Agrawal roshan.agra...@rackspace.com writes: The Solum community website is very close to be launched publicly. If you want to take an early peek into how it is coming along, here are the access details: www.Solum.iohttp://www.Solum.io User name: Solum, password: OpenStack The Solum logo is still in works, what we have now is meant to be a placeholder till we finalize on an awesome looking logo. Comments/suggestions welcome. (Cc Solum list for now, till everyone have had a chance to migrate to the openstack list) Is there any dynamic content in this site? Why are we not utilizing github pages or something that can be generated from git repository? Thanks and Regards Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
Hi all,
As a newbie of the community, I'm not familiar with unittest and how to use it
here. I've learned that Jenkins runs tests
everytime we submit some code. But how to write the test and what is a 'good
test' and a 'bad test'? I saw some commits
in gerrit but am unable to say if the written test is enough to judge the code,
since it is the author of the code who writes
the test. Is there a framework to follow or some rules/pratices to respect?
Do you have some links to help me out?
Thanks,
Toan
- Original Message -
From: Kyle Mestery (kmestery) kmest...@cisco.com
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Sent: Thursday, October 31, 2013 3:05:27 PM
Subject: Re: [openstack-dev] When is it okay for submitters to say 'I don't
want to add tests' ?
On Oct 31, 2013, at 8:56 AM, Clint Byrum cl...@fewbar.com wrote:
Excerpts from Mark McLoughlin's message of 2013-10-31 06:30:32 -0700:
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they might
be optional.
I totally agree with the sentiment but, especially when it's a newcomer
to the project, I try to put myself in the shoes of the patch submitter
and double-check whether what we're asking is reasonable.
Even with a long time contributor, empathy is an important part of
constructing reviews. We could make more robotic things that review for
test coverage, but we haven't, because this is a gray area. The role of
a reviewer isn't just get patches merged and stop defects. It is also to
grow the other developers.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug - think
typo like raise NotFund('foo') - and testing this code patch requires
more than adding a simple new scenario to existing tests ...
This goes back to my recent suggestion to help the person not with a -1
or a +2, but with an additional patch that fixes it.
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project doesn't
have full test coverage to begin with really makes us seem unwelcoming.
In cases like that, I'd be of a mind to go +2 Awesome! Thanks for
catching this! It would be great to have a unit test for this, but it's
clear the current code is broken so I'm fine with merging the fix
without a test. You could say it's now the reviewers responsibility to
merge a test, but if that requirement then turns off reviewers even
reviewing such a patch, then that doesn't help either.
I understand entirely why you choose this, and I think that is fine. I,
however, see this as a massive opportunity to teach. That code was only
broken because it was allowed it to be merged without tests. By letting
that situation continue, we only fix it today. The next major refactoring
has a high chance now of breaking that part of the code again.
So, rather than +2, I suggest -1 with compassion. Engage with the
submitter. If you don't know them, take a look at how hard it would be
to write a test for the behavior and give pointers to the exact test
suite that would need to be changed, or suggest a new test suite and
point at a good example to copy.
So, with all of this, let's make sure we don't forget to first
appreciate the effort that went into submitting the patch that lacks
tests.
I'm not going to claim that I've always practiced -1 with compassion,
so thanks for reminding us all that we're not just reviewing code, we
are having a dialog with real live people.
I think this is the key thing here, thanks for bringing this up Clint. At the
end of the day, patches are submitted by real people. If we want to grow
the committer base and help people to become better reviewers, taking
the time to show them
Re: [openstack-dev] Keystone Concurrency Races in SQL Assignment Backend
Okay Brant, sounds good. I'll start working on the SQL code today. Please bear in mind that I'm going to be on vacation next week, so a patch won't be ready until some time after November 11. On Thu, Oct 31, 2013 at 9:46 AM, Brant Knudson b...@acm.org wrote: Peter - We discussed better use of transactions in irc, but I don't think anyone has had a chance to look at it. This would be a very useful thing to have someone look at. I'm fine with holding off on the oslo.db sessions work until we're sure the code is correct w/r/t multi-processing so that tempest is going to pass consistently. - Brant On Wed, Oct 30, 2013 at 5:08 PM, Peter Feiner pe...@gridcentric.ca wrote: Hi Brant, In addition to the race you've fixed in https://review.openstack.org/#/c/50767/, it looks like there are quite a few more races in the SQL backend of keystone.assignment. I filed a bug to this effect: https://bugs.launchpad.net/keystone/+bug/1246489. The general problem is that transactions are used somewhat indiscriminately. The fix (i.e., using transactions judiciously) is straightforward and should be mostly independent of your ongoing oslo.db sessions port in https://review.openstack.org/#/c/49460/. So, unless you already have something in the works, I'll get started on that tomorrow. I'm eager to fix these races so https://review.openstack.org/#/c/42967/ can reliably pass tempest :-) Peter ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
On 10/30/2013 08:08 PM, Steven Dake wrote: On 10/30/2013 12:20 PM, Sandy Walsh wrote: On 10/30/2013 03:10 PM, Steven Dake wrote: I will -2 any patch that adds zookeeper as a dependency to Heat. Certainly any distributed locking solution should be plugin based and optional. Just as a database-oriented solution could be the default plugin. Sandy, Even if it is optional, some percentage of the userbase will enable it and expect the Heat community to debug and support it. But, that's the nature of every openstack project. I don't support HyperV in Nova or HBase in Ceilometer. The implementers deal with that support. I can help guide someone to those people but have no intentions of standing up those environments. Re: the Java issue, we already have optional components in other languages. I know Java is a different league of pain, but if it's an optional component and left as a choice of the deployer, should we care? -S PS As an aside, what are your issues with ZK? I realize zookeeper exists for a reason. But unfortunately Zookeeper is a server, rather then an in-process library. This means someone needs to figure out how to document, scale, secure, and provide high availability for this component. Yes, that's why we would use it. Same goes for rabbit and mysql. This is extremely challenging for the two server infrastructure components OpenStack server processes depend on today (AMQP, SQL). If the entire OpenStack community saw value in biting the bullet and accepting zookeeper as a dependency and taking on this work, I might be more ameniable. Why do other services need to agree on adopting ZK? If some Heat users need it, they can use it. Nova shouldn't care. What we are talking about in the review, however, is that the Heat team bite that bullet, which is a big addition to the scope of work we already execute for the ability to gain a distributed lock. I would expect there are simpler approaches to solve the problem without dragging the baggage of a new server component into the OpenStack deployment. Yes, there probably are, and alternatives are good. But, as others have attested, ZK is tried and true. Why not support it also? Using zookeeper as is suggested in the review is far different then the way Nova uses Zookeeper. With the Nova use case, Nova still operates just dandy without zookeeper. With zookeeper in the Heat usecase, it essentially becomes the default way people are expected to deploy Heat. Why, if it's a plugin? What I would prefer is taskflow over AMQP, to leverage existing server infrastructure (that has already been documented, scaled, secured, and HA-ified). Same problem exists, we're just pushing the ZK decision to another service. Regards -steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
On 10/31/2013 11:43 AM, Monty Taylor wrote: Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. Monty, is that just because the stack is very complicated now, or something personal against ZK (or Java specifically)? Curious. -S ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] changing old migrations is verboten
On 10/31/2013 08:01 AM, Sean Dague wrote: So there is a series of patches starting with - https://review.openstack.org/#/c/53417/ that go back and radically change existing migration files. This is really a no-no, unless there is a critical bug fix that absolutely requires it. Changing past migrations should be considered with the same level of weight as an N-2 backport, only done when there is huge upside to the change. I've -2ed the first 2 patches in the series, though that review applies to all of them (I figured a mailing list thread was probably more useful than -2ing everything in the series). There needs to be really solid discussion about the trade offs here before contemplating something as dangerous as this. +1 -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Comments on Steve Baker's Proposal on HOT Software Config
-Mike Spreitzer/Watson/IBM@IBMUS wrote: -To: "OpenStack Development Mailing List \(not for usage questions\)"openstack-dev@lists.openstack.orgFrom: Mike Spreitzer/Watson/IBM@IBMUSDate: 10/30/2013 03:56PMSubject: Re: [openstack-dev] [Heat] Comments on Steve Baker'sProposal on HOT Software ConfigLakshminaraya Renganarayana/Watson/IBM@IBMUS wrote on10/30/2013 03:35:32 PM: Zane Bitter zbit...@redhat.com wrote on 10/29/2013 08:46:21AM: ... In this method (i.e. option (2) above) shouldn't we be building the dependencygraph in Heat rather than running through them sequentially as specifiedby the user? In that case, we should use a dictionary not a list: app_server:type: OS::Nova::Serverproperties: components: install_user_profile: definition: InstallWasProfile params:user_id install_admin_profile: definition: InstallWasProfile params:admin_id I missed this implication of using a list! You are right, it shouldbe a dictionary and Heat would be building the dependence graph. Using a dictionary instead of a list can work, butI think we might be going overboard here. Do we expect the componentinvocations on a given VM instance to run concurrently? I think thathas been dissed before. Chef users are happy to let a role be a listof recipes, not a DAG. A list is simple; is there an actual problemwith it?Yes, there was some agreement on component invocations on a given VM instancebeing run sequentially. However, the issue here is slightly different. If, as adesign principle, Heat analyzes dependences between component invocations,then it should do that irrespective of whether the component invocations areon the same VM or from different VMs. Given this, a list of component invocationswould imply an ordering which is in addition to the ordering induced by dependences.Whereas a dictionary would not impose any additional ordering.Thanks,LN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][LBaaS] Thursday meeting follow-up
Hi, I have created two document to discuss SSL termination and L7 Rules at: SSL termination : https://docs.google.com/document/d/1qnoJLD1txY5wnjx4k480AtEGCOEtkPMvTzxPo3_DPcs/edit?usp=sharing SSL BP: https://blueprints.launchpad.net/neutron/+spec/lbaas-ssl-termination L7 Rules: https://docs.google.com/document/d/1Dm6fWxHmsgO6gXFWaYf-Ce36vYpMB0L13ASrB_BZ0Bc/edit?usp=sharing L7 BP: https://blueprints.launchpad.net/neutron/+spec/lbaas-l7-rules Please feel free to comment on the documents. We can also discuss at the summit. Regards, -Sam. From: Eugene Nikanorov [mailto:enikano...@mirantis.com] Sent: Friday, October 25, 2013 6:09 PM To: OpenStack Development Mailing List Subject: [openstack-dev] [Neutron][LBaaS] Thursday meeting follow-up Hi folks, Thanks to everyone who joined the meeting on Thursday. We've discussed desired features and changes in LBaaS service and identified dependencies between them. You can find them on etherpad: https://etherpad.openstack.org/p/neutron-icehouse-lbaas Most of the features are also captured in the document shared by Sam: https://docs.google.com/document/d/1Vjm57lh7PnXDelOy-VxsJkzc8QRiNN368sS11ePs_vA/edit?pli=1#https://docs.google.com/document/d/1Vjm57lh7PnXDelOy-VxsJkzc8QRiNN368sS11ePs_vA/edit?pli=1 IRC meeting logs: http://paste.openstack.org/show/49561/ Thanks, Eugene. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] changing old migrations is verboten
On 10/31/2013 11:23 AM, Jay Pipes wrote: On 10/31/2013 08:01 AM, Sean Dague wrote: So there is a series of patches starting with - https://review.openstack.org/#/c/53417/ that go back and radically change existing migration files. This is really a no-no, unless there is a critical bug fix that absolutely requires it. Changing past migrations should be considered with the same level of weight as an N-2 backport, only done when there is huge upside to the change. I've -2ed the first 2 patches in the series, though that review applies to all of them (I figured a mailing list thread was probably more useful than -2ing everything in the series). There needs to be really solid discussion about the trade offs here before contemplating something as dangerous as this. +1 There is a very real reason why we have a firm stance on this. There are a huge number of OpenStack instances out in the field, at all sorts of different past versions. We really try to promise that you can always forward upgrade your database. If you go back and change an old migration, you have not forked the past. Some people will have already taken that migration, and they have one view of the world, others haven't yet, they hit your updated version, and they now have different database. So 2 people with Havana would no longer be guaranteed to have the same data model set up by us. It's easy to believe that this change is really straight forward, it will be exactly the same model, but if it isn't, in any way, exactly the same (even in a way that we didn't realize yet that it mattered), you've forked the past. And that makes supporting users in these various forked versions of the world impossible. Migrations are basically idempotent. If you want to clean things up, do them in a new migration. Don't touch an old one unless it is causing corruption to someone's data so that fixing it with a future migration is not an option. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] changing old migrations is verboten
On 10/31/2013 11:56 AM, Sean Dague wrote: On 10/31/2013 11:23 AM, Jay Pipes wrote: On 10/31/2013 08:01 AM, Sean Dague wrote: So there is a series of patches starting with - https://review.openstack.org/#/c/53417/ that go back and radically change existing migration files. This is really a no-no, unless there is a critical bug fix that absolutely requires it. Changing past migrations should be considered with the same level of weight as an N-2 backport, only done when there is huge upside to the change. I've -2ed the first 2 patches in the series, though that review applies to all of them (I figured a mailing list thread was probably more useful than -2ing everything in the series). There needs to be really solid discussion about the trade offs here before contemplating something as dangerous as this. +1 There is a very real reason why we have a firm stance on this. There are a huge number of OpenStack instances out in the field, at all sorts of different past versions. We really try to promise that you can always forward upgrade your database. If you go back and change an old migration, you have not forked the past. Some people will have already taken that migration, and they have one view of the world, others haven't yet, they hit your updated version, and they now have different database. So 2 people with Havana would no longer be guaranteed to have the same data model set up by us. It's easy to believe that this change is really straight forward, it will be exactly the same model, but if it isn't, in any way, exactly the same (even in a way that we didn't realize yet that it mattered), you've forked the past. And that makes supporting users in these various forked versions of the world impossible. Migrations are basically idempotent. If you want to clean things up, do them in a new migration. Don't touch an old one unless it is causing corruption to someone's data so that fixing it with a future migration is not an option. LOL, I was +1'ing your thoughts, not +1'ing the proposal to have a solid discussion about the trade-offs :) Sorry for the confusion! -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [horizon / keystone] Marker could not be found?
On Thu, Oct 31, 2013 at 8:38 AM, Sebastian Porombka
porom...@uni-paderborn.de wrote:
Hello Folks.
I have a problem after grizzly-havana migration where i’m unable to
rescue myself.
When I open the Admin - Resource-Usage View i get no results – only a
red error box with the message *Error: *Unable to retrieve tenant list.“.
Horizon log:
[Thu Oct 31 11:39:44 2013] [error] Creating a new keystoneclient
connection to http://$controller:35357/v2.0.
[Thu Oct 31 11:39:44 2013] [error] REQ: curl -i -X GET
http://$controller:35357/v2.0/tenants?marker=tenant_markerlimit=21
-H User-Agent: python-keystoneclient -H Forwarded:
for=131.234.5.178;by=python-keystoneclient -H X-Auth-Token: 82[…]f46
[Thu Oct 31 11:39:44 2013] [error] REQ: curl -i -X GET
http://$controller:35357/v2.0/tenants?marker=tenant_markerlimit=21
-H User-Agent: python-keystoneclient -H Forwarded:
for=131.234.5.178;by=python-keystoneclient -H X-Auth-Token: 82[…]46
[Thu Oct 31 11:39:44 2013] [error] INFO:urllib3.connectionpool:Starting
new HTTP connection (1): $controller
[Thu Oct 31 11:39:44 2013] [error] DEBUG:urllib3.connectionpool:GET
/v2.0/tenants?marker=tenant_markerlimit=21 HTTP/1.1 400 88
[Thu Oct 31 11:39:44 2013] [error] RESP: [400]
CaseInsensitiveDict({'date': 'Thu, 31 Oct 2013 11:39:47 GMT', 'vary':
'X-Auth-Token', 'content-length': '88', 'content-type': 'application/json'})
[Thu Oct 31 11:39:44 2013] [error] RESP BODY: {error: {message:
Marker could not be found, code: 400, title: Bad Request}}
[Thu Oct 31 11:39:44 2013] [error]
[Thu Oct 31 11:39:44 2013] [error] RESP: [400]
CaseInsensitiveDict({'date': 'Thu, 31 Oct 2013 11:39:47 GMT', 'vary':
'X-Auth-Token', 'content-length': '88', 'content-type': 'application/json'})
[Thu Oct 31 11:39:44 2013] [error] RESP BODY: {error: {message:
Marker could not be found, code: 400, title: Bad Request}}
[Thu Oct 31 11:39:44 2013] [error]
[Thu Oct 31 11:39:44 2013] [error] Request returned failure status: 400
[Thu Oct 31 11:39:44 2013] [error] Request returned failure status: 400
[Thu Oct 31 11:39:44 2013] [error] Recoverable error: Marker could not be
found (HTTP 400)
Keystone Log:
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Matched GET
/tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.contrib.ec2.routers.Ec2Extension object at 0x4156f10} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.contrib.ec2.routers.Ec2Extension object at
0x4156f10, 'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Matched GET
/tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.contrib.s3.core.S3Extension object at 0x4156cd0} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.contrib.s3.core.S3Extension object at 0x4156cd0,
'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.354 17187 DEBUG routes.middleware [-] Matched GET
/tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.354 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.contrib.admin_crud.core.CrudExtension object at 0x41517d0}
__call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.contrib.admin_crud.core.CrudExtension object at
0x41517d0, 'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Matched GET
/tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Route path:
'{path_info:.*}', defaults: {'controller':
keystone.common.wsgi.ComposingRouter object at 0x4151e50} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.356 17187 DEBUG routes.middleware [-] Match dict:
{'controller': keystone.common.wsgi.ComposingRouter object at 0x4151e50,
'path_info': '/tenants'} __call__
/usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.356 17187 DEBUG routes.middleware [-] Matched GET
/tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.357 17187 DEBUG routes.middleware [-] Route path:
'/tenants', defaults: {'action':
Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions
On Mon, Oct 28, 2013 at 5:46 PM, Qing He qing...@radisys.com wrote: In my hard drive-less use case, I need an in-core-db/cache that can be in the same db cluster with real db (with hard drive) with the same sql api so that the current openstack code do not need to be changed, instead, just a pluggin with some configurations. This is pretty much the original use case that dogpile.cache grew out of, see: http://docs.sqlalchemy.org/en/rel_0_9/orm/examples.html#dogpile-caching -Original Message- From: Morgan Fainberg [mailto:m...@metacloud.com] Sent: Monday, October 28, 2013 10:22 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions In light of what Dolph said with regards to Keystone, we are using dogpile.cache to implement memoization in front of our driver calls. It it has the ability to cache directly as well, but it has been effective (so far) for our use-case. That being said, I am unsure if caching in front of MySQL is really what we want. I believe that we should be caching after processing work (hence memoization mechanism) instead of at the SQL layer. This also means we can be measured in what we cache (oh hey, it makes no sense to cache X because it needs to be real time or there isn't a performance issue with that query / call, but Y does a ton of processing and is an expensive join/temptable query). In my experience, unless the whole application is designed with caching in mind, caching something as broad as MySQL calls (or any SQL store) is likely going to net exactly what Shawn Hartsock stated, adding a second performance issue. --Morgan On Mon, Oct 28, 2013 at 10:05 AM, Shawn Hartsock hartso...@vmware.com wrote: I once heard a quote.. I had a performance problem, so I added caching. now I have two performance problems. this. 1,000 times this. Just to float this thought ... make sure it's considered... I've seen a *lot* of people misuse caching when what the really want is memoization. * http://stackoverflow.com/questions/1988804/what-is-memoization-and-how -can-i-use-it-in-python * http://stackoverflow.com/questions/10879137/how-can-i-memoize-a-class- instantiation-in-python ... I'm not sure what you're trying to do. So YMMV, TTFN, BBQ. # Shawn Hartsock - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Monday, October 28, 2013 12:12:49 PM Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions Excerpts from Dolph Mathews's message of 2013-10-28 08:40:19 -0700: It's not specific to mysql (or sql at all), but keystone is using dogpile.cache around driver calls to a similar effect. http://dogpilecache.readthedocs.org/en/latest/ It can persist to memcache, redis, etc. I once heard a quote.. I had a performance problem, so I added caching. now I have two performance problems. Caching is unbelievably awesome in the jobs it can do well. When the problem is straight forward and the requirements are few, it is just the right thing to relieve engineering pressure to make an application more scalable. However, IMO, more than narrow, well defined cache usage is a sign that the application needs some reworking to scale. I like the principle of let's use dogpile so we don't have to reinvent multi-level caching. However, let's make sure we look at each performance issue individually, rather than just throwing them all in a cache box and waving the memcache wand. https://github.com/openstack/keystone/blob/master/keystone/common/c ache/core.py On Fri, Oct 25, 2013 at 6:53 PM, Qing He qing...@radisys.com wrote: All, Has anyone looked at the options of putting a distributed caching system in front of mysql server to improve performance? This should be similar to Oracle Coherence, or VMware VFabric SQLFire. ** ** Thanks, ** ** Qing ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Migrating to newer full projects from what used to be part of nova
Hi everyone, Migrations from Essex to Grizzly/Havana are starting to hit my radar of responsible tasks and I'm disappointed that beyond this old wiki note [1] and a wealth of questions with very few answers [2], there is very little available to support the migrations from what used to be part of nova to the newer full projects. I really think that as Openstack grows and the projects split out, one of the focal areas really needs to be on ensuring that people using the older versions can migrate to the newer versions without needing to do all sorts of terrible hacks. Issues at hand, for now, are: 1) Migrating from nova-volume to cinder 2) Migrating from nova-network to neutron It'd be great if we could pool efforts to figure out an effective way of handling these migrations. Whether they're handled in the 'db sync' process, or by a set of companion utilities instead is immaterial to a deployer... as long as something suitable and effective is available to cater for the need. [1] https://wiki.openstack.org/wiki/MigrateToCinder [2] https://www.google.co.za/search?q=migrate+'nova-network'+quantumhttps://www.google.co.za/search?q=migrate+nova-network+quantumoq=migrate+nova-network+quantumaqs=chrome..69i57.12524j0j7sourceid=chromeespv=210es_sm=93ie=UTF-8 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] changing old migrations is verboten
Actually no confusion. :-) Joe Gordon just made me realize that I didn't really explain why we had that policy. That really should have been a follow up to my own post, not yours. Sorry if I made it look like I was arguing with you, which I wasn't.. :-) We're all good. Sean Dague http://dague.net On Oct 31, 2013 12:12 PM, Jay Pipes jaypi...@gmail.com wrote: On 10/31/2013 11:56 AM, Sean Dague wrote: On 10/31/2013 11:23 AM, Jay Pipes wrote: On 10/31/2013 08:01 AM, Sean Dague wrote: So there is a series of patches starting with - https://review.openstack.org/#**/c/53417/https://review.openstack.org/#/c/53417/that go back and radically change existing migration files. This is really a no-no, unless there is a critical bug fix that absolutely requires it. Changing past migrations should be considered with the same level of weight as an N-2 backport, only done when there is huge upside to the change. I've -2ed the first 2 patches in the series, though that review applies to all of them (I figured a mailing list thread was probably more useful than -2ing everything in the series). There needs to be really solid discussion about the trade offs here before contemplating something as dangerous as this. +1 There is a very real reason why we have a firm stance on this. There are a huge number of OpenStack instances out in the field, at all sorts of different past versions. We really try to promise that you can always forward upgrade your database. If you go back and change an old migration, you have not forked the past. Some people will have already taken that migration, and they have one view of the world, others haven't yet, they hit your updated version, and they now have different database. So 2 people with Havana would no longer be guaranteed to have the same data model set up by us. It's easy to believe that this change is really straight forward, it will be exactly the same model, but if it isn't, in any way, exactly the same (even in a way that we didn't realize yet that it mattered), you've forked the past. And that makes supporting users in these various forked versions of the world impossible. Migrations are basically idempotent. If you want to clean things up, do them in a new migration. Don't touch an old one unless it is causing corruption to someone's data so that fixing it with a future migration is not an option. LOL, I was +1'ing your thoughts, not +1'ing the proposal to have a solid discussion about the trade-offs :) Sorry for the confusion! -jay __**_ OpenStack-dev mailing list OpenStack-dev@lists.openstack.**org OpenStack-dev@lists.openstack.org http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-devhttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Migrating to newer full projects from what used to be part of nova
Yes, OS as big product tries to split hierarchy of responsibilities. Currently we have serveral tasks, i suppose. First one is upgrading from one stable release to another. Second one is migrating from one release to another. Currently OpenStack has several deployment projects: TripleO, Fuel (--- is a bit different from TripleO, it performs bare-metal deployment of whole OS). As way of covering issue of upgrading/migrating, personaly i see, we could suggest those projects to propose a way of solving it. 2013/10/31 Jesse Pretorius jesse.pretor...@gmail.com Hi everyone, Migrations from Essex to Grizzly/Havana are starting to hit my radar of responsible tasks and I'm disappointed that beyond this old wiki note [1] and a wealth of questions with very few answers [2], there is very little available to support the migrations from what used to be part of nova to the newer full projects. I really think that as Openstack grows and the projects split out, one of the focal areas really needs to be on ensuring that people using the older versions can migrate to the newer versions without needing to do all sorts of terrible hacks. Issues at hand, for now, are: 1) Migrating from nova-volume to cinder 2) Migrating from nova-network to neutron It'd be great if we could pool efforts to figure out an effective way of handling these migrations. Whether they're handled in the 'db sync' process, or by a set of companion utilities instead is immaterial to a deployer... as long as something suitable and effective is available to cater for the need. [1] https://wiki.openstack.org/wiki/MigrateToCinder [2] https://www.google.co.za/search?q=migrate+'nova-network'+quantumhttps://www.google.co.za/search?q=migrate+nova-network+quantumoq=migrate+nova-network+quantumaqs=chrome..69i57.12524j0j7sourceid=chromeespv=210es_sm=93ie=UTF-8 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack's OpenPGP Web of Trust
On 2013-10-31 12:24:49 +0100 (+0100), Thierry Carrez wrote: OK, made it official [...] And: https://wiki.openstack.org/wiki/OpenPGP_Web_of_Trust/Icehouse_Summit If you add yourself to the table there in the wiki by noon the day before, I'll try to bring a stack of printouts we can use as checklists. With only 30 minutes and places to get from/to between sessions, it's going to be tight timing if many people want to participate. Don't feel like you have to get through everyone on the list... it would be nice, but as long as enough interested attendees participate and we each get a few signatures out of the effort it will still be a great start. Just a reminder of how these things go, in case you haven't participated before--this is really about meeting in person and checking names/photos/key fingerprints. The actual key signing can (and should, in the interest of time) happen later at your convenience... probably after we get home from the summit for many of us because it will be a very busy week already. Also, be prepared that some participants will decide after reviewing your identification no to sign your key. This is perfectly natural, as key signing choices are a matter of personal preference (some people might care about strict name matching, may be uncertain of the validity of your ID, or could have a hard time recognizing your old photo from when you still had long hair and didn't need glasses yet). In my opinion that additional individuality only leads to a healthier web of trust and is not something you should ever consider an insult. Also, I'll be around at the conference every day with my ID and key fingerprint cards too (as I expect will some others). So if you can't get to the Thursday slot for other reasons and want to exchange ID with someone, just let them know and see if they might be able to work out a quick face-to-face in the hallway, at lunch or maybe during one of the evening parties. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Migrating to newer full projects from what used to be part of nova
On Thu, Oct 31, 2013 at 10:26 AM, Jesse Pretorius jesse.pretor...@gmail.com wrote: Hi everyone, Migrations from Essex to Grizzly/Havana are starting to hit my radar of responsible tasks and I'm disappointed that beyond this old wiki note [1] Is your disappointment that there isn't a path from Essex -- Grizzly/Havana, or are you unhappy with the content? and a wealth of questions with very few answers [2], there is very little available to support the migrations from what used to be part of nova to the newer full projects. I really think that as Openstack grows and the projects split out, one of the focal areas really needs to be on ensuring that people using the older versions can migrate to the newer versions without needing to do all sorts of terrible hacks. Issues at hand, for now, are: 1) Migrating from nova-volume to cinder So to be quite honest we never intended to make skips like you describe. Perhaps that wasn't such a good choice in retrospect. I'm willing to take a look at putting some special migrations in, but honestly given the changes in Nova alone between Essex and Folsom making a full jump from something like Essex to Havana is going to be challenging not just for adding Cinder, but overall. I know it's not ideal but right now the best/preferred solution may in fact be to do incremental updates as intended. Going forward maybe we can come up with something better. 2) Migrating from nova-network to neutron It'd be great if we could pool efforts to figure out an effective way of handling these migrations. Whether they're handled in the 'db sync' process, or by a set of companion utilities instead is immaterial to a deployer... as long as something suitable and effective is available to cater for the need. [1] https://wiki.openstack.org/wiki/MigrateToCinder [2] https://www.google.co.za/search?q=migrate+'nova-network'+quantum ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] changing old migrations is verboten
On Thu, Oct 31, 2013, Sean Dague s...@dague.net wrote: So there is a series of patches starting with - https://review.openstack.org/#/c/53417/ that go back and radically change existing migration files. This is really a no-no, unless there is a critical bug fix that absolutely requires it. Changing past migrations should be considered with the same level of weight as an N-2 backport, only done when there is huge upside to the change. I've -2ed the first 2 patches in the series, though that review applies to all of them (I figured a mailing list thread was probably more useful than -2ing everything in the series). There needs to be really solid discussion about the trade offs here before contemplating something as dangerous as this. The most important thing for DB migrations is that they remain functionality identical. Historically we have allowed many changes to DB migrations that kept them functionally identical to how they were before. Looking through the commit history, here's a sampling of changes: - _ was no longer monkey patched, necessitating a new import added - fix bugs causing testing problems - change copyright headers - remove unused code (creating logger, imports, etc) - fix bugs causing the migrations to fail to function (on PostgreSQL, downgrade bugs, etc) - style changes (removing use of locals(), whitespace, etc) - make migrations faster - add comments to clarify code - improve compatibility with newer versions of SQLAlchemy The reviews you're referencing seem to fall into what we have historically allowed. That said, I do agree there needs to be a higher burden of proof that the change being made is functionally identical to before. JE ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions
Dolph, Thanks! It is good that python example is provided! Qing From: Dolph Mathews [mailto:dolph.math...@gmail.com] Sent: Thursday, October 31, 2013 9:25 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions On Mon, Oct 28, 2013 at 5:46 PM, Qing He qing...@radisys.commailto:qing...@radisys.com wrote: In my hard drive-less use case, I need an in-core-db/cache that can be in the same db cluster with real db (with hard drive) with the same sql api so that the current openstack code do not need to be changed, instead, just a pluggin with some configurations. This is pretty much the original use case that dogpile.cache grew out of, see: http://docs.sqlalchemy.org/en/rel_0_9/orm/examples.html#dogpile-caching -Original Message- From: Morgan Fainberg [mailto:m...@metacloud.commailto:m...@metacloud.com] Sent: Monday, October 28, 2013 10:22 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions In light of what Dolph said with regards to Keystone, we are using dogpile.cache to implement memoization in front of our driver calls. It it has the ability to cache directly as well, but it has been effective (so far) for our use-case. That being said, I am unsure if caching in front of MySQL is really what we want. I believe that we should be caching after processing work (hence memoization mechanism) instead of at the SQL layer. This also means we can be measured in what we cache (oh hey, it makes no sense to cache X because it needs to be real time or there isn't a performance issue with that query / call, but Y does a ton of processing and is an expensive join/temptable query). In my experience, unless the whole application is designed with caching in mind, caching something as broad as MySQL calls (or any SQL store) is likely going to net exactly what Shawn Hartsock stated, adding a second performance issue. --Morgan On Mon, Oct 28, 2013 at 10:05 AM, Shawn Hartsock hartso...@vmware.commailto:hartso...@vmware.com wrote: I once heard a quote.. I had a performance problem, so I added caching. now I have two performance problems. this. 1,000 times this. Just to float this thought ... make sure it's considered... I've seen a *lot* of people misuse caching when what the really want is memoization. * http://stackoverflow.com/questions/1988804/what-is-memoization-and-how -can-i-use-it-in-python * http://stackoverflow.com/questions/10879137/how-can-i-memoize-a-class- instantiation-in-python ... I'm not sure what you're trying to do. So YMMV, TTFN, BBQ. # Shawn Hartsock - Original Message - From: Clint Byrum cl...@fewbar.commailto:cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Sent: Monday, October 28, 2013 12:12:49 PM Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions Excerpts from Dolph Mathews's message of 2013-10-28 08:40:19 -0700: It's not specific to mysql (or sql at all), but keystone is using dogpile.cache around driver calls to a similar effect. http://dogpilecache.readthedocs.org/en/latest/ It can persist to memcache, redis, etc. I once heard a quote.. I had a performance problem, so I added caching. now I have two performance problems. Caching is unbelievably awesome in the jobs it can do well. When the problem is straight forward and the requirements are few, it is just the right thing to relieve engineering pressure to make an application more scalable. However, IMO, more than narrow, well defined cache usage is a sign that the application needs some reworking to scale. I like the principle of let's use dogpile so we don't have to reinvent multi-level caching. However, let's make sure we look at each performance issue individually, rather than just throwing them all in a cache box and waving the memcache wand. https://github.com/openstack/keystone/blob/master/keystone/common/c ache/core.py On Fri, Oct 25, 2013 at 6:53 PM, Qing He qing...@radisys.commailto:qing...@radisys.com wrote: All, Has anyone looked at the options of putting a distributed caching system in front of mysql server to improve performance? This should be similar to Oracle Coherence, or VMware VFabric SQLFire. ** ** Thanks, ** ** Qing ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] Migrating to newer full projects from what used to be part of nova
On 10/31/2013 12:46 PM, John Griffith wrote: Issues at hand, for now, are: 1) Migrating from nova-volume to cinder So to be quite honest we never intended to make skips like you describe. Perhaps that wasn't such a good choice in retrospect. I'm willing to take a look at putting some special migrations in, but honestly given the changes in Nova alone between Essex and Folsom making a full jump from something like Essex to Havana is going to be challenging not just for adding Cinder, but overall. I know it's not ideal but right now the best/preferred solution may in fact be to do incremental updates as intended. Right, Nova (and probably other projects, too) only supports N to N+1 upgrades. If you need to go to N+2, you'll need to go through N+1 to get there. FWIW, improving upgrades is very high on the priority list across all of OpenStack. We have sessions all over the place to talk about it. It's getting a lot better. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On 10/31/2013 06:04 AM, Rosa, Andrea (HP Cloud Services) wrote: A - there is no test suite at all, adding one in unreasonable B - this thing cannot be tested in this context (e.g. functional tests are defined in a different tree) C - this particular thing is very hard to test D - testing this won't offer benefit In my opinion C instead of being an acceptable reason for not having tests is a symptom of one of the two things: 1) F = Submitter doesn't know how to write tests, in this case someone else can help with suggestions 2) The code we are trying to test is too complicated so it's time to refactor it And about D, In my opinion tests always offer benefits, like code coverage or helping in understanding the code. I think there are actually cases where C is valid. It's difficult to test certain kinds of race conditions, for example, unless you have very low-level hooks into the guts of the system in order to force the desired conditions to reliably occur at exactly the right time. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
I'm pretty sure the cats out of the bag. https://github.com/openstack/requirements/blob/master/global-requirements.t xt#L29 https://kazoo.readthedocs.org/en/latest/ -Josh On 10/31/13 7:43 AM, Monty Taylor mord...@inaugust.com wrote: On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [State-Management] Cancel IRC meeting for next 2 weeks
Hi all, I just wanted to cancel the state-management IRC meeting for the next 2 weeks due to people traveling and the HK summit. Since it won't be as productive as usual I think we can just skip it until after the summit :-) Likely most people will be in #openstack-state-management anyway, feel free to join! Much appreciated! As always: https://wiki.openstack.org/wiki/TaskFlow -Josh ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
Sigh. Yay We've added more competing methods of complexity!!! Seriously. We now think that rabbit and zookeeper and mysql are ALL needed? Joshua Harlow harlo...@yahoo-inc.com wrote: I'm pretty sure the cats out of the bag. https://github.com/openstack/requirements/blob/master/global-requirements.t xt#L29 https://kazoo.readthedocs.org/en/latest/ -Josh On 10/31/13 7:43 AM, Monty Taylor mord...@inaugust.com wrote: On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On 2013-10-31 09:05, Kyle Mestery (kmestery) wrote:
On Oct 31, 2013, at 8:56 AM, Clint Byrum cl...@fewbar.com wrote:
Excerpts from Mark McLoughlin's message of 2013-10-31 06:30:32 -0700:
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional
tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they
might
be optional.
I totally agree with the sentiment but, especially when it's a
newcomer
to the project, I try to put myself in the shoes of the patch
submitter
and double-check whether what we're asking is reasonable.
Even with a long time contributor, empathy is an important part of
constructing reviews. We could make more robotic things that review
for
test coverage, but we haven't, because this is a gray area. The role
of
a reviewer isn't just get patches merged and stop defects. It is also
to
grow the other developers.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug -
think
typo like raise NotFund('foo') - and testing this code patch
requires
more than adding a simple new scenario to existing tests ...
This goes back to my recent suggestion to help the person not with a
-1
or a +2, but with an additional patch that fixes it.
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project
doesn't
have full test coverage to begin with really makes us seem
unwelcoming.
In cases like that, I'd be of a mind to go +2 Awesome! Thanks for
catching this! It would be great to have a unit test for this, but
it's
clear the current code is broken so I'm fine with merging the fix
without a test. You could say it's now the reviewers responsibility
to
merge a test, but if that requirement then turns off reviewers even
reviewing such a patch, then that doesn't help either.
I understand entirely why you choose this, and I think that is fine.
I,
however, see this as a massive opportunity to teach. That code was
only
broken because it was allowed it to be merged without tests. By
letting
that situation continue, we only fix it today. The next major
refactoring
has a high chance now of breaking that part of the code again.
So, rather than +2, I suggest -1 with compassion. Engage with the
submitter. If you don't know them, take a look at how hard it would be
to write a test for the behavior and give pointers to the exact test
suite that would need to be changed, or suggest a new test suite and
point at a good example to copy.
So, with all of this, let's make sure we don't forget to first
appreciate the effort that went into submitting the patch that lacks
tests.
I'm not going to claim that I've always practiced -1 with
compassion,
so thanks for reminding us all that we're not just reviewing code, we
are having a dialog with real live people.
I think this is the key thing here, thanks for bringing this up Clint.
At the
end of the day, patches are submitted by real people. If we want to
grow
the committer base and help people to become better reviewers, taking
the time to show them the ropes is part of the game. I'd argue that is
in
fact part of what being a core is about in fact.
When in doubt, I tend to err on the side of no score with comments.
It's not ideal for every situation, but I like to think it gets my point
across without making it sound like I disapprove of the change itself.
If my suggestions are ignored completely, I've always got the -1 in my
back pocket to press the issue. :-)
-Ben
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
In the spirt of openness, yes I do think they are all needed. If they are not supported, then openstack is not open, it is a closed system. We should strive to innovate, not strive to be stuck with the status quo. To me it is a developers decision to pick the right solution, if that solution involves some complexity then u make it a pluggable solution. Your view of the right solution will likely not be the view of someone elses right solution in the end anyway (and u likely can't predict future solutions that will be applicable anyway). If u just say no to that plugin then u are just excluding people from participating in your project and this is imho against the spirt of openness in general. And those people who would have contributed will just start looking elsewhere for a solution which does work. This kills the openstack... On 10/31/13 10:17 AM, Monty Taylor mord...@inaugust.com wrote: Sigh. Yay We've added more competing methods of complexity!!! Seriously. We now think that rabbit and zookeeper and mysql are ALL needed? Joshua Harlow harlo...@yahoo-inc.com wrote: I'm pretty sure the cats out of the bag. https://github.com/openstack/requirements/blob/master/global-requirements .t xt#L29 https://kazoo.readthedocs.org/en/latest/ -Josh On 10/31/13 7:43 AM, Monty Taylor mord...@inaugust.com wrote: On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] [savanna] [trove] Place for software configuration
Hi Heat, Savanna and Trove teams, All this projects have common part related to software configuration management. For creation, an environment user should specify a hardware parameter for vms: choose flavor, decide use cinder or not, configure networks for virtual machines, choose topology for hole deployment. Next step is linking of software parameters with hardware specification. From the end user point of view, existence of three different places and three different ways (HEAT Hot DSL, Trove clustering API and Savanna Hadoop templates) for software configuration is not convenient, especially if user want to create an environment simultaneously involving components from Savanna, Heat and Trove. I can suggest two approaches to overcome this situations: Common library in oslo. This approach allows a deep domain specific customization. The user will still have 3 places with same UI where user should perform configuration actions. Heat or some other component for software configuration management. This approach is the best for end users. In feature possible will be some limitation on deep domain specific customization for configuration management. Heat, Savanna and Trove teams can you comment these ideas, what approach are the best? Alexander Kuznetsov. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Savanna] Release 0.3 retrospective
Hi, Here is the wiki page with Savanna release 0.3 retrospective: https://wiki.openstack.org/wiki/Savanna/Release_0.3_Retrospective Thanks everyone who sent your opinions. If someone wants to add more thoughts you are welcome to edit above page! -- Regards, Alexander Ignatov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
On 10/31/2013 01:32 PM, Joshua Harlow wrote: In the spirt of openness, yes I do think they are all needed. If they are not supported, then openstack is not open, it is a closed system. We should strive to innovate, not strive to be stuck with the status quo. To me it is a developers decision to pick the right solution, if that solution involves some complexity then u make it a pluggable solution. Your view of the right solution will likely not be the view of someone elses right solution in the end anyway (and u likely can't predict future solutions that will be applicable anyway). If u just say no to that plugin then u are just excluding people from participating in your project and this is imho against the spirt of openness in general. And those people who would have contributed will just start looking elsewhere for a solution which does work. This kills the openstack... Hrm. I certainly don't want to kill openstack, and I _certainly_ don't want to disallow a plugin. I apologize if it came across that way. What I was questioning is whether or not we wanted to add a hard requirement on zookeeper somewhere. Even Rabbit and MySQL are decisions with other options. It's entirely possible I misread the conversation of course... On 10/31/13 10:17 AM, Monty Taylor mord...@inaugust.com wrote: Sigh. Yay We've added more competing methods of complexity!!! Seriously. We now think that rabbit and zookeeper and mysql are ALL needed? Joshua Harlow harlo...@yahoo-inc.com wrote: I'm pretty sure the cats out of the bag. https://github.com/openstack/requirements/blob/master/global-requirements .t xt#L29 https://kazoo.readthedocs.org/en/latest/ -Josh On 10/31/13 7:43 AM, Monty Taylor mord...@inaugust.com wrote: On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] HK Summit - Image Creation and Customization unconference?
Hello, Would any of you attending the summit be interested in snagging an unconference session to discuss the state of play with image creation, customization and import? I can contribute an overview and demonstration of our Nova-native image building tool. I'd be interested in exploring integration points with Disk Image Builder and Glance. Any takers? -Ian ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] [savanna] [trove] Place for software configuration
On 10/31/2013 01:51 PM, Alexander Kuznetsov wrote: Hi Heat, Savanna and Trove teams, All this projects have common part related to software configuration management. For creation, an environment user should specify a hardware parameter for vms: choose flavor, decide use cinder or not, configure networks for virtual machines, choose topology for hole deployment. Next step is linking of software parameters with hardware specification. From the end user point of view, existence of three different places and three different ways (HEAT Hot DSL, Trove clustering API and Savanna Hadoop templates) for software configuration is not convenient, especially if user want to create an environment simultaneously involving components from Savanna, Heat and Trove. I can suggest two approaches to overcome this situations: Common library in oslo. This approach allows a deep domain specific customization. The user will still have 3 places with same UI where user should perform configuration actions. Heat or some other component for software configuration management. This approach is the best for end users. In feature possible will be some limitation on deep domain specific customization for configuration management. I think this would be my preference. In other words, describe and orchestrate a Hadoop or Database setup using HOT templates and using Heat as the orchestration engine. Best, -jay Heat, Savanna and Trove teams can you comment these ideas, what approach are the best? Alexander Kuznetsov. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Call for participation: Breakout meeting for API deep-dive
Team, In the interest of providing adequate advance notice, we will not consider any of the timeslots for today. I have edited the poll accordingly. It also looks like a number of us did not notice that there are a lot of proposed times hiding under the accordion view of the poll. So, please take a moment to re-visit the poll, and click the Show all 18 options button, and please mark the (If-Needbe) options if you have conflicting commitments that you would be willing to reschedule to attend the breakout meeting(s). Note: No date/time options were proposed for the week of the Icehouse OpenStack Design Summit next week in Hong Kong. Most of the proposed times are the week following next. Note: I plan to close the poll at 5:00 PM US/Pacific today and select the date/time for these based on the feedback you provide. Please chime in now so we can do our best to include you. Thanks, Adrian On Oct 30, 2013, at 6:03 PM, Adrian Otto adrian.o...@rackspace.com wrote: Team, Those of us interested in participating in a series of API deep-dive design discussions, are encouraged to provide date/time scheduling feedback at the link shown on our Breakout Meetings wiki page: https://wiki.openstack.org/wiki/Solum/BreakoutMeetings We look forward to your input to select the date/time of this series. Thanks, Adrian ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
Excerpts from Khanh-Toan Tran's message of 2013-10-31 07:22:06 -0700:
Hi all,
As a newbie of the community, I'm not familiar with unittest and how to use
it here. I've learned that Jenkins runs tests
everytime we submit some code. But how to write the test and what is a 'good
test' and a 'bad test'? I saw some commits
in gerrit but am unable to say if the written test is enough to judge the
code, since it is the author of the code who writes
the test. Is there a framework to follow or some rules/pratices to respect?
Do you have some links to help me out?
This is a nice synopsis of the concept of test driven development:
http://net.tutsplus.com/tutorials/python-tutorials/test-driven-development-in-python/
In OpenStack we always put tests in _base_module_name_/tests, So if you
are working on nova, you can see the unit tests in:
nova/tests
You can generally always run the tests by installing the 'tox' python
module/command on your system and running 'tox' in the root of the git
repository.
Projects use various testing helpers to make tests easier to read and
write. The most common one is testtools. A typical test will look like
this:
import testtools
from basemodule import submodule
class TestSubmoduleFoo(testtools.TestCase):
def test_foo_apple(self):
self.assertEquals(1, submodule.foo('apple'))
def test_foo_banana(self):
self.assertEquals(0, submodule.foo('banana'))
Often unit tests will include mocks and fakes to hide real world
interfacing code from the unit tests. You would do well to read up on
how those concepts work as well, google for 'python test mocking' and
'python test fakes'.
Good luck, and #openstack-dev is always there to try and help. :)
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On Thu, Oct 31, 2013 at 01:30:32PM +, Mark McLoughlin wrote:
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they might
be optional.
I totally agree with the sentiment but, especially when it's a newcomer
to the project, I try to put myself in the shoes of the patch submitter
and double-check whether what we're asking is reasonable.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug - think
typo like raise NotFund('foo') - and testing this code patch requires
more than adding a simple new scenario to existing tests ...
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project doesn't
have full test coverage to begin with really makes us seem unwelcoming.
The approach the Heat team have sometimes taken in this situation is to
merge the patch, but raise a bug (targetted at the next milestone)
identifying the missing coverage.
This has a few benefits (provided the patch in question is sufficiently
simple that patches aren't deemed essential)
- The bug gets fixed quickly
- The coverage still gets added, and we keep track of the gaps identified
- Less chance of discouraging new submitters
- Provides some low hanging fruit bugs, which new contributors can pick
up
I'm not saying we do this routinely, but it's a possible middle ground to
consider between -1 fix all our tests and +2 shipit!
I think something that's important to recognise is that sometimes (often?)
writing tests is *hard*. I mean, look at the barriers to entry, you need
to have some idea about tox, nose, testr, mox, mock, testscenarios, etc etc
The learning curve is really steep, and thats before you start considering
project specific test abstractions/fixtures/mocking-patterns which can all
be complex and non-obvious.
So +1 on a bit of compassion when reviewing, particularly if the
contributor is a new or casual contributor to the project.
Steve
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] welcoming new committers (was Re: When is it okay for submitters to say 'I don't want to add tests' ?)
On 10/31/2013 07:05 AM, Kyle Mestery (kmestery) wrote: [...] If we want to grow the committer base and help people to become better reviewers, taking the time to show them the ropes is part of the game. hijacking the thread using Kyle's comment as an excuse. It's not an 'if' but a 'since': since we are growing the committer base at an incredible pace we should help them become also good reviewers as rapidly possible. One thing I already mentioned and I'll start doing this week in the weekly Newsletter is to give a shoutout to those that do their first review this week. Another idea that Tom suggested is to use gerrit automation to send back to first time committers something in addition to the normal 'your patch is waiting for review' message. The message could be something like: thank you for your first contribution to OpenStack. Your patch will now be tested automatically by OpenStack testing frameworks and once the automatic tests pass, it will be reviewed by other friendly developers. They will give you comments and may require you to refine it. Nobody gets his patch approved at first try so don't be concerned when someone will require you to do more iterations. Patches usually take 3 to 7 days to be approved so be patient and be available on IRC to ask and answer questions about your work. The more you participate in the community the more rewarding it is for you. You may also notice that the more you get to know people and get to be known, the faster your patches will be reviewed and eventually approved. Get to know others and be known by doing code reviews: anybody can and should do it. With links to the wiki for more details, of course. This sort of messaging may help all the people that contribute tactically, those that are asked by their manager to land a patch in here and are simply lightly involved (not committed) in OpenStack. These are the ones that may have an incorrect perception of how easy it is to have patches landed in OpenStack as opposed to other large projects, like the kernel or android and complain about our time to traverse the review system. What do you think? How can we instruct gerrit to do this? /stef PS I put the text on https://etherpad.openstack.org/p/welcome-new-committers for refinements. -- Ask and answer questions on https://ask.openstack.org ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On 13-10-30 10:37 PM, Robert Collins wrote: This is a bit of a social norms thread I've been consistently asking for tests in reviews for a while now, and I get the occasional push-back. I think this falls into a few broad camps: A - there is no test suite at all, adding one in unreasonable B - this thing cannot be tested in this context (e.g. functional tests are defined in a different tree) C - this particular thing is very hard to test D - testing this won't offer benefit E - other things like this in the project don't have tests F - submitter doesn't know how to write tests G - submitter doesn't have time to write tests Now, of these, I think it's fine not add tests in cases A, B, C in combination with D, and D. I don't think E, F or G are sufficient reasons to merge something without tests, when reviewers are asking for them. G in the special case that the project really wants the patch landed - but then I'd expect reviewers to not ask for tests or to volunteer that they might be optional. Now, if I'm wrong, and folk have different norms about when to accept 'reason X not to write tests' as a response from the submitter - please let me know! -Rob I recently hit option A for nodepool. My patch was accepted, but I didn't know where to start for adding the testsuite for the project. So, that said, if option A keeps coming up, then I think the obvious choice is development needs to refocus to take the option of the table. -- Paul Belanger | PolyBeacon, Inc. Jabber: paul.belan...@polybeacon.com | IRC: pabelanger (Freenode) Github: https://github.com/pabelanger | Twitter: https://twitter.com/pabelanger ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] HK Summit - Image Creation and Customization unconference?
Excerpts from Ian McLeod's message of 2013-10-31 11:27:39 -0700: Hello, Would any of you attending the summit be interested in snagging an unconference session to discuss the state of play with image creation, customization and import? I can contribute an overview and demonstration of our Nova-native image building tool. I'd be interested in exploring integration points with Disk Image Builder and Glance. Any takers? I am certain that one of the tripleo-core devs will want to be there, if not many of us. We don't have that many sessions scheduled so I'd suggest just trying to aim at when we're not having Deployment sessions. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
I like Steve's suggestion:
The approach the Heat team have sometimes taken in this situation is to
merge the patch, but raise a bug (targetted at the next milestone)
identifying the missing coverage.
I'm (almost!) a first time contributor and I've put a fix on the backburner
as I find the time to ramp up on the unit tests suite. The fix was
reviewed 2 weeks ago, requesting unit tests - very reasonable but I
needed help to even start (I got no response from requests on IRC, ML, or
e-mail to the bug reporter).
If it wasn't for the good Upstream University people mentoring me - heh,
they deserve a plug! - I'm sure I would have moved on.
Thankfully, a cunning commit message provoked the guidance I needed so I
just need a long weekend to get back to the tests - which I have now.
I'm not sure in which cases you would/wouldn't want to split the bug for
the implementation of tests but I'm sure it would help other first timers
and encourage new contributors.
Regards,
Mike
On 31 October 2013 19:51, Steven Hardy sha...@redhat.com wrote:
On Thu, Oct 31, 2013 at 01:30:32PM +, Mark McLoughlin wrote:
On Thu, 2013-10-31 at 15:37 +1300, Robert Collins wrote:
This is a bit of a social norms thread
I've been consistently asking for tests in reviews for a while now,
and I get the occasional push-back. I think this falls into a few
broad camps:
A - there is no test suite at all, adding one in unreasonable
B - this thing cannot be tested in this context (e.g. functional tests
are defined in a different tree)
C - this particular thing is very hard to test
D - testing this won't offer benefit
E - other things like this in the project don't have tests
F - submitter doesn't know how to write tests
G - submitter doesn't have time to write tests
Nice breakdown.
Now, of these, I think it's fine not add tests in cases A, B, C in
combination with D, and D.
I don't think E, F or G are sufficient reasons to merge something
without tests, when reviewers are asking for them. G in the special
case that the project really wants the patch landed - but then I'd
expect reviewers to not ask for tests or to volunteer that they might
be optional.
I totally agree with the sentiment but, especially when it's a newcomer
to the project, I try to put myself in the shoes of the patch submitter
and double-check whether what we're asking is reasonable.
For example, if someone shows up to Nova with their first OpenStack
contribution, it fixes something which is unquestionably a bug - think
typo like raise NotFund('foo') - and testing this code patch requires
more than adding a simple new scenario to existing tests ...
That, for me, is an example of -1, we need a test! untested code is
broken! is really shooting the messenger, not valuing the newcomers
contribution and risks turning that person off the project forever.
Reviewers being overly aggressive about this where the project doesn't
have full test coverage to begin with really makes us seem unwelcoming.
The approach the Heat team have sometimes taken in this situation is to
merge the patch, but raise a bug (targetted at the next milestone)
identifying the missing coverage.
This has a few benefits (provided the patch in question is sufficiently
simple that patches aren't deemed essential)
- The bug gets fixed quickly
- The coverage still gets added, and we keep track of the gaps identified
- Less chance of discouraging new submitters
- Provides some low hanging fruit bugs, which new contributors can pick
up
I'm not saying we do this routinely, but it's a possible middle ground to
consider between -1 fix all our tests and +2 shipit!
I think something that's important to recognise is that sometimes (often?)
writing tests is *hard*. I mean, look at the barriers to entry, you need
to have some idea about tox, nose, testr, mox, mock, testscenarios, etc etc
The learning curve is really steep, and thats before you start considering
project specific test abstractions/fixtures/mocking-patterns which can all
be complex and non-obvious.
So +1 on a bit of compassion when reviewing, particularly if the
contributor is a new or casual contributor to the project.
Steve
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey
Agreed, I don't think we should enforce a hard requirement for the same reason we should avoid zookeeper. To me they are the same, saying 'no ZK' is the same as saying 'only mysql or rabbitmq'... Both not so open. On 10/31/13 11:04 AM, Monty Taylor mord...@inaugust.com wrote: On 10/31/2013 01:32 PM, Joshua Harlow wrote: In the spirt of openness, yes I do think they are all needed. If they are not supported, then openstack is not open, it is a closed system. We should strive to innovate, not strive to be stuck with the status quo. To me it is a developers decision to pick the right solution, if that solution involves some complexity then u make it a pluggable solution. Your view of the right solution will likely not be the view of someone elses right solution in the end anyway (and u likely can't predict future solutions that will be applicable anyway). If u just say no to that plugin then u are just excluding people from participating in your project and this is imho against the spirt of openness in general. And those people who would have contributed will just start looking elsewhere for a solution which does work. This kills the openstack... Hrm. I certainly don't want to kill openstack, and I _certainly_ don't want to disallow a plugin. I apologize if it came across that way. What I was questioning is whether or not we wanted to add a hard requirement on zookeeper somewhere. Even Rabbit and MySQL are decisions with other options. It's entirely possible I misread the conversation of course... On 10/31/13 10:17 AM, Monty Taylor mord...@inaugust.com wrote: Sigh. Yay We've added more competing methods of complexity!!! Seriously. We now think that rabbit and zookeeper and mysql are ALL needed? Joshua Harlow harlo...@yahoo-inc.com wrote: I'm pretty sure the cats out of the bag. https://github.com/openstack/requirements/blob/master/global-requiremen ts .t xt#L29 https://kazoo.readthedocs.org/en/latest/ -Josh On 10/31/13 7:43 AM, Monty Taylor mord...@inaugust.com wrote: On 10/30/2013 10:42 AM, Clint Byrum wrote: So, recently we've had quite a long thread in gerrit regarding locking in Heat: https://review.openstack.org/#/c/49440/ In the patch, there are two distributed lock drivers. One uses SQL, and suffers from all the problems you might imagine a SQL based locking system would. It is extremely hard to detect dead lock holders, so we end up with really long timeouts. The other is ZooKeeper. I'm on record as saying we're not using ZooKeeper. It is a little embarrassing to have taken such a position without really thinking things through. The main reason I feel this way though, is not because ZooKeeper wouldn't work for locking, but because I think locking is a mistake. The current multi-engine paradigm has a race condition. If you have a stack action going on, the state is held in the engine itself, and not in the database, so if another engine starts working on another action, they will conflict. The locking paradigm is meant to prevent this. But I think this is a huge mistake. The engine should store _all_ of its state in a distributed data store of some kind. Any engine should be aware of what is already happening with the stack from this state and act accordingly. That includes the engine currently working on actions. When viewed through this lense, to me, locking is a poor excuse for serializing the state of the engine scheduler. It feels like TaskFlow is the answer, with an eye for making sure TaskFlow can be made to work with distributed state. I am not well versed on TaskFlow's details though, so I may be wrong. It worries me that TaskFlow has existed a while and doesn't seem to be solving real problems, but maybe I'm wrong and it is actually in use already. Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper has some real advantages over using the database. But there is hesitance because it is not widely supported in OpenStack. What say you, OpenStack community? Should we keep ZooKeeper out of our.. zoo? Yes. I'm strongly opposed to ZooKeeper finding its way into the already complex pile of things we use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Migrating to newer full projects from what used to be part of nova
On 31 October 2013 18:46, John Griffith john.griff...@solidfire.com wrote: Is your disappointment that there isn't a path from Essex -- Grizzly/Havana, or are you unhappy with the content? Upgrading Essex-Folsom introduced both the challenge of upgrading nova-volume to cinder and the challenge of upgrading nova-network to quantum. Upgrading Folsom-Grizzly presents the challenge of migrating nova-network to quantum and assumes that nova-volume has already been migrated to cinder. Upgrading Grizzly-Havana finally closes the door on nova-network as far as I can see, although I may be wrong. It comes down to the fact that while we cater for migrating between versions of a particular project we don't cater particularly well for migrating between projects when a project splits out from a parent project as was the case for both of the above. I'm not disappointed about the lack of ability to jump from Essex to Grizzly or Havana, but rather the lack of ability to migrate between projects. I would find it entirely suitable to upgrade from Essex to Folsom, then migrate from nova-volume to cinder and from nova-network to quantum, then only to upgrade to Grizzly. So to be quite honest we never intended to make skips like you describe. This misses the point somewhat, as I've made (hopefully) clear above. It's not about the skipping, but rather the clean migration, even within the same version, from something that used to be inherent (like nova-volume and cinder) to something that is now a new project with its own code tree. Going forward maybe we can come up with something better. The reason for raising this now is primarily to do just that. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] HK Summit - Image Creation and Customization unconference?
On Thu, 2013-10-31 at 11:56 -0700, Clint Byrum wrote: Excerpts from Ian McLeod's message of 2013-10-31 11:27:39 -0700: Hello, Would any of you attending the summit be interested in snagging an unconference session to discuss the state of play with image creation, customization and import? I can contribute an overview and demonstration of our Nova-native image building tool. I'd be interested in exploring integration points with Disk Image Builder and Glance. Any takers? I am certain that one of the tripleo-core devs will want to be there, if not many of us. We don't have that many sessions scheduled so I'd suggest just trying to aim at when we're not having Deployment sessions. Those are exclusively on Tuesday, yes? http://icehousedesignsummit.sched.org/overview/type/tripleo+% 28deployment%29#.UnKs1x8u2al So any time Wednesday onward? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Migrating to newer full projects from what used to be part of nova
On 31 October 2013 19:05, Russell Bryant rbry...@redhat.com wrote: Right, Nova (and probably other projects, too) only supports N to N+1 upgrades. If you need to go to N+2, you'll need to go through N+1 to get there. Dead right, and especially with the aggressive schedule we have this is the only sane approach for now. FWIW, improving upgrades is very high on the priority list across all of OpenStack. We have sessions all over the place to talk about it. It's getting a lot better. Yes it is, and while the upgrades haven't been perfect so far the increased focus on getting the grenade testing in place are sure to make a big difference. I just feel that it's important to highlight the gap to bring it into the same discussion and hopefully raise some awareness for those who are in a prime position to make the difference. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On 10/30/2013 11:37 PM, Robert Collins wrote: This is a bit of a social norms thread I've been consistently asking for tests in reviews for a while now, and I get the occasional push-back. I think this falls into a few broad camps: A - there is no test suite at all, adding one in unreasonable B - this thing cannot be tested in this context (e.g. functional tests are defined in a different tree) C - this particular thing is very hard to test D - testing this won't offer benefit E - other things like this in the project don't have tests F - submitter doesn't know how to write tests G - submitter doesn't have time to write tests Now, of these, I think it's fine not add tests in cases A, B, C in combination with D, and D. I don't think E, F or G are sufficient reasons to merge something without tests, when reviewers are asking for them. G in the special case that the project really wants the patch landed - but then I'd expect reviewers to not ask for tests or to volunteer that they might be optional. Now, if I'm wrong, and folk have different norms about when to accept 'reason X not to write tests' as a response from the submitter - please let me know! I've done a lot of thinking around this topic [1][2] and really it comes down to this: everything can be tested and should be. There is an argument to A, but that goes beyond the scope of our use case I think. If I hear B, I would suspect the tests aren't unit tests, but are functional/integration tests (a common problem in OpenStack). Functional tests are brittle and usually have painful setup sequences. The other cases fall into the -1 camp for me. Tests required. That said, recently I was -1'ed for not updating a test, because I added code that didn't change the program flow, but introduced a new call. According to my rules, that didn't need a test, but I agreed with the logic that people would be upset if the call wasn't made (it was a notification). So a test was added. Totally valid argument. TL;DR: Tests are always required. We need to fix our tests to be proper unit tests and not functional/integration tests so it's easy to add new ones. -S [1] http://www.sandywalsh.com/2011/06/effective-units-tests-and-integration.html [2] http://www.sandywalsh.com/2011/08/pain-of-unit-tests-and-dynamically.html -Rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?
On 2013-10-31 13:30:32 + (+), Mark McLoughlin wrote: [...] In cases like that, I'd be of a mind to go +2 Awesome! Thanks for catching this! It would be great to have a unit test for this, but it's clear the current code is broken so I'm fine with merging the fix without a test. You could say it's now the reviewers responsibility to merge a test, but if that requirement then turns off reviewers even reviewing such a patch, then that doesn't help either. [...] I get the impression it was one of my +2s in a situation like this which spawned the thread (or at least was the straw which broke the camel's back), so apologies for stirring the beehive. Someone was trying to set up their own copy of several of our infrastructure projects, spotted a place in one where we had hard-coded something specific to our environment, and wanted to parameterize it upstream rather than paper over it on their end. The bug-reporter-turned-patch-contributor didn't know how to write a test which would confirm that parameter got passed through and we weren't performing tests yet for any similar parameters already in the daemon which I could point to as an example. I agree it's a judgement call between risking discouraging a new contributor vs. reducing test coverage further, but I was probably still a bit too hasty to suggest that we could add tests for those in a separate commit. In my case I didn't have the available time to instruct the contributor on how to write tests for this, but that also probably meant that I didn't really have time to be reviewing the change properly to begin with. I'm quite grateful to Robert for stepping in and helping walk them through it! We got more tests, I think they got a lot of benefit from the new experience as well, and I was appropriately humbled for my lax attitude over the situation which nearly allowed us to miss a great opportunity at educating another developer on the merits of test coverage. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [sqlalchemy-migrate] Blueprint for review: Add DB2 10.5 Support
I've got a sqlalchemy-migrate blueprint up for review to add DB2 support in migrate. https://blueprints.launchpad.net/sqlalchemy-migrate/+spec/add-db2-support This is a pre-req for getting DB2 support into Nova so I'm targeting icehouse-1. We've been running with the migrate patches internally since Folsom, but getting them into migrate was difficult before OpenStack took over maintenance of the project. Please let me know if there are any questions/issues or something I need to address here. Thanks, Matt Riedemann Cloud Solutions and OpenStack Development Email: mrie...@us.ibm.com Office Phone: 507-253-7622___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][keystone] Help with XML Tempest API tests
Hi all,
So firstly, if you're an XML guru, I apologize, the questions below are
probably really basic, I always prefer JSON or YAML, because every time I
deal with XML, I get a week-long-headache ;)
So I'm writing Tempest API tests for the keystone OS-TRUST extension, as
was previously requested - it's going pretty well, I'm finding some real
bugs. Here's a WIP review:
https://review.openstack.org/#/c/54810/
However, I've got a bit stuck formatting the POST body for the trust create
in XML (all works fine via JSON). The json body looks like:
{
trust: {
expires_at: 2013-02-27T18:30:59.99Z,
impersonation: true,
project_id: ddef321,
roles: [
{
name: member
}
],
trustee_user_id: 86c0d5,
trustor_user_id: a0fdfd
}
}
And looking at other XML tests which format a POST body, they do something
like:
post_body = Element(trust,
xmlns=XMLNS,
trustor_user_id=trustor_user_id,
trustee_user_id=trustee_user_id,
project_id=project_id,
impersonation=impersonation,
expires_at=expires_at)
This gives me a post body which looks weird (but seems to work):
trust impersonation=True
xmlns=http://docs.openstack.org/identity/api/v3;
trustor_user_id=efc6504105c54fbe95928a51459d06c9 expires_at=
trustee_user_id=f55efd1d617e4367891d202a811d7728
project_id=b5d498f9631244b59912ce2a0025cf8d/
So my questions are:
1. Why do we create a single element like this, instead of appending
subelements so the XML body looks more like the JSON request?
2. If any elements have a None value, they are encoded as a zero-length
string, is that expected?
3. How do I go about encoding the list of roles, as in the sample request
(it's a list of dicts, where each dict has one key called name)
Any help, review feedback or pointers to docs/examples would be hugely
appreciated!
Thanks,
Steve
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] Closes-Bug and Launchpad
Hi, Over the last few days I have noticed that bug fixes posted to gerrit are not updated in Launchpad. Am I doing something wrong? I think that the commit message is the correct format: Closes-Bug: #bug number. Any ideas? Thanks Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] HK Summit - Image Creation and Customization unconference?
On Thu, Oct 31, 2013 at 12:20 PM, Ian McLeod imcl...@redhat.com wrote: On Thu, 2013-10-31 at 11:56 -0700, Clint Byrum wrote: Excerpts from Ian McLeod's message of 2013-10-31 11:27:39 -0700: Hello, Would any of you attending the summit be interested in snagging an unconference session to discuss the state of play with image creation, customization and import? I can contribute an overview and demonstration of our Nova-native image building tool. I'd be interested in exploring integration points with Disk Image Builder and Glance. Any takers? Sounds like a good opportunity! I am certain that one of the tripleo-core devs will want to be there, if not many of us. We don't have that many sessions scheduled so I'd suggest just trying to aim at when we're not having Deployment sessions. Those are exclusively on Tuesday, yes? http://icehousedesignsummit.sched.org/overview/type/tripleo+% 28deployment%29#.UnKs1x8u2al So any time Wednesday onward? That broad schedule constraint is good for Glance as well, as all of our sessions are on Tuesday. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Closes-Bug and Launchpad
On 2013-10-31 12:43:19 -0700 (-0700), Gary Kotton wrote: Over the last few days I have noticed that bug fixes posted to gerrit are not updated in Launchpad. [...] Links to one or two example changes might help. There are a number of other subtle ways this can break down (wrong branch/no corresponding series task on the bug, missing bug supervisor permissions on that project for the Launchpad account used by our automation, transient API failures due to network or server issues, et cetera). I can probably spot the specific issue with a bit more context to go on. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] HK Summit - Image Creation and Customization unconference?
On Thu, 2013-10-31 at 12:50 -0700, Mark Washenberger wrote: On Thu, Oct 31, 2013 at 12:20 PM, Ian McLeod imcl...@redhat.com wrote: On Thu, 2013-10-31 at 11:56 -0700, Clint Byrum wrote: Excerpts from Ian McLeod's message of 2013-10-31 11:27:39 -0700: Hello, Would any of you attending the summit be interested in snagging an unconference session to discuss the state of play with image creation, customization and import? I can contribute an overview and demonstration of our Nova-native image building tool. I'd be interested in exploring integration points with Disk Image Builder and Glance. Any takers? Sounds like a good opportunity! So any time Wednesday onward? That broad schedule constraint is good for Glance as well, as all of our sessions are on Tuesday. Cool. I'm not arriving until late Monday. I've asked russellb to try to grab an unconference slot. However, I suppose I should just extend this request to anyone who is interested. So, could I ask that any interested parties look at the unconference signup sheet when you arrive and, if you don't already see: Image Building and Customization please grab a time slot from Wednesday onward? -Ian ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Horizon] Abdicating the PTL Position
Hi all, It saddens me to say that for a mix of reasons I have decided to abdicate my position as PTL for Horizon. If anything, the reasons are all good ones overall, I just have to make the right decision for both myself and the project. In the interim David Lyle will be the acting PTL. The Horizon core team has all weighed in with their confidence in his abilities, and he has confirmed his ability and interest in doing so. There will be a nomination period in coming weeks to determine the PTL for the full release cycle, should anyone else wish to run for the job as well. Thierry will announce more information about that soon. Rest assured, you're not rid of me entirely; I'm just making a decision to focus in on other areas for a time. Horizon is blessed to have other phenomenal candidates willing and able to lead, and I would rather see the project in the hands of someone who can devote their full attention and energy to it. I will still be around and engaged (though not in Hong Kong). I'll catch you all next time around. All the best, - Gabriel Hurley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] welcoming new committers (was Re: When is it okay for submitters to say 'I don't want to add tests' ?)
On Oct 31, 2013, at 1:49 PM, Stefano Maffulli stef...@openstack.org wrote: On 10/31/2013 07:05 AM, Kyle Mestery (kmestery) wrote: [...] If we want to grow the committer base and help people to become better reviewers, taking the time to show them the ropes is part of the game. hijacking the thread using Kyle's comment as an excuse. Hey, glad to provide you an opening Stef! It's not an 'if' but a 'since': since we are growing the committer base at an incredible pace we should help them become also good reviewers as rapidly possible. One thing I already mentioned and I'll start doing this week in the weekly Newsletter is to give a shoutout to those that do their first review this week. Another idea that Tom suggested is to use gerrit automation to send back to first time committers something in addition to the normal 'your patch is waiting for review' message. The message could be something like: thank you for your first contribution to OpenStack. Your patch will now be tested automatically by OpenStack testing frameworks and once the automatic tests pass, it will be reviewed by other friendly developers. They will give you comments and may require you to refine it. Nobody gets his patch approved at first try so don't be concerned when someone will require you to do more iterations. Patches usually take 3 to 7 days to be approved so be patient and be available on IRC to ask and answer questions about your work. The more you participate in the community the more rewarding it is for you. You may also notice that the more you get to know people and get to be known, the faster your patches will be reviewed and eventually approved. Get to know others and be known by doing code reviews: anybody can and should do it. With links to the wiki for more details, of course. This sort of messaging may help all the people that contribute tactically, those that are asked by their manager to land a patch in here and are simply lightly involved (not committed) in OpenStack. These are the ones that may have an incorrect perception of how easy it is to have patches landed in OpenStack as opposed to other large projects, like the kernel or android and complain about our time to traverse the review system. What do you think? How can we instruct gerrit to do this? I think this is a really good idea. I've seen occasions were new committers get antsy after waiting a few days (some even a few hours) and wondering why their patch isn't getting reviewed. Something like this would set the expectation for them correctly, and help to guide them to IRC to engage. Thanks, Kyle /stef PS I put the text on https://etherpad.openstack.org/p/welcome-new-committers for refinements. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] Abdicating the PTL Position
On 10/31/2013 04:06 PM, Gabriel Hurley wrote: Hi all, It saddens me to say that for a mix of reasons I have decided to abdicate my position as PTL for Horizon. If anything, the reasons are all good ones overall, I just have to make the right decision for both myself and the project. In the interim David Lyle will be the acting PTL. The Horizon core team has all weighed in with their confidence in his abilities, and he has confirmed his ability and interest in doing so. There will be a nomination period in coming weeks to determine the PTL for the full release cycle, should anyone else wish to run for the job as well. Thierry will announce more information about that soon. Rest assured, you're not rid of me entirely; I'm just making a decision to focus in on other areas for a time. Horizon is blessed to have other phenomenal candidates willing and able to lead, and I would rather see the project in the hands of someone who can devote their full attention and energy to it. I will still be around and engaged (though not in Hong Kong). I'll catch you all next time around. Thank you for your amazing work as PTL! I'm glad we're not rid of you entirely, but I am sad I'll miss you in Hong Kong. Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] A question about atomicity of repository commits
Hi, I'm new to the Openstack community. I have what is probably a naïve question. Suppose source code for a project is in Git repository A, while the Jenkins test suite for it is in Git repository B. First question: Does this happen in Openstack? (For example, are there nova tests in the tempest repository that get executed in Jenkins when nova is changed?) Given my assumption above, suppose I'm a contributor and I want to change some source in repository A. However, my source change affects the test results of a test in repository B. So I want to change that as well. My understanding is that I'd clone both repositories, and commit changes to each independently, and each would go to Gerrit as a separate change. Second question: How does one handle the chicken-and-egg problem? If the source change to A goes through Jenkins first, the automated tests would fail. But if the test change to B goes through Jenkins first, it would also fail because the old code in A has the old behavior. Thanks and kind regards, Dave ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] welcoming new committers (was Re: When is it okay for submitters to say 'I don't want to add tests' ?)
On 10/31/2013 04:15 PM, Kyle Mestery (kmestery) wrote: On Oct 31, 2013, at 1:49 PM, Stefano Maffulli stef...@openstack.org wrote: On 10/31/2013 07:05 AM, Kyle Mestery (kmestery) wrote: [...] If we want to grow the committer base and help people to become better reviewers, taking the time to show them the ropes is part of the game. hijacking the thread using Kyle's comment as an excuse. Hey, glad to provide you an opening Stef! It's not an 'if' but a 'since': since we are growing the committer base at an incredible pace we should help them become also good reviewers as rapidly possible. One thing I already mentioned and I'll start doing this week in the weekly Newsletter is to give a shoutout to those that do their first review this week. Another idea that Tom suggested is to use gerrit automation to send back to first time committers something in addition to the normal 'your patch is waiting for review' message. The message could be something like: thank you for your first contribution to OpenStack. Your patch will now be tested automatically by OpenStack testing frameworks and once the automatic tests pass, it will be reviewed by other friendly developers. They will give you comments and may require you to refine it. Nobody gets his patch approved at first try so don't be concerned when someone will require you to do more iterations. Patches usually take 3 to 7 days to be approved so be patient and be available on IRC to ask and answer questions about your work. The more you participate in the community the more rewarding it is for you. You may also notice that the more you get to know people and get to be known, the faster your patches will be reviewed and eventually approved. Get to know others and be known by doing code reviews: anybody can and should do it. With links to the wiki for more details, of course. This sort of messaging may help all the people that contribute tactically, those that are asked by their manager to land a patch in here and are simply lightly involved (not committed) in OpenStack. These are the ones that may have an incorrect perception of how easy it is to have patches landed in OpenStack as opposed to other large projects, like the kernel or android and complain about our time to traverse the review system. What do you think? How can we instruct gerrit to do this? I think this is a really good idea. I've seen occasions were new committers get antsy after waiting a few days (some even a few hours) and wondering why their patch isn't getting reviewed. Something like this would set the expectation for them correctly, and help to guide them to IRC to engage. I agree! I think this is an excelent idea, and I think it's totally implementable. I'm not sure what all the details will be of that implementation, but I'm certain it can be done. We're all crazy with summit - could you file a bug at bugs.launchpad.net/openstack-ci so that we don't lose track of it? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] welcoming new committers (was Re: When is it okay for submitters to say 'I don't want to add tests' ?)
On Oct 31, 2013, at 3:21 PM, Monty Taylor mord...@inaugust.com wrote: On 10/31/2013 04:15 PM, Kyle Mestery (kmestery) wrote: On Oct 31, 2013, at 1:49 PM, Stefano Maffulli stef...@openstack.org wrote: On 10/31/2013 07:05 AM, Kyle Mestery (kmestery) wrote: [...] If we want to grow the committer base and help people to become better reviewers, taking the time to show them the ropes is part of the game. hijacking the thread using Kyle's comment as an excuse. Hey, glad to provide you an opening Stef! It's not an 'if' but a 'since': since we are growing the committer base at an incredible pace we should help them become also good reviewers as rapidly possible. One thing I already mentioned and I'll start doing this week in the weekly Newsletter is to give a shoutout to those that do their first review this week. Another idea that Tom suggested is to use gerrit automation to send back to first time committers something in addition to the normal 'your patch is waiting for review' message. The message could be something like: thank you for your first contribution to OpenStack. Your patch will now be tested automatically by OpenStack testing frameworks and once the automatic tests pass, it will be reviewed by other friendly developers. They will give you comments and may require you to refine it. Nobody gets his patch approved at first try so don't be concerned when someone will require you to do more iterations. Patches usually take 3 to 7 days to be approved so be patient and be available on IRC to ask and answer questions about your work. The more you participate in the community the more rewarding it is for you. You may also notice that the more you get to know people and get to be known, the faster your patches will be reviewed and eventually approved. Get to know others and be known by doing code reviews: anybody can and should do it. With links to the wiki for more details, of course. This sort of messaging may help all the people that contribute tactically, those that are asked by their manager to land a patch in here and are simply lightly involved (not committed) in OpenStack. These are the ones that may have an incorrect perception of how easy it is to have patches landed in OpenStack as opposed to other large projects, like the kernel or android and complain about our time to traverse the review system. What do you think? How can we instruct gerrit to do this? I think this is a really good idea. I've seen occasions were new committers get antsy after waiting a few days (some even a few hours) and wondering why their patch isn't getting reviewed. Something like this would set the expectation for them correctly, and help to guide them to IRC to engage. I agree! I think this is an excelent idea, and I think it's totally implementable. I'm not sure what all the details will be of that implementation, but I'm certain it can be done. Awesome, thanks Monty! We're all crazy with summit - could you file a bug at bugs.launchpad.net/openstack-ci so that we don't lose track of it? Done: https://bugs.launchpad.net/openstack-ci/+bug/1246879 Thanks, Kyle ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Infra] [Tempest] [Ironic] Testing Ironic
Hi all, Ironic has reached a point where it is capable of being added to integration tests and the gate pipeline. Does that mean we /should/ add it? I think so, and I'd like to know what others think, but let me be more specific about what I'm talking about. At this point, it is possible for devstack to start ironic-api and ironic-conductor services, register them with keystone, init the db, etc. Then, tempest [1] can perform basic actions like create, update, delete records. Testing the hardware drivers will come later, which is where we'll see integration tests with glance, neutron, etc, become more important. Roman has been working on enabling these tests [2] and I'd like to nudge the reviewers of the tempest and infra teams to take a look at his patches. At the very least, I'd like to make folks aware of this work so we can all discuss it at the summit. [1] Tempest tests for Ironic API: https://review.openstack.org/#/c/48109 [2] Pre-cache Ironic to slaves: https://review.openstack.org/#/c/54569 Enable Ironic in devstack-gate: https://review.openstack.org/#/c/53899 Enable tempest tests in the experimental pipeline: https://review.openstack.org/#/c/53917 Thanks! Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] [savanna] [trove] Place for software configuration
Excerpts from Alexander Kuznetsov's message of 2013-10-31 10:51:54 -0700: Hi Heat, Savanna and Trove teams, All this projects have common part related to software configuration management. For creation, an environment user should specify a hardware parameter for vms: choose flavor, decide use cinder or not, configure networks for virtual machines, choose topology for hole deployment. Next step is linking of software parameters with hardware specification. From the end user point of view, existence of three different places and three different ways (HEAT Hot DSL, Trove clustering API and Savanna Hadoop templates) for software configuration is not convenient, especially if user want to create an environment simultaneously involving components from Savanna, Heat and Trove. I'm having a hard time extracting the problem statement. I _think_ that the problem is: As a user I want to tune my software for my available hardware. So what you're saying is, if you select a flavor that has 4GB of RAM for your application, you want to also tell your application that it can use 3GB of RAM for an in-memory cache. Likewise, if one has asked Trove for an 8GB flavor, they will want to tell it to use 6.5GB of RAM for InnoDB buffer cache. What you'd like to see is one general pattern to express these types of things? I can suggest two approaches to overcome this situations: Common library in oslo. This approach allows a deep domain specific customization. The user will still have 3 places with same UI where user should perform configuration actions. Heat or some other component for software configuration management. This approach is the best for end users. In feature possible will be some limitation on deep domain specific customization for configuration management. Can you maybe be more concrete with your proposed solutions? The lack of a clear problem statement combined with these vague solutions has thoroughly confused me. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] A question about atomicity of repository commits
On Thu, Oct 31, 2013 at 1:20 PM, Birdsall, Dave dave.birds...@hp.com wrote: Hi, I'm new to the Openstack community. I have what is probably a naïve question. Suppose source code for a project is in Git repository A, while the Jenkins test suite for it is in Git repository B. First question: Does this happen in Openstack? (For example, are there nova tests in the tempest repository that get executed in Jenkins when nova is changed?) Given my assumption above, suppose I'm a contributor and I want to change some source in repository A. However, my source change affects the test results of a test in repository B. So I want to change that as well. My understanding is that I'd clone both repositories, and commit changes to each independently, and each would go to Gerrit as a separate change. Correct. You will probably want to drop a comment on each review referring to the other so that reviewers understand what is going on. Second question: How does one handle the chicken-and-egg problem? If the source change to A goes through Jenkins first, the automated tests would fail. But if the test change to B goes through Jenkins first, it would also fail because the old code in A has the old behavior. You have to make the changes in backward and forward compatible ways, this will allow the tests to pass for at least one of the changes which will then unblock the other change. Thanks and kind regards, Dave Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] A question about atomicity of repository commits
On 1 November 2013 09:20, Birdsall, Dave dave.birds...@hp.com wrote: Hi, I'm new to the Openstack community. I have what is probably a naïve question. Suppose source code for a project is in Git repository A, while the Jenkins test suite for it is in Git repository B. First question: Does this happen in Openstack? (For example, are there nova tests in the tempest repository that get executed in Jenkins when nova is changed?) Yes, in the devstack-gate-*-tempest jobs. Given my assumption above, suppose I'm a contributor and I want to change some source in repository A. However, my source change affects the test results of a test in repository B. So I want to change that as well. My understanding is that I'd clone both repositories, and commit changes to each independently, and each would go to Gerrit as a separate change. Yes. Second question: How does one handle the chicken-and-egg problem? If the source change to A goes through Jenkins first, the automated tests would fail. But if the test change to B goes through Jenkins first, it would also fail because the old code in A has the old behavior. There are a few nuances here. Firstly, tests in separate repositories are usually loosely coupled, so most such changes should not need changes to tests in concert with the code in the other repo. Secondly, we hold backwards compatibility as an important value, so where such changes might need changes in concert, we often will choose to make the change backwards compatible, avoiding any issue. In the rare care where we can't make it backwards compatible, and the test will be broken, we can disable the test, land the new change, enable the test, or - better - make the test handle both behaviours, land the new change, make the test only handle the new behaviour. Thanks and kind regards, Dave ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Solum] HOLD THE DATES: Design Workshop @SFO
BEGIN:VCALENDAR METHOD:REQUEST PRODID:Microsoft Exchange Server 2010 VERSION:2.0 BEGIN:VTIMEZONE TZID:Central Standard Time BEGIN:STANDARD DTSTART:16010101T02 TZOFFSETFROM:-0500 TZOFFSETTO:-0600 RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11 END:STANDARD BEGIN:DAYLIGHT DTSTART:16010101T02 TZOFFSETFROM:-0600 TZOFFSETTO:-0500 RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3 END:DAYLIGHT END:VTIMEZONE BEGIN:VEVENT ORGANIZER;CN=Roshan Agrawal:MAILTO:roshan.agra...@rackspace.com ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN=openstack- d...@lists.openstack.org:MAILTO:openstack-dev@lists.openstack.org DESCRIPTION;LANGUAGE=en-US:When: Tuesday\, November 19\, 2013\, 9:30 AM to Wednesday\, November 20\, 2013\, 3:00 PM. (UTC-06:00) Central Time (US C anada)\nWhere: Rackspace SFO office\n\n*~*~*~*~*~*~*~*~*~*\n\nBlocking tim e on the calendars for the 2 day design workshop at SFO. More details to f ollow\, but here is the current plan:\n\n\nWorkshop sessions\nNov 19 Tuesd ay 9:30 am - 5 pm\nNov 20 Wednesday 9:30 am - 3:00 pm (we break early on W ednesday if some of us want to catch an evening return flight)\n\n\nEtherp ad for event planning-\nhttps://etherpad.openstack.org/p/SolumSFOCommunity Workshop\n\n SUMMARY;LANGUAGE=en-US:[Solum] HOLD THE DATES: Design Workshop @SFO DTSTART;TZID=Central Standard Time:20131119T093000 DTEND;TZID=Central Standard Time:20131120T15 UID:04008200E00074C5B7101A82E008117CE6277ED6CE01000 01000EB5D62ED45B251458DF7C660180FDDBA CLASS:PUBLIC PRIORITY:5 DTSTAMP:20131031T211416Z TRANSP:OPAQUE STATUS:CONFIRMED SEQUENCE:0 LOCATION;LANGUAGE=en-US:Rackspace SFO office X-MICROSOFT-CDO-APPT-SEQUENCE:0 X-MICROSOFT-CDO-OWNERAPPTID:2111502609 X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY X-MICROSOFT-CDO-ALLDAYEVENT:FALSE X-MICROSOFT-CDO-IMPORTANCE:1 X-MICROSOFT-CDO-INSTTYPE:0 X-MICROSOFT-DISALLOW-COUNTER:FALSE BEGIN:VALARM ACTION:DISPLAY DESCRIPTION:REMINDER TRIGGER;RELATED=START:-P1D END:VALARM END:VEVENT END:VCALENDAR ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] welcoming new committers (was Re: When is it okay for submitters to say 'I don't want to add tests' ?)
On Thu, 2013-10-31 at 11:49 -0700, Stefano Maffulli wrote: Another idea that Tom suggested is to use gerrit automation to send back to first time committers something in addition to the normal 'your patch is waiting for review' message. The message could be something like: thank you for your first contribution to OpenStack. Your patch will now be tested automatically by OpenStack testing frameworks and once the automatic tests pass, it will be reviewed by other friendly developers. They will give you comments and may require you to refine it. Nobody gets his patch approved at first try so don't be concerned when someone will require you to do more iterations. Patches usually take 3 to 7 days to be approved so be patient and be available on IRC to ask and answer questions about your work. The more you participate in the community the more rewarding it is for you. You may also notice that the more you get to know people and get to be known, the faster your patches will be reviewed and eventually approved. Get to know others and be known by doing code reviews: anybody can and should do it. ... Very nicely done. I really like it. Thanks, Mark. PS I put the text on https://etherpad.openstack.org/p/welcome-new-committers for refinements. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] Abdicating the PTL Position
Hey, I'm just curious. You quitting the PTL position to focus on Nebula efforts or ? 2013/10/31 Gabriel Hurley gabriel.hur...@nebula.com Hi all, It saddens me to say that for a mix of reasons I have decided to abdicate my position as PTL for Horizon. If anything, the reasons are all good ones overall, I just have to make the right decision for both myself and the project. In the interim David Lyle will be the acting PTL. The Horizon core team has all weighed in with their confidence in his abilities, and he has confirmed his ability and interest in doing so. There will be a nomination period in coming weeks to determine the PTL for the full release cycle, should anyone else wish to run for the job as well. Thierry will announce more information about that soon. Rest assured, you're not rid of me entirely; I'm just making a decision to focus in on other areas for a time. Horizon is blessed to have other phenomenal candidates willing and able to lead, and I would rather see the project in the hands of someone who can devote their full attention and energy to it. I will still be around and engaged (though not in Hong Kong). I'll catch you all next time around. All the best, - Gabriel Hurley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] Abdicating the PTL Position
Nevermind and ignore the last e-mail, it was wrongly intended. Endre 2013/10/31 Endre Karlson endre.karl...@gmail.com Hey, I'm just curious. You quitting the PTL position to focus on Nebula efforts or ? 2013/10/31 Gabriel Hurley gabriel.hur...@nebula.com Hi all, It saddens me to say that for a mix of reasons I have decided to abdicate my position as PTL for Horizon. If anything, the reasons are all good ones overall, I just have to make the right decision for both myself and the project. In the interim David Lyle will be the acting PTL. The Horizon core team has all weighed in with their confidence in his abilities, and he has confirmed his ability and interest in doing so. There will be a nomination period in coming weeks to determine the PTL for the full release cycle, should anyone else wish to run for the job as well. Thierry will announce more information about that soon. Rest assured, you're not rid of me entirely; I'm just making a decision to focus in on other areas for a time. Horizon is blessed to have other phenomenal candidates willing and able to lead, and I would rather see the project in the hands of someone who can devote their full attention and energy to it. I will still be around and engaged (though not in Hong Kong). I'll catch you all next time around. All the best, - Gabriel Hurley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Horizon] Abdicating the PTL Position
@Gabriel, thanks for an awesome time leading Horizon in towards what is now a awesome Framework / UI for OpenStack! I'm sure you'll bring awesome to whatever you're doing next! It's very sad to see good people leave the PTL positions but hey, time goes by and people do new things. Good luck and see you around dude, it was awesome to meet you in Portland in the last summit :) Endre. 2013/10/31 Endre Karlson endre.karl...@gmail.com Nevermind and ignore the last e-mail, it was wrongly intended. Endre 2013/10/31 Endre Karlson endre.karl...@gmail.com Hey, I'm just curious. You quitting the PTL position to focus on Nebula efforts or ? 2013/10/31 Gabriel Hurley gabriel.hur...@nebula.com Hi all, It saddens me to say that for a mix of reasons I have decided to abdicate my position as PTL for Horizon. If anything, the reasons are all good ones overall, I just have to make the right decision for both myself and the project. In the interim David Lyle will be the acting PTL. The Horizon core team has all weighed in with their confidence in his abilities, and he has confirmed his ability and interest in doing so. There will be a nomination period in coming weeks to determine the PTL for the full release cycle, should anyone else wish to run for the job as well. Thierry will announce more information about that soon. Rest assured, you're not rid of me entirely; I'm just making a decision to focus in on other areas for a time. Horizon is blessed to have other phenomenal candidates willing and able to lead, and I would rather see the project in the hands of someone who can devote their full attention and energy to it. I will still be around and engaged (though not in Hong Kong). I'll catch you all next time around. All the best, - Gabriel Hurley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
