[openstack-dev] Cinder + taskflow

2014-02-03 Thread Joshua Harlow
Hi all, After talking with john g. about taskflow in cinder and seeing more and more reviews showing up I wanted to start a thread to gather all our lessons learned and how we can improve a little before continuing to add too many more refactoring and more reviews (making sure everyone is understa

[openstack-dev] [State-Management] Agenda for tommorow meeting at 2000 UTC

2014-01-29 Thread Joshua Harlow
said library. - Discuss about any other ideas, questions and answers (and more!). Any other topics are welcome :-) See you all soon! -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-inc.com ___ OpenStack-dev mailing list Open

Re: [openstack-dev] [Nova][Neutron] nova-network in Icehouse and beyond

2014-01-29 Thread Joshua Harlow
I think depending on use-case it can be accomplished. The steps we have been thinking at Y! 1. Take offline APIs & nova-compute (so new/existing VMs can't be scheduled/modified) -- existing running VMs will not be affected. 2. Save/dump nova database. 3. Translate nova database entries into corre

[openstack-dev] Hacking repair scripts

2014-01-27 Thread Joshua Harlow
Hi all, I have had in my ~/bin for a while a little script that I finally got around to tuning up and I thought others might be interested in it/find it useful. The concept is similar to https://pypi.python.org/pypi/autopep8 but does a really simple action to start. As many of u know the impor

Re: [openstack-dev] Mistral + taskflow mini-meetup

2014-01-27 Thread Joshua Harlow
Flow but moving forward we we’d like to try to marry these two technologies to be more aligned in terms of APIs and feature sets. Renat Akhmerov @ Mirantis Inc. On 27 Jan 2014, at 13:21, Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote: Hi all, In order to encourage further discussion of

Re: [openstack-dev] [oslo] log message translations

2014-01-27 Thread Joshua Harlow
+1 I've never understood this either personally. >From what I know most all (correct me if I am wrong) open source projects don't translate log messages; so it seems odd to be the special snowflake project/s. Do people find this type of translation useful? It'd be nice to know how many people re

[openstack-dev] Mistral + taskflow mini-meetup

2014-01-27 Thread Joshua Harlow
Hi all, In order to encourage further discussion off IRC and more in public I'd like to share a etherpad that was worked on during a 'meetup' with some of the mistral folks and me. https://etherpad.openstack.org/p/taskflow-mistral-jan-meetup It was more of a (mini) in-person meetup but I thoug

Re: [openstack-dev] [Ironic] Node groups and multi-node operations

2014-01-26 Thread Joshua Harlow
Jan 26, 2014, at 12:25 PM, "Robert Collins" > wrote: > >> On 27 January 2014 08:04, Joshua Harlow wrote: >> Doesn't nova already have logic for creating N virtual machines (similar to >> a group) in the same request? I thought it did (maybe it doesn'

Re: [openstack-dev] [Ironic] Node groups and multi-node operations

2014-01-26 Thread Joshua Harlow
Doesn't nova already have logic for creating N virtual machines (similar to a group) in the same request? I thought it did (maybe it doesn't anymore in the v3 API), creating N bare metal machines seems like it would comply to that api? Sent from my really tiny device... > On Jan 22, 2014, at 4:

Re: [openstack-dev] Fwd: [TripleO] mid-cycle meetup?

2014-01-25 Thread Joshua Harlow
nnyvale, but it's good to know we have options if we overflow available meeting rooms or suchlike :) Cheers, -- Chris Jones On 25 Jan 2014, at 03:22, Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote: Yahoo! is right in the middle of sunnyvale; and probably has enough free space

Re: [openstack-dev] Fwd: [TripleO] mid-cycle meetup?

2014-01-24 Thread Joshua Harlow
Yahoo! is right in the middle of sunnyvale; and probably has enough free space to handle all u folks (some rooms are quite big really). I can talk with some folks here about hosting all of this here if that’s desired (depending on how big). -Josh From: Roman Alekseenkov mailto:ralekseen...@mi

Re: [openstack-dev] [Ironic] File Injection (and the lack thereof)

2014-01-24 Thread Joshua Harlow
Also just to note; file-injection seems unneeded when cloud-init can use this: http://cloudinit.readthedocs.org/en/latest/topics/examples.html#writing-out -arbitrary-files That I believe is in most modern versions of cloud-init (forgot when I implemented that). Just FYI :) -Josh On 1/24/14, 3:

Re: [openstack-dev] [Ironic] File Injection (and the lack thereof)

2014-01-24 Thread Joshua Harlow
Cloud-init 0.7.5 (not yet released) will have the ability to read from an ec2-metadata server using SSL. In a recent change I did we now use requests which correctly does SSL for the ec2-metadata/ec2-userdata reading. - http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/910 Fo

Re: [openstack-dev] Reviewing reviewers

2014-01-23 Thread Joshua Harlow
Another one: https://github.com/harlowja/gerrit_view#qgerrit That's similar to :) Sent from my really tiny device... > On Jan 23, 2014, at 2:42 PM, "Zane Bitter" wrote: > > I don't know about other projects, but we in the Heat project are constantly > on the lookout for people who can be con

Re: [openstack-dev] [oslo] memoizer aka cache

2014-01-23 Thread Joshua Harlow
ear easy and simple to initially do/use, but doing it correctly is really really hard (especially at any type of scale). -Josh On 1/23/14, 1:35 PM, "Renat Akhmerov" mailto:rakhme...@mirantis.com>> wrote: > >On 23 Jan 2014, at 08:41, Joshua Harlow >mailto:harlo...@yahoo-inc.c

Re: [openstack-dev] [oslo] memoizer aka cache

2014-01-23 Thread Joshua Harlow
at any type of scale). -Josh On 1/23/14, 1:35 PM, "Renat Akhmerov" wrote: > >On 23 Jan 2014, at 08:41, Joshua Harlow wrote: > >> So to me memoizing is typically a premature optimization in a lot of >>cases. And doing it incorrectly leads to overfilling the python

Re: [openstack-dev] [oslo] memoizer aka cache

2014-01-23 Thread Joshua Harlow
So to me memoizing is typically a premature optimization in a lot of cases. And doing it incorrectly leads to overfilling the python processes memory (your global dict will have objects in it that can't be garbage collected, and with enough keys+values being stored will act just like a memory le

[openstack-dev] [State-Management] Agenda for tommorow meeting at 2000 UTC

2014-01-22 Thread Joshua Harlow
See you all soon! -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-inc.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [State-Management] Proposal to add Changbin Liu to taskflow-core

2014-01-16 Thread Joshua Harlow
Greetings all stackers, I propose that we add Changbin Liu[1] to the taskflow-core team[2]. Changbin has been actively contributing to taskflow for a while now, both in helping develop code and helping with the review load. He has provided quality reviews and is doing an awesome job with the vari

Re: [openstack-dev] sqla 0.8 ... and sqla 0.9

2014-01-16 Thread Joshua Harlow
Also with: https://review.openstack.org/#/c/66051/ On 1/16/14, 10:40 AM, "Jeremy Stanley" wrote: >On 2014-01-12 07:27:11 -0500 (-0500), Sean Dague wrote: >> With the taskflow update, the only thing between upping our sqla >> requirement to < 0.8.99 is pbr's requirements integration test >> gett

[openstack-dev] [State-Management] Agenda for tommorow meeting at 2000 UTC

2014-01-15 Thread Joshua Harlow
, feedback...) - Any other ongoing taskflow work (questions, comments, feedback...) - Discuss about any other potential new use-cases for said library. - Discuss about any other ideas, questions and answers (and more!). Any other topics are welcome :-) See you all soon! -- Joshua Harlow

Re: [openstack-dev] [oslo] dependency analysis for graduating more oslo libraries

2014-01-15 Thread Joshua Harlow
Very nice write up +1 A question, many modules are pulling in oslo.log as seen from the dependency graph which itself then pulls in oslo.config. Is the plan to just have all these modules use the regular python logging and have oslo.log be a plugin/formatter/adapter to python logging? Likely p

Re: [openstack-dev] [nova][neutron] top gate bugs: a plea for help

2014-01-11 Thread Joshua Harlow
+1 Very interesting to read about these bottlenecks and very grateful they are being addressed. Sent from my really tiny device... > On Jan 11, 2014, at 8:44 AM, "Sean Dague" wrote: > > First, thanks a ton for diving in on all this Russell. The big push by the > Nova team recently is really

Re: [openstack-dev] [all] Organizing a Gate Blocking Bug Fix Day

2014-01-09 Thread Joshua Harlow
And my ring, my precious. Count me in! On 1/9/14, 6:06 AM, "Julien Danjou" wrote: >On Thu, Jan 09 2014, Sean Dague wrote: > >> I'm hopefully that if we can get everyone looking at this one a single >>day, >> we can start to dislodge the log jam that exists. > >I will help you bear this burd

[openstack-dev] [State-Management] Agenda for tommorow meeting at 2000 UTC

2014-01-08 Thread Joshua Harlow
welcome :-) See you all soon! -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-inc.com<mailto:harlo...@yahoo-inc.com> ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailm

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-06 Thread Joshua Harlow
velopment Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade On Sun, Jan 5, 2014 at 5:02 PM, James E. Blair mailto:jebl...@openstack.org>> wrote: Joshua Harl

Re: [openstack-dev] [Gantt] Looking for some answers...

2014-01-06 Thread Joshua Harlow
Was the history filtered out using something like http://git-scm.com/docs/git-filter-branch?? There seems to be a lot of commit history that isn't related to gantt files (baremetal…) Was the plan to figure out which files to keep, then cleanup that commit history? I wouldn't expect https://g

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-05 Thread Joshua Harlow
Agreed, we are going to expand it and work on figuring out how to test against multiple versions. It does work with 0.8 and it seems even like 0.9 works fine also. But all compatible also means I can't guarantee 0.10 (if it comes out) will work since afaik semver means sqlalchemy could still bre

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-05 Thread Joshua Harlow
With regards to the futures module it should just work fine with the packaging of https://pypi.python.org/pypi/futures which is a backport of the 3.2 concurrent futures package into 2.6,2.7, so that's the package there. Feel free to bug me on irc if u want any other help with dependencies, the

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-04 Thread Joshua Harlow
I was more of referring to general dependency issues, sqlalchemy hopefully never again but one never knows :P Sent from my really tiny device... > On Jan 4, 2014, at 8:40 AM, "Thomas Goirand" wrote: > >> On 01/05/2014 12:12 AM, Joshua Harlow wrote: >> it won

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-04 Thread Joshua Harlow
Such a bad state seems like FUD. Taskflow was just syncing its requirements with the same requirements that everyone else is... Those global requirements have <0.7.99 in them as we speak (which is why taskflow picked up that version). The issue here will be worked through and fixed, it won't b

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-03 Thread Joshua Harlow
really sure which is the best, none seem super-great, but sometimes u just work with what u got :-P On 1/3/14, 3:32 PM, "Sean Dague" wrote: >On 01/03/2014 06:14 PM, Joshua Harlow wrote: >> Since the library model is what most everyone else uses outside of >> opens

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-03 Thread Joshua Harlow
Since the library model is what most everyone else uses outside of openstack (I assume?) what can we do to get there so that this model works as well? Expanding dependencies recursively seems like it could help? This could then detect transitive dependency issues (and doesn't seem so hard to do).

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-03 Thread Joshua Harlow
low/+bug/1265886). If we really need to I can push out a 0.1.2 with the unpinned version (one of the above reviews). -Josh From: Doug Hellmann mailto:doug.hellm...@dreamhost.com>> Date: Friday, January 3, 2014 at 11:44 AM To: Joshua Harlow mailto:harlo...@yahoo-inc.com>> Cc: "O

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-03 Thread Joshua Harlow
Sounds good to me. Talked on #openstack-infra with some folks there and just awaiting next steps. Doesn't seem like should be anything to hard to adjust/move/... -Josh On 1/3/14, 11:27 AM, "Sean Dague" wrote: >On 01/03/2014 12:45 PM, Joshua Harlow wrote: >> Ok, I

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-03 Thread Joshua Harlow
e-oslo-splitting-the-incubator would seem to cause even more of these types of new 3rd party libraries to appear on pypi (and therefore causing similar issues of transitive dependencies as taskflow). Will bug u on #openstack-infra soon :-) On 1/3/14, 9:05 AM, "Sean Dague" wrote: >

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

2014-01-03 Thread Joshua Harlow
So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy>=0.7

Re: [openstack-dev] [trove][mistral] scheduled tasks

2013-12-31 Thread Joshua Harlow
Agreed taskflow doesn't currently provide scheduling as it was thought that reliable execution that can be restarted and resumed is the foundation that someone using taskflow can easily provide scheduling ontop of... Better IMHO to have a project doing this foundation well (since openstack would

Re: [openstack-dev] [trove][mistral] scheduled tasks

2013-12-30 Thread Joshua Harlow
Any reason to not use taskflow (https://wiki.openstack.org/wiki/TaskFlow) to help u here?? I think it could be easily adapted to do what u want, and would save u from having to recreate the same task execution logic that everyone seems to rebuild… -Josh From: Greg Hill mailto:greg.h...@rackspa

[openstack-dev] [State-Management] No meeting this week

2013-12-23 Thread Joshua Harlow
Since its xmas in most of the world lets skip the IRC meeting this week (normally on thursdays). See you all soon and have a great vacation! P.S. #openstack-state-management if u feel the need to chat :-) -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-in

[openstack-dev] [State-Management] Agenda for today meeting at 2000 UTC

2013-12-19 Thread Joshua Harlow
/TaskFlow ## Agenda (30-60 mins): - Discuss any action items from last meeting. - Discuss about any other potential new use-cases for said library. - Discuss about any other ideas, questions and answers (and more!). Any other topics are welcome :-) See you all soon! -- Joshua Harlow

Re: [openstack-dev] Re-initializing or dynamically configuring cinder driver

2013-12-16 Thread Joshua Harlow
quot;iKhan" mailto:ik.ibadk...@gmail.com>> wrote: Ok, I though we can make make cinder-volume aware of SIGTERM call and make sure it terminates with cleaning all the existing operations. If its not possible then probably SIGHUB is the only solution. :( On Mon, Dec 16, 2013 at 10:25 AM, Joshua

Re: [openstack-dev] Re-initializing or dynamically configuring cinder driver

2013-12-15 Thread Joshua Harlow
w this is the hard way of achieving the objective and SIGHUP approach will handle it more gracefully. As you mentioned it is a major change, tentatively can we use SIGTERM to achieve the objective? On Mon, Dec 16, 2013 at 9:50 AM, Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote: In your

Re: [openstack-dev] Re-initializing or dynamically configuring cinder driver

2013-12-15 Thread Joshua Harlow
n Sun, Dec 15, 2013 at 12:49 PM, Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote: I don't currently know of a one size fits all solution here. There was talk at the summit of having the cinder app respond to a SIGHUP signal and attempting to reload config on this signal. Dynamic reloa

Re: [openstack-dev] Re-initializing or dynamically configuring cinder driver

2013-12-14 Thread Joshua Harlow
I don't currently know of a one size fits all solution here. There was talk at the summit of having the cinder app respond to a SIGHUP signal and attempting to reload config on this signal. Dynamic reloading is tricky business (basically u need to unravel anything holding references to the old c

Re: [openstack-dev] [Nova] [Neutron] How do we know a host is ready to have servers scheduled onto it?

2013-12-12 Thread Joshua Harlow
Maybe time to revive something like: https://review.openstack.org/#/c/12759/ >From experience, all sites (and those internal to yahoo) provide a /status (or equivalent) that is used for all sorts of things (from basic load-balancing up/down) to other things like actually introspecting the state

[openstack-dev] [State-Management] Agenda for tomorrow meeting at 2000 UTC

2013-12-11 Thread Joshua Harlow
topics are welcome :-) See you all soon! -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-inc.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] FW: [Bug 1257788] Re: nova-network doesn't handle large amount of concurrent booting instances well

2013-12-10 Thread Joshua Harlow
Yup, IMHO this is where the rally[1] project, and others that attempt to push the boundaries are really helping openstack become much better around these types of issues. At yahoo! in a short-time it will not be that uncommon to boot 100+ VMs concurrently (traffic spikes, other random QA testing th

Re: [openstack-dev] Unified Guest Agent proposal

2013-12-07 Thread Joshua Harlow
Sounds like a pretty neat idea. I like it! Another idea: instead of saying pushing for one agent to rule them all why don't we come up with a desired reference spec (maybe including a reference implementation?) that the salt, chef, mcollective, (other...) can base theres off of. In a way this

Re: [openstack-dev] Unified Guest Agent proposal

2013-12-06 Thread Joshua Harlow
Agreed, Chatting earlier today on #cloud-init about all of this I think scott convinced me that maybe we (the joint we in the community at large) should think about asking ourselves what do we really want a guest agent for/to do? If it's for software installation or user management then aren't

Re: [openstack-dev] Unified Guest Agent proposal

2013-12-06 Thread Joshua Harlow
Another idea that I'll put up for consideration (since I work with the cloud-init codebase also). Cloud-init[1] which currently does lots of little useful initialization types of activities (similar to the racker agents activities) has been going through some of the same questions[2] as to should

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-12-06 Thread Joshua Harlow
Could jsonschema[1] be used here to do the options schema part? It works on dictionaries (and really isn't tied to json). But maybe I am missing some greater context/understanding (see other emails). [1] https://pypi.python.org/pypi/jsonschema Sent from my really tiny device... > On Dec 6, 201

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-12-06 Thread Joshua Harlow
the same way u wrote it, part of this ML should be about teaching others your viewpoints, not getting frustrated over simple things like misunderstandings... Sent from my really tiny device... > On Dec 6, 2013, at 9:16 AM, "Mark McLoughlin" wrote: > >> On Fri, 2013-12-06

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-12-06 Thread Joshua Harlow
Previous not precious, ha, durn autocorrect, lol. Sent from my really tiny device... > On Dec 6, 2013, at 9:50 AM, "Joshua Harlow" wrote: > > Forgive me for not understanding your precious email (which I guess was > confusing for me to understand). This one clears that

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-12-06 Thread Joshua Harlow
I really have to agree with this. It's especially important if oslo.messaging is also used in libraries like taskflow. If oslo.messaging imposes that users of it must use oslo.config then by using it in taskflow, taskflow then imposes the same oslo.config usage. This makes all libraries that use

[openstack-dev] [State-Management] Agenda for tomorrow meeting at 2000 UTC

2013-12-04 Thread Joshua Harlow
soon! -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-inc.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] When should things get added to Oslo.Incubator

2013-12-03 Thread Joshua Harlow
quot;Mark McLoughlin" wrote: >On Tue, 2013-12-03 at 22:31 +, Joshua Harlow wrote: >> Sure, no one has said it. But it seems to be implied, otherwise these >> types of discussions wouldn't occur. Right? > >You're assuming the Nova objects API is at a point

Re: [openstack-dev] When should things get added to Oslo.Incubator

2013-12-03 Thread Joshua Harlow
Sure, no one has said it. But it seems to be implied, otherwise these types of discussions wouldn't occur. Right? On 12/3/13 2:25 PM, "Mark McLoughlin" wrote: >On Tue, 2013-12-03 at 22:07 +0000, Joshua Harlow wrote: > >> Process for process sake imho has been a pro

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-12-03 Thread Joshua Harlow
oslo.messaging (so maybe it can be adjusted to work with things other than eventlet to make it python33 compat). On 12/3/13 2:01 AM, "Julien Danjou" wrote: >On Mon, Dec 02 2013, Joshua Harlow wrote: > >> Thanks for writing this up, looking forward to seeing this happen so >>t

Re: [openstack-dev] When should things get added to Oslo.Incubator

2013-12-03 Thread Joshua Harlow
Why not just make it a pypi party library from the start? Call it 'super-objects' or something. Why does it have to be connected to oslo.incubator? Process for process sake imho has been a problem for oslo. Why not just go straight to building a library (does it matter if it's in oslo?) that is

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-12-02 Thread Joshua Harlow
Thanks for writing this up, looking forward to seeing this happen so that oslo.messaging can be used outside of the core openstack projects (and be used in libraries that do not want to force a oslo.cfg model onto users of said libraries). Any idea of a timeline as to when this would be reflected

[openstack-dev] [State-Management] No meeting tommorow

2013-11-27 Thread Joshua Harlow
Hi all state-management folks (and others), Since its turkey day (thanksgiving) for most of the people in the U.S. lets skip the meeting this week. #openstack-state-management if u need to find us anyway. See u next week folks! -Josh ___ OpenStac

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Joshua Harlow
+2 Sent from my really tiny device... > On Nov 25, 2013, at 5:02 AM, "Davanum Srinivas" wrote: > > Many thanks to everyone who helped with the many fixes. Kudos to > Joe/Clark for spear heading the effort! > > -- dims > >> On Mon, Nov 25, 2013 at 12:00 AM, Joe Gordon wrote: >> Hi All, >> >>

Re: [openstack-dev] [Nova] Remove ratelimits from nova v3 api

2013-11-25 Thread Joshua Harlow
+2 Turnstile seems to be a good middle ground (other companies I know have solutions for this problem via other software), does anyone have operational knowledge they can share about how turnstile has worked out for them? Sent from my really tiny device... > On Nov 25, 2013, at 9:22 AM, "Kevin

Re: [openstack-dev] tenant or project

2013-11-24 Thread Joshua Harlow
+2 This kind of confusion is actually very bad from an external perspective and from a user perspective. Should this just get resolved by the TC once and for all? I remember this same project vs tenant question happening like 2 years ago (maybe less) and it makes us all look sort if "mad" if

Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration

2013-11-20 Thread Joshua Harlow
rom you just now. I will probably join 'State management' IRC meeting. ( If I could get up early:) or join 'Cinder' IRC meeting) Haruka Tanizawa 2013/11/21 Joshua Harlow mailto:harlo...@yahoo-inc.com>> Howdy! My guess is there is very much enough work to go around and it

[openstack-dev] [State-Management] Agenda for tomorrow meeting at 2000 UTC

2013-11-20 Thread Joshua Harlow
other potential new use-cases for said library. - Discuss about any other ideas, problems, open-reviews, issues, solutions, questions (and more!). Any other topics are welcome :-) See you all soon! -- Joshua Harlow It's openstack, relax... | harlo...@yahoo-in

Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration

2013-11-20 Thread Joshua Harlow
w is important for me from point of cancelling with using taskflow. So, let me help you to impl taskflow to API. Sincerely, Haruka Tanizawa 2013/11/20 Joshua Harlow mailto:harlo...@yahoo-inc.com>> Sweet! Feel free to ask lots of questions and jump on both irc channels (#openstack-stat

Re: [openstack-dev] Propose "project story wiki" idea

2013-11-20 Thread Joshua Harlow
Agreed I like the idea. It reminds me of the blog the solum team is setting up. I think I asked then when they announced that blog if there was plans to make it easy for other projects to also have there own supported blog. http://lists.openstack.org/pipermail/openstack-dev/2013-October/017977.

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Joshua Harlow
At yahoo at least 50+ simultaneous will be the common case (maybe we are special). Think of what happens on www.yahoo.com say on the olympics, news.yahoo.com could need 50+ very very quickly (especially if say a gold medal is won by some famous person). So I wouldn't discount those being the commo

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Joshua Harlow
er entity. More details @ http://www.slideshare.net/harlowja/taskflow-27820295 From: Salvatore Orlando mailto:sorla...@nicira.com>> Date: Tuesday, November 19, 2013 2:22 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org&g

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Joshua Harlow
Sorry that was I prefer #3 (not #2) at the end there. Keyboard failure ;) On 11/19/13 10:27 AM, "Joshua Harlow" wrote: >Personally I would prefer #3 from the below. #2 I think will still have to >deal with consistency issues, just switching away from a DB doesn't mak

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Joshua Harlow
And also of course, nearly forgot a similar situation/review in heat. https://review.openstack.org/#/c/49440/ Except theres was/is dealing with stack locking (a heat concept). On 11/19/13 10:33 AM, "Joshua Harlow" wrote: >If you start adding these states you might really want to

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Joshua Harlow
If you start adding these states you might really want to consider the following work that is going on in other projects. It surely appears that everyone is starting to hit the same problem (and joining efforts would produce a more beneficial result). Relevant icehouse etherpads: - https://etherp

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Joshua Harlow
wrong IMHO). #1 could also work, but then u hit a vertical scaling limit (works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer #2 since I think it is honestly needed under all solutions. On 11/19/13 9:29 AM, "Chris Friesen" wrote: >On 11/18/2013 06:47 PM, Joshua H

Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration

2013-11-19 Thread Joshua Harlow
Sweet! Feel free to ask lots of questions and jump on both irc channels (#openstack-state-management and #openstack-cinder) if u need any help that can be better solved in real time chat. Thanks for helping getting this ball rolling :-) Sent from my really tiny device... > On Nov 19, 2013, at

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-18 Thread Joshua Harlow
From: Joe Gordon mailto:joe.gord...@gmail.com>> Date: Monday, November 18, 2013 5:33 PM To: Joshua Harlow mailto:harlo...@yahoo-inc.com>> Cc: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-18 Thread Joshua Harlow
An idea related to this, what would need to be done to make the DB have the exact state that a compute node is going through (and therefore the scheduler would not make unreliable/racey decisions, even when there are multiple schedulers). It's not like we are dealing with a system which can not

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-18 Thread Joshua Harlow
Same question here. I also wonder why a new service. Yes the capabilities might be different (but then can't this be fixed this is nova by making nova support different capabilities in different hypervisors better). The api's also might be different (but then can't this be fixed by making the api p

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-18 Thread Joshua Harlow
An idea, make the lock more granular. Instead of @utils.synchronized('any-name') I wonder if u could do something like. with utils.synchronized('any-name-$device-id'): # Code here Then at least u won't be locking at the method level (which means no concurrency). Would that work? From: Edgar M

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-11-18 Thread Joshua Harlow
I see it as important (and not theoretical). I'd like to use oslo.messaging (and oslo.messaging.rpc) in taskflow (to prototype how a distributed engine would work using it) and bringing in a library which requires global configuration is almost a non-starter for me. Although taskflow is targeted a

[openstack-dev] [State-Management] Proposal to add Anastasia Karpinska to taskflow-core

2013-11-18 Thread Joshua Harlow
Greetings all stackers, I propose that we add Anastasia Karpinska to the taskflow-core team [1]. Anastasia has been actively contributing to taskflow for a while now, both in code and reviews. She provides high quality reviews and is doing an awesome job with the various taskflow concepts (and is

Re: [openstack-dev] Split of the openstack-dev list (summary so far)

2013-11-15 Thread Joshua Harlow
Another thing that I remember from talking with people who work at yahoo on the hadoop project and was an insight early on for me. I remember those folks saying that about 2 hours of there day is spent on catching up on mailing list emails and reviews. This is/was a change in how they operated when

[openstack-dev] [State-Management] Meeting skip for this week

2013-11-14 Thread Joshua Harlow
Since I just got back from HK yesterday, and I think others are still recovering from that trip lets skip the weekly IRC meeting this week. Also note that the US time has changed (due to the evil thing called DST). Link: https://wiki.openstack.org/wiki/TaskFlow Channel: #openstack-state-manageme

Re: [openstack-dev] [ALL] Removing generate_uuid() from uuidutils

2013-11-13 Thread Joshua Harlow
I prefer the library to be named 'josh', kthxbye ;) On 11/13/13 7:26 PM, "Clint Byrum" wrote: >Excerpts from Michael Still's message of 2013-11-13 13:24:52 -0800: >> On Thu, Nov 14, 2013 at 8:20 AM, Eric Windisch >>wrote: >> >> > I don't think it is a problem to remove the code in oslo first,

[openstack-dev] [State-Management] Slides from taskflow speaker session

2013-11-06 Thread Joshua Harlow
Thanks all for showing up! http://www.slideshare.net/harlowja/taskflow-27820295 Not sure if there is an official page for this, anyway link is above (likely video link somewhere also). Questions, comments, feedback welcome! Thxs all for making this possible :-) -josh Sent from my really tiny

Re: [openstack-dev] [nova][scheduler]The database access in the scheduler filters

2013-11-01 Thread Joshua Harlow
I think there has been, and I think there will be a good design summit session for this. http://icehousedesignsummit.sched.org/event/cde73dadfd67eaae5bf98b90ba7de07 3#.UnPwKiRQ3mw I think what u have suggested could be a way to do it, as all databases do is set intersections and unions in the end

Re: [openstack-dev] [Solum] Some initial code copying for db/migration

2013-11-01 Thread Joshua Harlow
I think there is a summit topic about what to do about a good 'oslo.db' (not sure if it got scheduled?) I'd always recommend reconsidering just copying what nova/cinder and a few others have for there db structure. I don't think that has turned out so well in the long term (a 6000+ line file is n

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-31 Thread Joshua Harlow
wrote: > > >On 10/31/2013 01:32 PM, Joshua Harlow wrote: >> In the spirt of openness, yes I do think they are all needed. >> >> If they are not supported, then openstack is not open, it is a closed >> system. >> >> We should strive to innovate, not striv

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-31 Thread Joshua Harlow
wrote: >Sigh. > >Yay We've added more competing methods of complexity!!! > >Seriously. We now think that rabbit and zookeeper and mysql are ALL >needed? > >Joshua Harlow wrote: > >>I'm pretty sure the cats out of the bag. >> >>https://githu

[openstack-dev] [State-Management] Cancel IRC meeting for next 2 weeks

2013-10-31 Thread Joshua Harlow
Hi all, I just wanted to cancel the state-management IRC meeting for the next 2 weeks due to people traveling and the HK summit. Since it won't be as productive as usual I think we can just skip it until after the summit :-) Likely most people will be in #openstack-state-management anyway, fee

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-31 Thread Joshua Harlow
I'm pretty sure the cats out of the bag. https://github.com/openstack/requirements/blob/master/global-requirements.t xt#L29 https://kazoo.readthedocs.org/en/latest/ -Josh On 10/31/13 7:43 AM, "Monty Taylor" wrote: > > >On 10/30/2013 10:42 AM, Clint Byrum wrote: >> So, recently we've had quite

Re: [openstack-dev] [Solum] An early peek into the Solum.io website

2013-10-31 Thread Joshua Harlow
Just wondering, Will said infrastructure be made available to all other openstack projects? It'd be nice to have that for other projects to (a nova blog, a taskflow blog...). It seems/feels a little awkward to me to have solum be a special snowflake here. Sent from my really tiny device... >

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-30 Thread Joshua Harlow
Lock free solutions usually involve very predictable and very well understood (and typically provably correct) state machines. In openstack u have state machines but I think there is work to be done to increase there understandability and predictability. Lock free algorithms are also extremely

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-30 Thread Joshua Harlow
nt from my really tiny device... > On Oct 30, 2013, at 12:32 PM, "Jay Pipes" wrote: > > Has anyone looked into using concoord for distributed locking? > > https://pypi.python.org/pypi/concoord > > Best, > -jay > >> On 10/30/2013 02:39 PM, Joshua H

Re: [openstack-dev] When is it okay for submitters to say 'I don't want to add tests' ?

2013-10-30 Thread Joshua Harlow
+1 I think C always means something is not architected/designed correctly or invokes the 'code smell'[1] to me. If its to hard to test, then it will be to hard to change, which will in the long term be a liability. Better to avoid the liability upfront by just fixing the cause of the 'smell' in t

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-30 Thread Joshua Harlow
Yup, galera, thx! :) As for the: "It also doesn't handle the case where u can automatically recover from the current resource owner (nova-compute for example) dying." So heat is actively working on some resources, doing its thing, its binary crashes (or kill -9 occurs), what happens? The same qu

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-30 Thread Joshua Harlow
/13 1:02 PM, "Chris Friesen" wrote: >On 10/30/2013 01:34 PM, Joshua Harlow wrote: >> To me u just made state consistency be a lock by another name. >> >> A lock protects a region of code from being mutually accessed > >Personally I view a lock as protecting a se

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-30 Thread Joshua Harlow
+1 ;) On 10/30/13 12:37 PM, "Clint Byrum" wrote: >Excerpts from Joshua Harlow's message of 2013-10-30 11:57:30 -0700: >> As for the mutex and locking and all that problem. >> >> I would expect locking to be a necessity at some point for openstack. >> >> Even if the state transitions are the lo

Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

2013-10-30 Thread Joshua Harlow
To me u just made state consistency be a lock by another name. A lock protects a region of code from being mutually accessed, u can do the same with state consistency. In the end though u need some reliable transactional consistent storage (a database, zookeeper, other) to store that state (whi

<    5   6   7   8   9   10   11   12   >