Re: [openstack-dev] [nova] Expand resource name allowed characters

[Kenichi Oomichi]

Hi Chris,

Thanks for bring it up here,

> -Original Message-
> From: Chris St. Pierre [mailto:stpie...@metacloud.com]
> Sent: Saturday, September 13, 2014 2:53 AM
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] [nova] Expand resource name allowed characters
> 
> We have proposed that the allowed characters for all resource names in Nova 
> (flavors, aggregates, etc.) be expanded to
> all printable unicode characters and horizontal spaces: 
> https://review.openstack.org/#/c/119741
> 
> Currently, the only allowed characters in most resource names are 
> alphanumeric, space, and [.-_].
> 
> 
> We have proposed this change for two principal reasons:
> 
> 1. We have customers who have migrated data forward since Essex, when no 
> restrictions were in place, and thus have characters
> in resource names that are disallowed in the current version of OpenStack. 
> This is only likely to be useful to people
> migrating from Essex or earlier, since the current restrictions were added in 
> Folsom.
> 
> 2. It's pretty much always a bad idea to add unnecessary restrictions without 
> a good reason. While we don't have an immediate
> need to use, for example, the ever-useful http://codepoints.net/U+1F4A9 in a 
> flavor name, it's hard to come up with a
> reason people *shouldn't* be allowed to use it.
> 
> That said, apparently people have had a need to not be allowed to use some 
> characters, but it's not clear why:
> https://bugs.launchpad.net/nova/+bug/977187
> So I guess if anyone knows any reason why these printable characters should 
> not be joined in holy resource naming, speak
> now or forever hold your peace.

I also could not find the reason of current restriction on the bug report,
and I'd like to know it as the history.
On v2 API(not v2.1), each resource name contains the following restriction
for its name:

  Resource  | Length  | Pattern
 ---+-+--
  aggregate | 1-255   | nothing
  backup| nothing | nothing
  flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
| |   [a-zA-Z0-9. _-]*$'
  keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
  server| 1-255   | nothing
  cell  | 1-255   | don't contain "." and "!"

On v2.1 API, we have applied the same restriction rule[1] for whole resource
names for API consistency, so maybe we need to consider this topic for whole
names.

[1]: 
https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

Thanks
Ken'ichi Ohmichi

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Angus Lees]

On Fri, 12 Sep 2014 01:08:04 PM Doug Hellmann wrote:
> On Sep 12, 2014, at 1:03 PM, Ihar Hrachyshka  wrote:
> > Signed PGP part
> > 
> > On 12/09/14 17:30, Mike Bayer wrote:
> > > On Sep 12, 2014, at 10:40 AM, Ihar Hrachyshka 
> > > 
> > > wrote:
> > >> Signed PGP part On 12/09/14 16:33, Mike Bayer wrote:
> > >>> I agree with this, changing the MySQL driver now is not an
> > >>> option.
> > >> 
> > >> That was not the proposal. The proposal was to introduce support
> > >> to run against something different from MySQLdb + a gate job for
> > >> that alternative. The next cycle was supposed to do thorough
> > >> regression testing, benchmarking, etc. to decide whether we're ok
> > >> to recommend that alternative to users.
> > > 
> > > ah, well that is a great idea.  But we can have that throughout
> > > Kilo anyway, why not ?
> > 
> > Sure, it's not the end of the world. We'll just need to postpone work
> > till RC1 (=opening of master for new stuff), pass spec bureauracy
> > (reapplying for kilo)... That's some burden, but not tragedy.
> > 
> > The only thing that I'm really sad about is that Juno users won't be
> > able to try out that driver on their setup just to see how it works,
> > so it narrows testing base to gate while we could get some valuable
> > deployment feedback in Juno already.
> 
> It’s all experimental, right? And implemented in libraries? So those users
> could update oslo.db and sqlalchemy-migrate and test the results under
> Juno.

Note that it's also theoretically possible to run the migrate with mysqldb and 
the regular production service (post-migrate) with an alternate mysql driver 
that won't deadlock eventlet...  (ie: there's no reason the mysql driver 
choice needs to be universal and simultaneous)


I'm sad that we (as a project) still haven't been able to make this 
technically trivial fix - or even make it an option for testing - after the 
original problem was identified and the fix proposed 2.5+ months ago.  I'm 
encouraged to see various meta-threads popping up discussing issues with our 
development model and hopefully we can do better in future :(

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Sahara] Working on DOC bug

[Andrew Lazarev]

>I believe I also have to change the image here
http://docs.openstack.org/developer/sahara/overview.htm

It will be updated automatically once fix is merged.

Andrew.

On Fri, Sep 12, 2014 at 8:28 PM, Sharan Kumar M  wrote:

> Hi,
>
> For working on the mentioned DOC bug, I believe I also have to change the 
> image here http://docs.openstack.org/developer/sahara/overview.html. Is there 
> any particular tool which the community uses for images? And what about the 
> font that goes in the images?
>
> Also, thanks to Andrew for giving a gist of the bug and how to proceed.
>
> Thanks.
>
> > Hi Sharan,
> >
> > Sahara uses python clients to communicate with other components. So, you
> > can start with inspecting "requirements.txt" file on "python-*"
> > dependancies. Next, you need to understand what is really used and how.
> > Also, some components could be used indirectly via nova client (e.g.
> > security groups management).
> >
> > For this particular bug I see that Heat component is not described. Sahara
> > can provision cluster using Heat if it is configured to use heat
> > infrastructure engine. Also Sahara uses neutron for floating IPs and
> > security groups management if openstack cluster uses neutron. Probably
> > adding description of these two services is enough to close bug.
> >
> > Thanks,
> > Andrew.
> >
> > On Fri, Sep 12, 2014 at 12:13 PM, Sharan Kumar M <
> > sharan.monikantan at gmail.com 
> > <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>> wrote:
> >
> > >* Hi,
> *> >> >* After digging a little into OpenStack basics, setting up devstack and
> *> >* starting with sahara, I browsed the list of bugs on Launchpad. I saw a 
> low
> *> >* hanging fruit bug, which is a wishlist as well.
> *> >* https://bugs.launchpad.net/sahara/+bug/1350063 
> <https://bugs.launchpad.net/sahara/+bug/1350063>
> *> >>* > But before I start, I would like to get some advice on how to get 
> started
> *>* > with it, any resources for this bug etc. Just to let you all know, I am 
> new
> *>* > to OpenStack and this would probably be the first patch I submit.
> *> >>* > Thanks,
> *>* > Sharan Kumar M
> *> >>* > ___
> *>* > OpenStack-dev mailing list
> *>* > OpenStack-dev at lists.openstack.org 
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> *>* > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> *> >> >> -- next part --
> > An HTML attachment was scrubbed...
> > URL: 
> > <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140912/c4d0eb20/attachment.html>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Sahara] Working on DOC bug

[Sharan Kumar M]

Hi,

For working on the mentioned DOC bug, I believe I also have to change
the image here http://docs.openstack.org/developer/sahara/overview.html.
Is there any particular tool which the community uses for images? And
what about the font that goes in the images?

Also, thanks to Andrew for giving a gist of the bug and how to proceed.

Thanks.

> Hi Sharan,
>
> Sahara uses python clients to communicate with other components. So, you
> can start with inspecting "requirements.txt" file on "python-*"
> dependancies. Next, you need to understand what is really used and how.
> Also, some components could be used indirectly via nova client (e.g.
> security groups management).
>
> For this particular bug I see that Heat component is not described. Sahara
> can provision cluster using Heat if it is configured to use heat
> infrastructure engine. Also Sahara uses neutron for floating IPs and
> security groups management if openstack cluster uses neutron. Probably
> adding description of these two services is enough to close bug.
>
> Thanks,
> Andrew.
>
> On Fri, Sep 12, 2014 at 12:13 PM, Sharan Kumar M <
> sharan.monikantan at gmail.com 
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>> wrote:
>
> >* Hi,
*> >> >* After digging a little into OpenStack basics, setting up devstack and
*> >* starting with sahara, I browsed the list of bugs on Launchpad. I saw a low
*> >* hanging fruit bug, which is a wishlist as well.
*> >* https://bugs.launchpad.net/sahara/+bug/1350063
<https://bugs.launchpad.net/sahara/+bug/1350063>
*> >>* > But before I start, I would like to get some advice on how to
get started
*>* > with it, any resources for this bug etc. Just to let you all
know, I am new
*>* > to OpenStack and this would probably be the first patch I submit.
*> >>* > Thanks,
*>* > Sharan Kumar M
*> >>* > ___
*>* > OpenStack-dev mailing list
*>* > OpenStack-dev at lists.openstack.org
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
*>* > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
*> >> >> -- next part --
> An HTML attachment was scrubbed...
> URL: 
> <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140912/c4d0eb20/attachment.html>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Octavia] Which threading library?

[Joshua Harlow]

So my personal opinion is that u should design your API independent of either 
of these (asyncio, eventlet, threads...),

I think if u design the key parts of your API with the future[1] concept in 
mind then you can easily connect into the future (asyncio or other).

The asyncio layer has a future wrapper of its own (it actually has its own 
future related class[2], both of which are in the standard library to add to 
the confusion/fun...),

But if you target that kind of API (one that can return async objects, futures 
or other) then I think you will be tolerant of changes for asyncio or other...

With taskflow[3] I've tried to do this. It runs tasks/atoms (a taskflow 
concept) via futures and executors (one of its executors actually runs things 
remotely), which hopefully makes it easier to switch in a asyncio executor or 
other in the future, and it also has connections to make it transparent that 
eventlet is just another future executor[4]. But I guess it becomes a question 
of what platform (3.4+) do you want to support  and what design decisions you 
are ok with making that don't affect your current system to drastically...

[1] https://docs.python.org/dev/library/concurrent.futures.html#future-objects
[2] https://docs.python.org/3/library/asyncio-task.html#future
[3] http://docs.openstack.org/developer/taskflow/
[4] 
https://github.com/openstack/taskflow/blob/master/taskflow/utils/eventlet_utils.py#L98

My 2 cents,

Josh

On Sep 12, 2014, at 2:53 PM, Eichberger, German  
wrote:

> Hi,
>  
> I think the “standard” threading library for OpenStack is eventlet – however, 
> it seems that Oslo is spearheading efforts to get to a more compatible one 
> (seehttp://techs.enovance.com/6562/asyncio-openstack-python3) I am now 
> wondering since we are starting fresh if we should embrace (a potential) 
> future or stick with eventlet and all its flaws?
>  
> Thoughts?
>  
> German
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Sean Dague]

On 09/12/2014 01:14 PM, Doug Hellmann wrote:
> 
> On Sep 12, 2014, at 12:03 PM, Sean Dague  wrote:
> 
>> On 09/12/2014 11:52 AM, Doug Hellmann wrote:
>>>
>>> On Sep 12, 2014, at 11:21 AM, Mike Bayer  wrote:
>>>

 On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:

> I assume you, gentle OpenStack developers, often find yourself in a hair
> tearing out moment of frustration about why local unit tests are doing
> completely insane things. The code that it is stack tracing on is no
> where to be found, and yet it fails.
>
> And then you realize that part of oslo doesn't exist any more
> except there are still pyc files laying around. Gah!
>
> I've proposed the following to Nova and Python novaclient -
> https://review.openstack.org/#/c/121044/
>
> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.

 my VPN was down and I didn’t get this thread just now, but I am strongly 
 -1 on this as added to tox.ini, my response is 
 http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.

 Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
 *your* environment.  Don’t force it on our automated tests or on my 
 environment.   .pyc files make a difference in behavior, and if we banish 
 them from all testing, then our code is never tested within the 
 environment that it will normally be run in after shipment.

 I’d far prefer a simple script added to tox.ini which deletes orphaned 
 .pyc files only, if a change to tox.ini must be made.
>>>
>>> I have to agree with Mike here. Cleaning up our dev environments using a 
>>> little automation is better than disabling a feature of the interpreter 
>>> that may have unforeseen consequences in behavior or performance. The more 
>>> we introduce unusual settings like this into our environments and tools, 
>>> the more edge cases and weirdness we’re going to find in those tools that 
>>> keep us from doing the work we really want to be doing.
>>>
>>> We could use a git hook (see my earlier message in this thread) or we could 
>>> add a command to tox to remove them before starting the tests. Neither of 
>>> those solutions would affect the runtime behavior in a way that makes our 
>>> dev environments fundamentally different from a devstack or production 
>>> deployment.
>>
>> You believe that unit tests are going to change in the way they run so
>> dramatically with this change that it invalidates their use?
>>
>> Do we have examples of what changes if you do and don't have pyc files
>> there?
>>
>> Remember, we're not changing integration testing with this. This is
>> solely unit testing.
>>
>> The reason I don't like "just fix it in your local env" is you are then
>> exporting the complexity to developers. For something that they should
>> really not have to get bitten by... a lot.
> 
> Adding a command to tox to remove the files would be less intrusive than 
> disabling their creation.
> 
> We have had bad experiences mixing features to produce unusual dev 
> environments that are different from the way the software really runs. All of 
> the issues we had with namespace packages were caused by a bug in that 
> implementation exposed because we were doing something unusual in devstack, 
> for example.
> 
> Adding some variation of “find $(python setup.py --name) --name ‘*.pyc’ | 
> xargs rm -f” to tox.ini before running testr solves the problem you have 
> identified without introducing any side-effects.

Ok, while I'm not actually convinced that PYTHONDONTWRITEBYTECODE=true
is a problem (especially after looking at the actual source of the
python interpreter, where it's pretty clear everything is abstracted
through a set of AST classes regardless of whether these files are there
or not), I changed my upstream proposal to just the same purge line we'd
be using in Nova run_tests.sh forever before every tox run.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][cinder] Averting the Nova crisis by splitting out virt drivers

[Chris Friesen]

On 09/12/2014 04:59 PM, Joe Gordon wrote:



On Thu, Sep 11, 2014 at 2:18 AM, Daniel P. Berrange mailto:berra...@redhat.com>> wrote:



FYI, for Juno at least I really don't consider that even the libvirt
driver got acceptable review times in any sense. The pain of waiting
for reviews in libvirt code I've submitted this cycle is what prompted
me to start this thread. All the virt drivers are suffering way more
than they should be, but those without core team representation suffer


Can't you replace the word 'libvirt code' with 'nova code' and this
would still be true? Do you think landing virt driver code is harder
then landing non virt driver code? If so do you have any numbers to back
this up?

If the issue here is 'landing code in nova is too painful', then we
should discuss solving that more generalized issue first, and maybe we
conclude that pulling out the virt drivers gets us the most bang for our
buck.  But unless we have that more general discussion, saying the right
fix for that is to spend a large amount of time  working specifically on
virt driver related issues seems premature.


I agree that this is a nova issue in general, though I suspect that the 
virt drivers have quite separate developer communities so maybe they 
feel the pain more clearly.  But I think the solution is the same in 
both cases:


1) Allow people to be responsible for a subset of the nova code 
(scheduler, virt, conductor, compute, or even just a single driver). 
They would have significant responsibility for that area of the code. 
This would serve several purposes--people with deep domain-specific 
knowledge would be able to review code that touches that domain, and it 
would free up the nova core team to look at the higher-level picture. 
For changes that cross domains, the people from the relevant domains 
would need to be involved.


2) Modify the gate tests such that changes that are wholly contained 
within a single area of code are not blocked by gate-blocking-bugs in 
unrelated areas of the code.


Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Tempest Bug triage

[Masayuki Igawa]

Hi,

On Sat, Sep 13, 2014 at 1:18 AM, Mauro S M Rodrigues
 wrote:
> On 09/11/2014 04:52 PM, David Kranz wrote:
>>
>> So we had a Bug Day this week and the results were a bit disappointing due
>> to lack of participation. We went from 124 New bugs to 75. There were also
>> many cases where bugs referred to logs that no longer existed. This suggests
>> that we really need to keep up with bug triage in real time. Since bug
>> triage should involve the Core review team, we propose to rotate the
>> responsibility of triaging bugs weekly. I put up an etherpad here
>> https://etherpad.openstack.org/p/qa-bug-triage-rotation and I hope the
>> tempest core review team will sign up. Given our size, this should involve
>> signing up once every two months or so. I took next week.
>>
>>  -David
>
> +1, I'm not core team but I just assigned myself to the last week of
> September and first of December.
>
> Also, given the bad quality of some reports we may want to come up with a
> template of "need to have" data on the bug reports. I really haven't look at
> it lately but we use to have several reports with just a bunch of traces, or
> just a link..

+1

How about like this?[1]

Description of problem:

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1130926
-- 
Masayuki Igawa

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][cinder] Averting the Nova crisis by splitting out virt drivers

[Joe Gordon]

On Thu, Sep 11, 2014 at 2:18 AM, Daniel P. Berrange 
wrote:

> On Thu, Sep 11, 2014 at 09:23:34AM +1000, Michael Still wrote:
> > On Thu, Sep 11, 2014 at 8:11 AM, Jay Pipes  wrote:
> >
> > > a) Sorting out the common code is already accounted for in Dan B's
> original
> > > proposal -- it's a prerequisite for the split.
> >
> > Its a big prerequisite though. I think we're talking about a release
> > worth of work to get that right. I don't object to us doing that work,
> > but I think we need to be honest about how long its going to take. It
> > will also make the core of nova less agile, as we'll find it hard to
> > change the hypervisor driver interface over time. Do we really think
> > its ready to be stable?
>
> Yes, in my proposal I explicitly said we'd need to have Kilo
> for all the prep work to clean up the virt API, before only
> doing the split in Lx.
>
> The actual nova/virt/driver.py has been more stable over the
> past few releases than I thought it would be. In terms of APIs
> we're not really modified existing APIs, mostly added new ones.
> Where we did modify existing APIs, we could have easily taken
> the approach of adding a new API in parallel and deprecating
> the old entry point to maintain compat.
>
> The big change which isn't visible directly is the conversion
> of internal nova code to use objects. Finishing this conversion
> is clearly a pre-requisite to any such split, since we'd need
> to make sure all data passed into the nova virt APIs as parameters
> is stable & well defined.
>
> > As an alternative approach...
> >
> > What if we pushed most of the code for a driver into a library?
> > Imagine a library which controls the low level operations of a
> > hypervisor -- create a vm, attach a NIC, etc. Then the driver would
> > become a shim around that which was relatively thin, but owned the
> > interface into the nova core. The driver handles the nova specific
> > things like knowing how to create a config drive, or how to
> > orchestrate with cinder, but hands over all the hypervisor operations
> > to the library. If we found a bug in the library we just pin our
> > dependancy on the version we know works whilst we fix things.
> >
> > In fact, the driver inside nova could be a relatively generic "library
> > driver", and we could have multiple implementations of the library,
> > one for each hypervisor.
>
> I don't think that particularly solves the problem, particularly
> the ones you are most concerned about above of API stability. The
> naive impl of any "library" for the virt driver would pretty much
> mirror the nova virt API. The virt driver impls would thus have to
> do the job of taking the Nova objects passed in as parameters and
> turning them into something "stable" to pass to the library. Except
> now instead of us only having to figure out a stable API in one
> place, every single driver has to reinvent the wheel defining their
> own stable interface & objects. I'd also be concerned that ongoing
> work on drivers is still going to require alot of patches to Nova
> to update the shims all the time, so we're still going to contend
> on resource fairly highly.
>
> > > b) The conflict Dan is speaking of is around the current situation
> where we
> > > have a limited core review team bandwidth and we have to pick and
> choose
> > > which virt driver-specific features we will review. This leads to bad
> > > feelings and conflict.
> >
> > The way this worked in the past is we had cores who were subject
> > matter experts in various parts of the code -- there is a clear set of
> > cores who "get" xen or libivrt for example and I feel like those
> > drivers get reasonable review times. What's happened though is that
> > we've added a bunch of drivers without adding subject matter experts
> > to core to cover those drivers. Those newer drivers therefore have a
> > harder time getting things reviewed and approved.
>
> FYI, for Juno at least I really don't consider that even the libvirt
> driver got acceptable review times in any sense. The pain of waiting
> for reviews in libvirt code I've submitted this cycle is what prompted
> me to start this thread. All the virt drivers are suffering way more
> than they should be, but those without core team representation suffer
>

Can't you replace the word 'libvirt code' with 'nova code' and this would
still be true? Do you think landing virt driver code is harder then landing
non virt driver code? If so do you have any numbers to back this up?

If the issue here is 'landing code in nova is too painful', then we should
discuss solving that more generalized issue first, and maybe we conclude
that pulling out the virt drivers gets us the most bang for our buck.  But
unless we have that more general discussion, saying the right fix for that
is to spend a large amount of time  working specifically on virt driver
related issues seems premature.


> to an even greater degree.  And this is ignoring the point Jay & I
> were making about h

[openstack-dev] [Octavia] Which threading library?

[Eichberger, German]

Hi,

I think the "standard" threading library for OpenStack is eventlet - however, 
it seems that Oslo is spearheading efforts to get to a more compatible one (see 
http://techs.enovance.com/6562/asyncio-openstack-python3) I am now wondering 
since we are starting fresh if we should embrace (a potential) future or stick 
with eventlet and all its flaws?

Thoughts?

German
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] masking X-Auth-Token in debug output - proposed consistency

[Morgan Fainberg]

-Original Message-
From: Brant Knudson 
Reply: OpenStack Development Mailing List (not for usage questions) 
>
Date: September 12, 2014 at 14:32:20
To: OpenStack Development Mailing List (not for usage questions) 
>
Subject:  Re: [openstack-dev] masking X-Auth-Token in debug output - proposed 
consistency

> On Fri, Sep 12, 2014 at 12:02 PM, Tripp, Travis S  
> wrote:
>  
> >
> > From Jamie Lennox:
> > >> We handle this in the keystoneclient Session object by just printing
> > REDACTED or something similar.
> > >> The problem with using a SHA1 is that for backwards compatability we
> > often use the SHA1 of a PKI token
> > >> as if it were a UUID token and so this is still sensitive data. There
> > is working in keystone by morganfainberg
> > >> (which i think was merged) to add a new audit_it which will be able to
> > identify a token across calls without
> > >> exposing any sensitive information. We will support this in session
> > when available.
> >
> > From Sean Dague
> > > So the problem is that means we are currently leaking secrets and making
> > the logs unreadable.
> >
> > > It seems like we should move forward with the {SHA1} ... and if that is
> > still sensitive, address that later.
> > > Not addressing it basically keeps the exposure and destroys usability of
> > the code because there is so much garbage printed out.
> >
> > I understand Sean's point about debugging. Right now the glanceclient is
> > just printing ***. So it isn't printing a lot of excess and isn't leaking
> > anything sensitive. The other usability concern with the *** that Sean
> > previously mentioned was having a short usable string might be useful for
> > debugging.
> >
> > Morgan and Jamie, You think switching to SHA1 in actually adds a potential
> > security vulnerability to glanceclient that doesn't exist now. If that is
> > true, I think it would override the additional debugging concern of using
> > SHA1 for now. Can you please confirm?
> >
> > If only for consistency sake, I could switch to "TOKEN_REDACTED" like the
> > code sample Morgan sent. [1]
> >
> > [1]
> > https://github.com/openstack/python-keystoneclient/blob/01cabf6bbbee8b5340295f3be5e1fa7111387e7d/keystoneclient/session.py#L126-L131
> >   
> >
>  
> As the person who proposed the change to print TOKEN_REDACTED, I'd be happy
> to see it printed as {SHA1} instead. I only had it print
> TOKEN_REDACTED because I was concerned that we were still logging tokens
> and wanted to get something merged that didn't do that rather than waiting
> for the perfect solution to come along.
>  
> Since we have configurable token hashing algorithm support in keystone and
> auth_token middleware, it's possible that someone could lose their sanity
> and decide to use sha1 as the hash algorithm (it defaults to MD5, which
> some security standards say is inadequate), and now your logs have usable
> token IDs instead of an unusable hash, ***, TOKEN_REDACTED, or whatever. We
> could accept this as a risk, and we could mitigate the risk some by
> changing keystone to reject sha1 as a hashing algorithm.
>  
> - Brant

Ideally, I want to see the use of the audit_ids (in each token as of Juno) as 
the end goal. If we can get there as fast as changing to the {SHA1}, I’d 
advocate for that. Brant nicely outlined why we didn’t go with SHA1 earlier on 
in the cycle.

I think we are close to solving this in a better way than using sha1, but if we 
need a stop-gap we can go towards that for the short term (and disallow sha1 as 
a hash for Keystone).

—Morgan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] masking X-Auth-Token in debug output - proposed consistency

[Brant Knudson]

On Fri, Sep 12, 2014 at 12:02 PM, Tripp, Travis S 
wrote:

>
> From Jamie Lennox:
> >> We handle this in the keystoneclient Session object by just printing
> REDACTED or something similar.
> >> The problem with using a SHA1 is that for backwards compatability we
> often use the SHA1 of a PKI token
> >> as if it were a UUID token and so this is still sensitive data. There
> is working in keystone by morganfainberg
> >> (which i think was merged) to add a new audit_it which will be able to
> identify a token across calls without
> >> exposing any sensitive information. We will support this in session
> when available.
>
> From Sean Dague
> > So the problem is that means we are currently leaking secrets and making
> the logs unreadable.
>
> > It seems like we should move forward with the {SHA1} ... and if that is
> still sensitive, address that later.
> > Not addressing it basically keeps the exposure and destroys usability of
> the code because there is so much garbage printed out.
>
> I understand Sean's point about debugging.  Right now the glanceclient is
> just printing ***.  So it isn't printing a lot of excess and isn't leaking
> anything sensitive.  The other usability concern with the ***  that Sean
> previously mentioned was having a short usable string might be useful for
> debugging.
>
> Morgan and Jamie, You think switching to SHA1 in actually adds a potential
> security vulnerability to glanceclient that doesn't exist now. If that is
> true, I think it would override the additional debugging concern of using
> SHA1 for now.  Can you please confirm?
>
> If only for consistency sake, I could switch to "TOKEN_REDACTED" like the
> code sample Morgan sent. [1]
>
> [1]
> https://github.com/openstack/python-keystoneclient/blob/01cabf6bbbee8b5340295f3be5e1fa7111387e7d/keystoneclient/session.py#L126-L131
>

As the person who proposed the change to print TOKEN_REDACTED, I'd be happy
to see it printed as {SHA1} instead. I only had it print
TOKEN_REDACTED because I was concerned that we were still logging tokens
and wanted to get something merged that didn't do that rather than waiting
for the perfect solution to come along.

Since we have configurable token hashing algorithm support in keystone and
auth_token middleware, it's possible that someone could lose their sanity
and decide to use sha1 as the hash algorithm (it defaults to MD5, which
some security standards say is inadequate), and now your logs have usable
token IDs instead of an unusable hash, ***, TOKEN_REDACTED, or whatever. We
could accept this as a risk, and we could mitigate the risk some by
changing keystone to reject sha1 as a hashing algorithm.

- Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] Request for J3 FFE - NetApp: storage pools for scheduler

[Mike Perez]

On 14:24 Fri 05 Sep , Alex Meade wrote:
> Hi Cinder Folks,
> 
> I would like to request a FFE for cinder pools support with the NetApp
> drivers[1][2].

Looks like this is being reviewed now.

-- 
Mike Perez

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Dolph Mathews]

On Sep 12, 2014 10:05 AM, "Russell Bryant"  wrote:
>
> On 09/12/2014 07:37 AM, Thierry Carrez wrote:
> > If you think this is wrong and think the "design summit suggestion"
> > website is a better way to do it, let me know why! If some programs
> > really can't stand the 'etherpad/IRC' approach I'll see how we can spin
> > up a limited instance.
>
> I think this is fine, especially if it's a better reflection of reality
> and lets the teams work more efficiently.

+1 this is the direction that Keystone's summit session planning was
naturally headed. The session submission website was headed in a direction
where it felt like more of a formality that didn't quite serve its intended
purpose (aggregating and organizing discussion ideas), and didn't
particularly benefit the community as a result, than it felt like a useful
tool.

The scheduling piece of it (which mapped topics to timeslots and pushed to
sched.org), which I presume most non-PTLs ever saw, generally worked great.

> However, one of the benefits of the old submission system was the
> clarity of the process and openness to submissions from anyone.  We
> don't want to be in a situation where non-core folks feel like they have
> a harder time submitting a session.

I really hope that a less structured process will make it easier for
everyone to contribute and advocate for discussion ideas in a collaborative
manner. The design summit isn't about "my session topic" vs "your session
topic," it's about collaborating with our peers, regardless of whether that
peer group is 2 people or 100. A discussion shouldn't be "rejected" because
the group of interested parties is too small, it should simply find its
place.

>
> Once this is settled, as long as the wiki pages [1][2] reflect the
> process and is publicized, it should be fine.
>
> [1] https://wiki.openstack.org/wiki/Summit
> [2] https://wiki.openstack.org/wiki/Summit/Planning
>
> --
> Russell Bryant
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Sahara] Working on DOC bug

[Andrew Lazarev]

Hi Sharan,

Sahara uses python clients to communicate with other components. So, you
can start with inspecting "requirements.txt" file on "python-*"
dependancies. Next, you need to understand what is really used and how.
Also, some components could be used indirectly via nova client (e.g.
security groups management).

For this particular bug I see that Heat component is not described. Sahara
can provision cluster using Heat if it is configured to use heat
infrastructure engine. Also Sahara uses neutron for floating IPs and
security groups management if openstack cluster uses neutron. Probably
adding description of these two services is enough to close bug.

Thanks,
Andrew.

On Fri, Sep 12, 2014 at 12:13 PM, Sharan Kumar M <
sharan.monikan...@gmail.com> wrote:

> Hi,
>
> After digging a little into OpenStack basics, setting up devstack and
> starting with sahara, I browsed the list of bugs on Launchpad. I saw a low
> hanging fruit bug, which is a wishlist as well.
> https://bugs.launchpad.net/sahara/+bug/1350063
>
> But before I start, I would like to get some advice on how to get started
> with it, any resources for this bug etc. Just to let you all know, I am new
> to OpenStack and this would probably be the first patch I submit.
>
> Thanks,
> Sharan Kumar M
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

[Tim Hinrichs]

+1

Tim

On Sep 12, 2014, at 11:47 AM, Sean Roberts  wrote:

> +1
> 
> ~sean
> 
>> On Sep 12, 2014, at 11:28 AM, Peter Balland  wrote:
>> 
>> Hello,
>> 
>> I would like to nominate Aaron Rosen for the congress-core team.
>> 
>> Aaron has been involved in congress for the last few months providing
>> valuable reviews as well as leading the keystone integration and
>> congressclient work.
>> 
>> References:
>> 
>> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress,
>> n,z
>> 
>> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
>> ngressclient,n,z
>> 
>> https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
>> ss,n,z
>> 
>> Please respond with +1's or any concerns.
>> 
>> Thanks,
>> 
>> Peter
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Sahara] Working on DOC bug

[Sharan Kumar M]

Hi,

After digging a little into OpenStack basics, setting up devstack and
starting with sahara, I browsed the list of bugs on Launchpad. I saw a low
hanging fruit bug, which is a wishlist as well.
https://bugs.launchpad.net/sahara/+bug/1350063

But before I start, I would like to get some advice on how to get started
with it, any resources for this bug etc. Just to let you all know, I am new
to OpenStack and this would probably be the first patch I submit.

Thanks,
Sharan Kumar M
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] tempest test error

[Nikesh Kumar Mahalka]

Hi,
I deployed a juno devstack on ubuntu 14.04 server.

Below are the services,which are running fine
$ cinder service-list
+--+-+--+-+---++-+
|  Binary  | Host| Zone |
Status | State | Updated_at | Disabled Reason |
+--+-+--+-+---++-+
| cinder-scheduler | juno-devstack-server| nova |
enabled |   up  | 2014-09-12T19:07:00.00 |   None  |
|  cinder-volume   | juno-devstack-server@dothill_driver | nova |
enabled |   up  | 2014-09-12T19:06:55.00 |   None  |
+--+-+--+-+---++-+


nova service-list
++--+--+--+-+---++-+
| Id | Binary   | Host | Zone | Status  |
State | Updated_at | Disabled Reason |
++--+--+--+-+---++-+
| 1  | nova-conductor   | juno-devstack-server | internal | enabled |
up| 2014-09-12T19:08:06.00 | -   |
| 2  | nova-cert| juno-devstack-server | internal | enabled |
up| 2014-09-12T19:08:08.00 | -   |
| 3  | nova-network | juno-devstack-server | internal | enabled |
up| 2014-09-12T19:08:08.00 | -   |
| 4  | nova-compute | juno-devstack-server | nova | enabled |
up| 2014-09-12T19:07:59.00 | -   |
| 5  | nova-scheduler   | juno-devstack-server | internal | enabled |
up| 2014-09-12T19:08:05.00 | -   |
| 6  | nova-consoleauth | juno-devstack-server | internal | enabled |
up| 2014-09-12T19:08:00.00 | -   |
++--+--+--+-+---++-+

But when I am running tempest test and i am getting below error:

Ran 1744 tests in 3133.358s

FAILED (failures=105)
vedams@juno-devstack-server:~/devstack$
tempest.api.compute.volumes.test_volumes_get.VolumesGetTestXML
tempest.api.compute.volumes.test_volumes_get.VolumesGetTestXML:
command not found
vedams@juno-devstack-server:~/devstack$
test_volume_create_get_delete[gate,smoke] FAIL
test_volume_create_get_delete[gate,smoke]: command not found
vedams@juno-devstack-server:~/devstack$ setUpClass
(tempest.api.compute.volumes.test_volumes_list
-bash: syntax error near unexpected token
`tempest.api.compute.volumes.test_volumes_list'
vedams@juno-devstack-server:~/devstack$ VolumesTestXML)
   FAIL
-bash: syntax error near unexpected token `)'



Regards
Nikesh

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

[Sean Roberts]

+1

~sean

> On Sep 12, 2014, at 11:28 AM, Peter Balland  wrote:
> 
> Hello,
> 
> I would like to nominate Aaron Rosen for the congress-core team.
> 
> Aaron has been involved in congress for the last few months providing
> valuable reviews as well as leading the keystone integration and
> congressclient work.
> 
> References:
> 
> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress,
> n,z
> 
> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
> ngressclient,n,z
> 
> https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
> ss,n,z
> 
> Please respond with +1's or any concerns.
> 
> Thanks,
> 
> Peter
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [zaqar] Juno Performance Testing (Round 2)

[Joe Gordon]

On Tue, Sep 9, 2014 at 12:19 PM, Kurt Griffiths <
kurt.griffi...@rackspace.com> wrote:

> Hi folks,
>
> In this second round of performance testing, I benchmarked the new Redis
> driver. I used the same setup and tests as in Round 1 to make it easier to
> compare the two drivers. I did not test Redis in master-slave mode, but
> that likely would not make a significant difference in the results since
> Redis replication is asynchronous[1].
>
> As always, the usual benchmarking disclaimers apply (i.e., take these
> numbers with a grain of salt; they are only intended to provide a ballpark
> reference; you should perform your own tests, simulating your specific
> scenarios and using your own hardware; etc.).
>
> ## Setup ##
>
> Rather than VMs, I provisioned some Rackspace OnMetal[3] servers to
> mitigate noisy neighbor when running the performance tests:
>
> * 1x Load Generator
> * Hardware
> * 1x Intel Xeon E5-2680 v2 2.8Ghz
> * 32 GB RAM
> * 10Gbps NIC
> * 32GB SATADOM
> * Software
> * Debian Wheezy
> * Python 2.7.3
> * zaqar-bench
> * 1x Web Head
> * Hardware
> * 1x Intel Xeon E5-2680 v2 2.8Ghz
> * 32 GB RAM
> * 10Gbps NIC
> * 32GB SATADOM
> * Software
> * Debian Wheezy
> * Python 2.7.3
> * zaqar server
> * storage=mongodb
> * partitions=4
> * MongoDB URI configured with w=majority
> * uWSGI + gevent
> * config: http://paste.openstack.org/show/100592/
> * app.py: http://paste.openstack.org/show/100593/
> * 3x MongoDB Nodes
> * Hardware
> * 2x Intel Xeon E5-2680 v2 2.8Ghz
> * 128 GB RAM
> * 10Gbps NIC
> * 2x LSI Nytro WarpDrive BLP4-1600[2]
> * Software
> * Debian Wheezy
> * mongod 2.6.4
> * Default config, except setting replSet and enabling periodic
>   logging of CPU and I/O
> * Journaling enabled
> * Profiling on message DBs enabled for requests over 10ms
> * 1x Redis Node
> * Hardware
> * 2x Intel Xeon E5-2680 v2 2.8Ghz
> * 128 GB RAM
> * 10Gbps NIC
> * 2x LSI Nytro WarpDrive BLP4-1600[2]
> * Software
> * Debian Wheezy
> * Redis 2.4.14
> * Default config (snapshotting and AOF enabled)
> * One process
>
> As in Round 1, Keystone auth is disabled and requests go over HTTP, not
> HTTPS. The latency introduced by enabling these is outside the control of
> Zaqar, but should be quite minimal (speaking anecdotally, I would expect
> an additional 1-3ms for cached tokens and assuming an optimized TLS
> termination setup).
>
> For generating the load, I again used the zaqar-bench tool. I would like
> to see the team complete a large-scale Tsung test as well (including a
> full HA deployment with Keystone and HTTPS enabled), but decided not to
> wait for that before publishing the results for the Redis driver using
> zaqar-bench.
>
> CPU usage on the Redis node peaked at around 75% for the one process. To
> better utilize the hardware, a production deployment would need to run
> multiple Redis processes and use Zaqar's backend pooling feature to
> distribute queues across the various instances.
>
> Several different messaging patterns were tested, taking inspiration
> from: https://wiki.openstack.org/wiki/Use_Cases_(Zaqar)
>
> Each test was executed three times and the best time recorded.
>
> A ~1K sample message (1398 bytes) was used for all tests.
>
> ## Results ##
>
> ### Event Broadcasting (Read-Heavy) ###
>
> OK, so let's say you have a somewhat low-volume source, but tons of event
> observers. In this case, the observers easily outpace the producer, making
> this a read-heavy workload.
>
> Options
> * 1 producer process with 5 gevent workers
> * 1 message posted per request
> * 2 observer processes with 25 gevent workers each
> * 5 messages listed per request by the observers
> * Load distributed across 4[6] queues
> * 10-second duration
>

10 seconds is way too short


>
> Results
> * Redis
> * Producer: 1.7 ms/req,  585 req/sec
> * Observer: 1.5 ms/req, 1254 req/sec
> * Mongo
> * Producer: 2.2 ms/req,  454 req/sec
> * Observer: 1.5 ms/req, 1224 req/sec


If zaqar is like amazon SQS, then the latency for a single message and the
throughput for a single tenant is not important. I wouldn't expect anyone
who has latency sensitive work loads or needs massive throughput to use
zaqar, as these people wouldn't use SQS either. The consistency of the
latency (shouldn't change under load) and zaqar's ability to scale
horizontally mater much more. What I would be great to see some other
things benchmarked instead:

* graph latency versus number of concurrent active tenants
* graph latency versus message size
* How throughput scales as you scale up the number of assorted za

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Jay Pipes]

Had to laugh about the PILE OF POO character :) Comments inline...

On 09/12/2014 01:52 PM, Chris St. Pierre wrote:

We have proposed that the allowed characters for all resource names in
Nova (flavors, aggregates, etc.) be expanded to all printable unicode
characters and horizontal spaces: https://review.openstack.org/#/c/119741

Currently, the only allowed characters in most resource names are
alphanumeric, space, and [.-_].

We have proposed this change for two principal reasons:

1. We have customers who have migrated data forward since Essex, when no
restrictions were in place, and thus have characters in resource names
that are disallowed in the current version of OpenStack. This is only
likely to be useful to people migrating from Essex or earlier, since the
current restrictions were added in Folsom.


As this will affect the public REST APIs, the change should have a 
blueprint spec at the very least written for it. I don't remember why 
precisely the restrictions were put in place to begin with, but I'd 
imagine it probably had to do with early database schemas that may not 
have supported UTF-8 on the name columns by default.


Unfortunately, to my dismay, we "addressed" this in a number of projects 
(like Nova) by just changing ALL string columns to UTF-8, which, as I've 
stated a few times on the ML and in meetings, blows up index and 
temporary table sizes dramatically in MySQL. We should really be using 
UTF-8 on demand for columns like names. Having UTF-8 character set and 
collations on string fields for, say, UUID fields, is a giant waste of 
space.


Anyway, all this to say, "yeah, there really shouldn't be restrictions 
like this on the name columns, but we need to be careful about changes 
to public APIs and it would be good to have the API microversioning in 
place before we accept such a change."



2. It's pretty much always a bad idea to add unnecessary restrictions
without a good reason. While we don't have an immediate need to use, for
example, the ever-useful http://codepoints.net/U+1F4A9 in a flavor name,
it's hard to come up with a reason people *shouldn't* be allowed to use it.


LOL. Love it. Somebody at a public cloud should trademark the first 
public PILE OF POO flavor name.


Best,
-jay


That said, apparently people have had a need to not be allowed to use
some characters, but it's not clear why:
https://bugs.launchpad.net/nova/+bug/977187

So I guess if anyone knows any reason why these printable characters
should not be joined in holy resource naming, speak now or forever hold
your peace.

Thanks!

--
Chris St. Pierre
Senior Software Engineer
metacloud.com 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-dev][Cinder] FFE request for adding Huawei SDSHypervisor driver and connector

[Mike Perez]

On 17:38 Fri 12 Sep , Thierry Carrez wrote:
> Zhangni wrote:
> > I'd like to request an Juno feature freeze exception for this blueprint
> > and Spec:
> > 
> > https://blueprints.launchpad.net/cinder/+spec/huawei-sdshypervisor-driver
> 
> I would say it's way too late at this point for a new driver in Juno. At
> this point we should be focused on fixing what we already have, not add
> more surface.
> 
> Cheers,
> 
> -- 
> Thierry Carrez (ttx)

+1

Zhangni, you are however on track for the window to have your driver in for
K-1 [1]. Please make sure to communicate with Duncan Thomas about your
third-party CI [2].

[1] - 
http://lists.openstack.org/pipermail/openstack-dev/2014-September/044990.html
[2] - 
https://wiki.openstack.org/wiki/Cinder/how-to-contribute-a-driver#Third_Party_CI_Requirement_Policy

-- 
Mike Perez

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

[Peter Balland]

Hello,

I would like to nominate Aaron Rosen for the congress-core team.

Aaron has been involved in congress for the last few months providing
valuable reviews as well as leading the keystone integration and
congressclient work.

References:

https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress,
n,z

https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
ngressclient,n,z

https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
ss,n,z

Please respond with +1's or any concerns.

Thanks,

Peter


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Octavia] Responsibilities for controller drivers

[Brandon Logan]

IN IRC the topic came up about supporting many-to-many load balancers to
amphorae.  I believe a consensus was made that allowing only one-to-many
load balancers to amphorae would be the first step forward, and
re-evaluate later, since colocation and apolocation will need to work
(which brings up another topic, defining what it actually means to be
colocated: On the same amphorae, on the same amphorae host, on the same
cell/cluster, on the same data center/availability zone. That should be
something we discuss later, but not right now).

I am fine with that decisions, but Doug brought up a good point that
this could very well just be a decision for the controller driver and
Octavia shouldn't mandate this for all drivers.  So I think we need to
clearly define what decisions are the responsibility of the controller
driver versus what decisions are mandated by Octavia's construct.

Items I can come up with off the top of my head:

1) LB:Amphora - M:N vs 1:N
2) VIPs:LB - M:N vs 1:N
3) Pool:HMs - 1:N vs 1:1

I'm sure there are others.  I'm sure each one will need to be evaluated
on a case-by-case basis.  We will be walking a fine line between
flexibility and complexity.  We just need to define how far over that
line and in which direction we are willing to go.

Thanks,
Brandon
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Anita Kuno]

On 09/12/2014 11:54 AM, Thierry Carrez wrote:
> Anita Kuno wrote:
>> My question involves third party discussions. Now I know at least
>> Neutron is going to have a chat about drivers which involves third party
>> ci accounts as a supportive aspect of that discussion, but I am
>> wondering about the framework for a discussion which I can recommend
>> attendees of the third party meetings to attend. They are shaping up to
>> be an attentive, forward thinking group and are supporting each other
>> which I was hoping for from the beginning so I am very heartened by our
>> progress. I am feeling that as a group folks have questions and concerns
>> they would appreciate the opportunity to air in a mutually constructive
>> venue.
>>
>> What day and where would be the mutually constructive venue?
>>
>> I held off on Joe's thread since third party ci affects 4 or 5 programs,
>> not enough to qualify in my mind as a topic that is OpenStack wide, but
>> the programs it affects are quite affected, so I do feel it is time to
>> mention it.
> 
> I think those discussions could happen in a cross-project workshop.
> We'll run 2 or 3 of those in parallel all day Tuesday, so there is
> definitely room there.
> 
Thank you, I will co-ordinate with the group on an etherpad and start to
prioritize items that we want to discuss.

Thanks Thierry,
Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 12:29 PM, Daniel P. Berrange  wrote:

> On Fri, Sep 12, 2014 at 04:23:09PM +, Jeremy Stanley wrote:
>> On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote:
>> [...]
>>> Agreed, the problem with stale .pyc files is that it never occurs to
>>> developers that .pyc files are causing the problem until after you've
>>> wasted (potentially hours of) time debugging the problem. Avoiding
>>> this pain for all developers out of the box is a clear win overall
>>> and makes openstack development less painful.
>> 
>> I've been bitten by similar issues often enough that I regularly git
>> clean -dfx my checkouts or at least pass -r to tox so that it will
>> recreate its virtualenvs from scratch. Yes it does add some extra
>> time to the next test run, but you can iterate fairly tightly after
>> that as long as you're not actively moving stuff around while you
>> troubleshoot (and coupled with a git hook like Doug described for
>> cleaning on topic branch changes would be a huge boon as well).
> 
> I'm not debating whether there are ways to clean up your env to avoid
> the problem /after/ it occurs. The point is to stop the problem occuring
> in the first place to avoid placing this uneccessary clean up burden
> on devs.  Intentionally leaving things setup so that contributors hit
> bugs like stale .pyc files is just user hostile.

if we’re going to start diluting the test environment to suit developer 
environments, then the CI builds should use a different tox target that does 
*not* specify this environment variable.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 12:13 PM, Jeremy Stanley  wrote:

> On 2014-09-12 11:36:20 -0400 (-0400), Mike Bayer wrote:
> [...]
>> not to mention PYTHONHASHSEED only works on Python 3.  What is the
>> issue in tox you’re referring to ?
> 
> Huh? The overrides we added to numerous projects' tox.ini files to
> stem the breakage in Python 2.x unit tests from hash seed
> randomization in newer tox releases would seem to contradict your
> assertion. Also documentation...
> 
> https://docs.python.org/2.7/using/cmdline.html#envvar-PYTHONHASHSEED
> 
> (New in version 2.6.8.)

Python 3’s documentation says “new in version 3.2.3”, so, confusing that they 
backported it to 2.6 at the same time but google searches tend to point you 
right here:

https://docs.python.org/3.3/using/cmdline.html#envvar-PYTHONHASHSEED



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Expand resource name allowed characters

[Chris St. Pierre]

We have proposed that the allowed characters for all resource names in Nova
(flavors, aggregates, etc.) be expanded to all printable unicode characters
and horizontal spaces: https://review.openstack.org/#/c/119741

Currently, the only allowed characters in most resource names are
alphanumeric, space, and [.-_].

We have proposed this change for two principal reasons:

1. We have customers who have migrated data forward since Essex, when no
restrictions were in place, and thus have characters in resource names that
are disallowed in the current version of OpenStack. This is only likely to
be useful to people migrating from Essex or earlier, since the current
restrictions were added in Folsom.

2. It's pretty much always a bad idea to add unnecessary restrictions
without a good reason. While we don't have an immediate need to use, for
example, the ever-useful http://codepoints.net/U+1F4A9 in a flavor name,
it's hard to come up with a reason people *shouldn't* be allowed to use it.

That said, apparently people have had a need to not be allowed to use some
characters, but it's not clear why:
https://bugs.launchpad.net/nova/+bug/977187

So I guess if anyone knows any reason why these printable characters should
not be joined in holy resource naming, speak now or forever hold your peace.

Thanks!

-- 
Chris St. Pierre
Senior Software Engineer
metacloud.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Clint Byrum]

Excerpts from Flavio Percoco's message of 2014-09-12 00:22:35 -0700:
> On 09/12/2014 03:29 AM, Clint Byrum wrote:
> > Excerpts from Zane Bitter's message of 2014-09-11 15:21:26 -0700:
> >> On 09/09/14 19:56, Clint Byrum wrote:
> >>> Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700:
>  On 9/9/14, 12:03 PM, Monty Taylor wrote:
> > On 09/04/2014 01:30 AM, Clint Byrum wrote:
> >> Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700:
> >>> Greetings,
> >>>
> >>> Last Tuesday the TC held the first graduation review for Zaqar. During
> >>> the meeting some concerns arose. I've listed those concerns below with
> >>> some comments hoping that it will help starting a discussion before 
> >>> the
> >>> next meeting. In addition, I've added some comments about the project
> >>> stability at the bottom and an etherpad link pointing to a list of use
> >>> cases for Zaqar.
> >>>
> >>
> >> Hi Flavio. This was an interesting read. As somebody whose attention 
> >> has
> >> recently been drawn to Zaqar, I am quite interested in seeing it
> >> graduate.
> >>
> >>> # Concerns
> >>>
> >>> - Concern on operational burden of requiring NoSQL deploy expertise to
> >>> the mix of openstack operational skills
> >>>
> >>> For those of you not familiar with Zaqar, it currently supports 2 
> >>> nosql
> >>> drivers - MongoDB and Redis - and those are the only 2 drivers it
> >>> supports for now. This will require operators willing to use Zaqar to
> >>> maintain a new (?) NoSQL technology in their system. Before expressing
> >>> our thoughts on this matter, let me say that:
> >>>
> >>>   1. By removing the SQLAlchemy driver, we basically removed the
> >>> chance
> >>> for operators to use an already deployed "OpenStack-technology"
> >>>   2. Zaqar won't be backed by any AMQP based messaging technology 
> >>> for
> >>> now. Here's[0] a summary of the research the team (mostly done by
> >>> Victoria) did during Juno
> >>>   3. We (OpenStack) used to require Redis for the zmq matchmaker
> >>>   4. We (OpenStack) also use memcached for caching and as the oslo
> >>> caching lib becomes available - or a wrapper on top of dogpile.cache -
> >>> Redis may be used in place of memcached in more and more deployments.
> >>>   5. Ceilometer's recommended storage driver is still MongoDB,
> >>> although
> >>> Ceilometer has now support for sqlalchemy. (Please correct me if I'm
> >>> wrong).
> >>>
> >>> That being said, it's obvious we already, to some extent, promote some
> >>> NoSQL technologies. However, for the sake of the discussion, lets 
> >>> assume
> >>> we don't.
> >>>
> >>> I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't
> >>> keep avoiding these technologies. NoSQL technologies have been around
> >>> for years and we should be prepared - including OpenStack operators - 
> >>> to
> >>> support these technologies. Not every tool is good for all tasks - one
> >>> of the reasons we removed the sqlalchemy driver in the first place -
> >>> therefore it's impossible to keep an homogeneous environment for all
> >>> services.
> >>>
> >>
> >> I whole heartedly agree that non traditional storage technologies that
> >> are becoming mainstream are good candidates for use cases where SQL
> >> based storage gets in the way. I wish there wasn't so much FUD
> >> (warranted or not) about MongoDB, but that is the reality we live in.
> >>
> >>> With this, I'm not suggesting to ignore the risks and the extra burden
> >>> this adds but, instead of attempting to avoid it completely by not
> >>> evolving the stack of services we provide, we should probably work on
> >>> defining a reasonable subset of NoSQL services we are OK with
> >>> supporting. This will help making the burden smaller and it'll give
> >>> operators the option to choose.
> >>>
> >>> [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/
> >>>
> >>>
> >>> - Concern on should we really reinvent a queue system rather than
> >>> piggyback on one
> >>>
> >>> As mentioned in the meeting on Tuesday, Zaqar is not reinventing 
> >>> message
> >>> brokers. Zaqar provides a service akin to SQS from AWS with an 
> >>> OpenStack
> >>> flavor on top. [0]
> >>>
> >>
> >> I think Zaqar is more like SMTP and IMAP than AMQP. You're not really
> >> trying to connect two processes in real time. You're trying to do fully
> >> asynchronous messaging with fully randomized access to any message.
> >>
> >> Perhaps somebody should explore whether the approaches taken by large
> >> scale IMAP providers could be applied to Zaqar.
> >>
> >> Anyway, I can't imagine writ

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Clint Byrum]

Excerpts from Mark McLoughlin's message of 2014-09-12 03:27:42 -0700:
> On Wed, 2014-09-10 at 14:51 +0200, Thierry Carrez wrote:
> > Flavio Percoco wrote:
> > > [...]
> > > Based on the feedback from the meeting[3], the current main concern is:
> > > 
> > > - Do we need a messaging service with a feature-set akin to SQS+SNS?
> > > [...]
> > 
> > I think we do need, as Samuel puts it, "some sort of durable
> > message-broker/queue-server thing". It's a basic application building
> > block. Some claim it's THE basic application building block, more useful
> > than database provisioning. It's definitely a layer above pure IaaS, so
> > if we end up splitting OpenStack into layers this clearly won't be in
> > the inner one. But I think "IaaS+" basic application building blocks
> > belong in OpenStack one way or another. That's the reason I supported
> > Designate ("everyone needs DNS") and Trove ("everyone needs DBs").
> > 
> > With that said, I think yesterday there was a concern that Zaqar might
> > not fill the "some sort of durable message-broker/queue-server thing"
> > role well. The argument goes something like: if it was a queue-server
> > then it should actually be built on top of Rabbit; if it was a
> > message-broker it should be built on top of postfix/dovecot; the current
> > architecture is only justified because it's something in between, so
> > it's broken.
> > 
> > I guess I don't mind that much zaqar being "something in between":
> > unless I misunderstood, exposing extra primitives doesn't prevent the
> > "queue-server" use case from being filled. Even considering the
> > message-broker case, I'm also not convinced building it on top of
> > postfix/dovecot would be a net win compared to building it on top of
> > Redis, to be honest.
> 
> AFAICT, this part of the debate boils down to the following argument:
> 
>   If Zaqar implemented messaging-as-a-service with only queuing 
>   semantics (and no random access semantics), it's design would 
>   naturally be dramatically different and "simply" implement a 
>   multi-tenant REST API in front of AMQP queues like this:
> 
> https://www.dropbox.com/s/yonloa9ytlf8fdh/ZaqarQueueOnly.png?dl=0
> 
>   and that this architecture would allow for dramatically improved 
>   throughput for end-users while not making the cost of providing the 
>   service prohibitive to operators.
> 
> You can't dismiss that argument out-of-hand, but I wonder (a) whether
> the claimed performance improvement is going to make a dramatic
> difference to the SQS-like use case and (b) whether backing this thing
> with an RDBMS and multiple highly available, durable AMQP broker
> clusters is going to be too much of a burden on operators for whatever
> performance improvements it does gain.

Having had experience taking queue-only data out of RDBMS's and even SMTP
solutions, and putting them into queues, I can say that it was generally
quite a bit more reliable and cheaper to maintain.

However, as I've been thinking about this more, I am concerned about the
complexity of trying to use a stateless protocol like HTTP for reliable
delivery, given that these queues all use a session model that relies
on connection persistence. That may very well invalidate my hypothesis.

> 
> But the troubling part of this debate is where we repeatedly batter the
> Zaqar team with hypotheses like these and appear to only barely
> entertain their carefully considered justification for their design
> decisions like:
> 
>   
> https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#Is_Zaqar_a_provisioning_service_or_a_data_API.3F
>   
> https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#What_messaging_patterns_does_Zaqar_support.3F
> 
> I would like to see an SQS-like API provided by OpenStack, I accept the
> reasons for Zaqar's design decisions to date, I respect that those
> decisions were made carefully by highly competent members of our
> community and I expect Zaqar to evolve (like all projects) in the years
> ahead based on more real-world feedback, new hypotheses or ideas, and
> lessons learned from trying things out.

I have read those and I truly believe that the Zaqar team, who I believe
are already a valuable part of the OpenStack family, are doing good work.
Seriously, I believe it is valuable as is and I trust them to do what
they have stated they will do.

Let me explain my position again. Heat is in dire need of a way to
communicate with instances that is efficient. It has no need for a full
messaging stack.. just a way for users to have things pushed from Heat
to their instances efficiently.

So, to reiterate why I keep going on about this: If a messaging service
is to become an integrated part of OpenStack's release, we should think
carefully about the ramifications for operators _and_ users of not
having a light weight queue-only option, when that seems to fit _most_
of the use cases.

___
OpenStack-dev 

Re: [openstack-dev] [solum] pep8 - splitting expressions

[Kevin L. Mitchell]

On Thu, 2014-09-11 at 18:35 -0500, Ed Leafe wrote:
> On Sep 11, 2014, at 5:05 PM, Kevin L. Mitchell  
> wrote:
> 
> > I'd suggest trying:
> > 
> >res = amodel.Assemblies(
> >uri=common.ASSEM_URI_STR % pecan.request.host_url,
> >name='Solum_CAMP_assemblies',
> >type='assemblies',
> >description=common.ASSEM_DESC_STR,
> >assembly_links=a_links,
> >parameter_definitions_uri=common.ASSEM_PARAM_STR %
> >pecan.request.host_url)
> > 
> > By moving the first argument to a line by itself, pep8 can be satisfied
> > by indenting the following lines by 4 spaces.
> 
> When not using visual indentation, the standard is to indent two
> levels (i.e., 8 spaces), to distinguish it from typical blocks. But
> yes, I much prefer this than creating temporary names to accommodate
> the visual indentation style.

Using an 8 space indent here would result in an E126 pep8 error, which
is why I use 4 spaces for this case.  Of course, I don't know if Solum
enforces E126.
-- 
Kevin L. Mitchell 
Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Doug Hellmann]

On Sep 12, 2014, at 12:48 PM, Chris Dent  wrote:

> On Fri, 12 Sep 2014, Doug Hellmann wrote:
> 
>> We could use a git hook (see my earlier message in this thread) or we
>> could add a command to tox to remove them before starting the tests.
>> Neither of those solutions would affect the runtime behavior in a way
>> that makes our dev environments fundamentally different from a
>> devstack or production deployment.
> 
> For reference, I've always been in the habit of managing automated
> tasks in code checkouts with a Makefile that has targets with
> depencies: e.g 'make test' will always do the 'clean' target and
> 'clean' does something like find . -name "*.pyc" ...
> 
> My stumble into openstack to find tox being the one _visible_ source
> of automation was quite a shock. Things like git hooks do not count
> as visible, despite being useful.
> 
> The idea being that the systems we work with as developers need to
> be both discoverable and introspectable. Or to be clear:
> transparent.
> 
> tox, in general, is vaguely magical, and it would be a shame to add
> some more. Yes, the pyc thing screwing with your stuff is bad and
> bit me hard a few times but once I knew what it was what I really
> wanted was a quick (and centrally approved) way to do 'make clean'.
> 
> I guess tox -eclean or something would be the same but it feels
> wrong somehow.

I wasn’t clear. I’m proposing that we add a call to “rm” before our command to 
execute the tests. The command to cleanly run the tests would still be “tox -e 
py27” or whatever you’re using now. The cleanup step would be transparent, just 
as the “clean” target is in your “make test” example.

Doug


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Doug Hellmann]

On Sep 12, 2014, at 12:03 PM, Sean Dague  wrote:

> On 09/12/2014 11:52 AM, Doug Hellmann wrote:
>> 
>> On Sep 12, 2014, at 11:21 AM, Mike Bayer  wrote:
>> 
>>> 
>>> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
>>> 
 I assume you, gentle OpenStack developers, often find yourself in a hair
 tearing out moment of frustration about why local unit tests are doing
 completely insane things. The code that it is stack tracing on is no
 where to be found, and yet it fails.
 
 And then you realize that part of oslo doesn't exist any more
 except there are still pyc files laying around. Gah!
 
 I've proposed the following to Nova and Python novaclient -
 https://review.openstack.org/#/c/121044/
 
 Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>>> 
>>> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
>>> on this as added to tox.ini, my response is 
>>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
>>> 
>>> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
>>> *your* environment.  Don’t force it on our automated tests or on my 
>>> environment.   .pyc files make a difference in behavior, and if we banish 
>>> them from all testing, then our code is never tested within the environment 
>>> that it will normally be run in after shipment.
>>> 
>>> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
>>> files only, if a change to tox.ini must be made.
>> 
>> I have to agree with Mike here. Cleaning up our dev environments using a 
>> little automation is better than disabling a feature of the interpreter that 
>> may have unforeseen consequences in behavior or performance. The more we 
>> introduce unusual settings like this into our environments and tools, the 
>> more edge cases and weirdness we’re going to find in those tools that keep 
>> us from doing the work we really want to be doing.
>> 
>> We could use a git hook (see my earlier message in this thread) or we could 
>> add a command to tox to remove them before starting the tests. Neither of 
>> those solutions would affect the runtime behavior in a way that makes our 
>> dev environments fundamentally different from a devstack or production 
>> deployment.
> 
> You believe that unit tests are going to change in the way they run so
> dramatically with this change that it invalidates their use?
> 
> Do we have examples of what changes if you do and don't have pyc files
> there?
> 
> Remember, we're not changing integration testing with this. This is
> solely unit testing.
> 
> The reason I don't like "just fix it in your local env" is you are then
> exporting the complexity to developers. For something that they should
> really not have to get bitten by... a lot.

Adding a command to tox to remove the files would be less intrusive than 
disabling their creation.

We have had bad experiences mixing features to produce unusual dev environments 
that are different from the way the software really runs. All of the issues we 
had with namespace packages were caused by a bug in that implementation exposed 
because we were doing something unusual in devstack, for example.

Adding some variation of “find $(python setup.py --name) --name ‘*.pyc’ | xargs 
rm -f” to tox.ini before running testr solves the problem you have identified 
without introducing any side-effects.

Doug

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Clint Byrum]

Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700:
> Clint Byrum wrote:
> > Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700:
> >> Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is
> >> to optimize Zaqar for delivering messages and supporting different
> >> messaging patterns.
> > 
> > Awesome! Just please don't expect people to get excited about it for
> > the lighter weight queueing workloads that you've claimed as use cases.
> > 
> > I totally see Horizon using it to keep events for users. I see Heat
> > using it for stack events as well. I would bet that Trove would benefit
> > from being able to communicate messages to users.
> > 
> > But I think in between Zaqar and the backends will likely be a lighter
> > weight queue-only service that the users can just subscribe to when they
> > don't want an inbox. And I think that lighter weight queue service is
> > far more important for OpenStack than the full blown random access
> > inbox.
> > 
> > I think the reason such a thing has not appeared is because we were all
> > sort of running into "but Zaqar is already incubated". Now that we've
> > fleshed out the difference, I think those of us that need a lightweight
> > multi-tenant queue service should add it to OpenStack.  Separately. I hope
> > that doesn't offend you and the rest of the excellent Zaqar developers. It
> > is just a different thing.
> > 
> >> Should we remove all the semantics that allow people to use Zaqar as a
> >> queue service? I don't think so either. Again, the semantics are there
> >> because Zaqar is using them to do its job. Whether other folks may/may
> >> not use Zaqar as a queue service is out of our control.
> >>
> >> This doesn't mean the project is broken.
> > 
> > No, definitely not broken. It just isn't actually necessary for many of
> > the stated use cases.
> 
> Clint,
> 
> If I read you correctly, you're basically saying the Zaqar is overkill
> for a lot of people who only want a multi-tenant queue service. It's
> doing A+B. Why does that prevent people who only need A from using it ?
> 
> Is it that it's actually not doing A well, from a user perspective ?
> Like the performance sucks, or it's missing a key primitive ?
> 
> Is it that it's unnecessarily complex to deploy, from a deployer
> perspective, and that something only doing A would be simpler, while
> covering most of the use cases?
> 
> Is it something else ?
> 
> I want to make sure I understand your objection. In the "user
> perspective" it might make sense to pursue both options as separate
> projects. In the "deployer perspective" case, having a project doing A+B
> and a project doing A doesn't solve anything. So this affects the
> decision we have to take next Tuesday...

I believe that Zaqar does two things, inbox semantics, and queue
semantics. I believe the queueing is a side-effect of needing some kind
of queue to enable users to store and subscribe to messages in the
inbox.

What I'd rather see is an API for queueing, and an API for inboxes
which integrates well with the queueing API. For instance, if a user
says "give me an inbox" I think Zaqar should return a queue handle for
sending into the inbox the same way Nova gives you a Neutron port if
you don't give it one. You might also ask for a queue to receive push
messages from the inbox. Point being, the queues are not the inbox,
and the inbox is not the queues.

However, if I just want a queue, just give me a queue. Don't store my
messages in a randomly addressable space, and don't saddle the deployer
with the burden of such storage. Put the queue API in front of a scalable
message queue and give me a nice simple HTTP API. Users would likely be
thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all
make use of just this. Only Horizon seems to need a place to keep the
messages around while users inspect them.

Whether that is two projects, or one, separation between the two API's,
and thus two very different types of backends, is something I think
will lead to more deployers wanting to deploy both, so that they can
bill usage appropriately and so that their users can choose wisely.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Doug Hellmann]

On Sep 12, 2014, at 1:03 PM, Ihar Hrachyshka  wrote:

> Signed PGP part
> On 12/09/14 17:30, Mike Bayer wrote:
> >
> > On Sep 12, 2014, at 10:40 AM, Ihar Hrachyshka 
> > wrote:
> >
> >> Signed PGP part On 12/09/14 16:33, Mike Bayer wrote:
> >>> I agree with this, changing the MySQL driver now is not an
> >>> option.
> >>
> >> That was not the proposal. The proposal was to introduce support
> >> to run against something different from MySQLdb + a gate job for
> >> that alternative. The next cycle was supposed to do thorough
> >> regression testing, benchmarking, etc. to decide whether we're ok
> >> to recommend that alternative to users.
> >
> > ah, well that is a great idea.  But we can have that throughout
> > Kilo anyway, why not ?
> 
> Sure, it's not the end of the world. We'll just need to postpone work
> till RC1 (=opening of master for new stuff), pass spec bureauracy
> (reapplying for kilo)... That's some burden, but not tragedy.
> 
> The only thing that I'm really sad about is that Juno users won't be
> able to try out that driver on their setup just to see how it works,
> so it narrows testing base to gate while we could get some valuable
> deployment feedback in Juno already.

It’s all experimental, right? And implemented in libraries? So those users 
could update oslo.db and sqlalchemy-migrate and test the results under Juno.

Doug


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] masking X-Auth-Token in debug output - proposed consistency

[Tripp, Travis S]

>From Jamie Lennox:
>> We handle this in the keystoneclient Session object by just printing 
>> REDACTED or something similar. 
>> The problem with using a SHA1 is that for backwards compatability we often 
>> use the SHA1 of a PKI token
>> as if it were a UUID token and so this is still sensitive data. There is 
>> working in keystone by morganfainberg
>> (which i think was merged) to add a new audit_it which will be able to 
>> identify a token across calls without
>> exposing any sensitive information. We will support this in session when 
>> available. 

>From Sean Dague
> So the problem is that means we are currently leaking secrets and making the 
> logs unreadable.

> It seems like we should move forward with the {SHA1} ... and if that is still 
> sensitive, address that later. 
> Not addressing it basically keeps the exposure and destroys usability of the 
> code because there is so much garbage printed out.

I understand Sean's point about debugging.  Right now the glanceclient is just 
printing ***.  So it isn't printing a lot of excess and isn't leaking anything 
sensitive.  The other usability concern with the ***  that Sean previously 
mentioned was having a short usable string might be useful for debugging.

Morgan and Jamie, You think switching to SHA1 in actually adds a potential 
security vulnerability to glanceclient that doesn't exist now. If that is true, 
I think it would override the additional debugging concern of using SHA1 for 
now.  Can you please confirm?  

If only for consistency sake, I could switch to "TOKEN_REDACTED" like the code 
sample Morgan sent. [1]

[1] 
https://github.com/openstack/python-keystoneclient/blob/01cabf6bbbee8b5340295f3be5e1fa7111387e7d/keystoneclient/session.py#L126-L131
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 12/09/14 17:30, Mike Bayer wrote:
> 
> On Sep 12, 2014, at 10:40 AM, Ihar Hrachyshka 
> wrote:
> 
>> Signed PGP part On 12/09/14 16:33, Mike Bayer wrote:
>>> I agree with this, changing the MySQL driver now is not an
>>> option.
>> 
>> That was not the proposal. The proposal was to introduce support
>> to run against something different from MySQLdb + a gate job for
>> that alternative. The next cycle was supposed to do thorough
>> regression testing, benchmarking, etc. to decide whether we're ok
>> to recommend that alternative to users.
> 
> ah, well that is a great idea.  But we can have that throughout
> Kilo anyway, why not ?

Sure, it's not the end of the world. We'll just need to postpone work
till RC1 (=opening of master for new stuff), pass spec bureauracy
(reapplying for kilo)... That's some burden, but not tragedy.

The only thing that I'm really sad about is that Juno users won't be
able to try out that driver on their setup just to see how it works,
so it narrows testing base to gate while we could get some valuable
deployment feedback in Juno already.

> 
> 
> 
> 
> 
> ___ OpenStack-dev
> mailing list OpenStack-dev@lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUEydwAAoJEC5aWaUY1u57CbYIAKCwyAj/+xyGlcFeUJ04Jtxi
1mwl3IjO6Ue5BfdrrO7128MHMINUojcA4VnQv3jNfwjJ1j1TqWQ+/6uoFHiGn7uA
ga1SVNGar1SkIbc8OqkdbEOd2tI36rvF9qA7dEP1pVJYuwT+iNRmPgiieDrsSpXu
40F3zZQLPfAFSqaANBDeh6sq2OxPF99IG15X49YqCjmI5+cwRCw331LCdZXAV/lq
yHrIZDYrfFSMSHoldAVtb4dJLu06rQNuDTwWMrdEXKmlkNv00EfK3V+Er0E/lq8E
7QKH05dGbcRj0/qaofiQlPvAn/UomIFDHv9zZdV4UEKWxKo1oRB6cKUAv7LhGS4=
=CnKn
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo] official recommendations to handle oslo-incubator sync requests

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

There seems to be no objections to that wording, so I went forward and
added it to [1], plus added the note about those rules to [2].

[1]: https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator
[2]: https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes

On 19/08/14 15:52, Ihar Hrachyshka wrote:
> Hi all,
> 
> I've found out there are no clear public instructions on how to
> handle oslo-incubator synchronizations in master and stable
> branches neither at [1] nor at [2]. Though my observations show
> that there is some oral tradition around community on how we handle
> those review, as follows:
> 
> 1. For master oslo-incubator sync requests, we tend to sync the
> whole modules or even all the modules that a project uses (unless
> some specific obstacles to do so). This is to use the latest and
> greatest code from Oslo subproject, fetch all possible bug fixes
> and goodies, and make the synchronized copy of it as similar to
> upstream (=oslo-incubator) as possible.
> 
> 2. For stable branches, the process is a bit different. For those 
> branches, we don't generally want to introduce changes that are
> not related to specific issues in a project. So in case of
> backports, we tend to do per-patch consideration when synchronizing
> from incubator.
> 
> 3. Backporting for stable branches is a bit complicated process.
> When reviewing synchronization requests for those branches, we
> should not only check that the code is present in all consequent
> branches of the appropriate project (f.e. for Havana, in both Juno
> and Icehouse), but also that the patches being synchronized were
> successfully backported to corresponding stable branches of
> oslo-incubator. So the usual way of oslo-incubator bug fix is (in
> case of e.g. Neutron):
> 
> oslo-incubator (master) -> neutron (master) -> oslo-incubator 
> (stable/icehouse) -> neutron (stable/icehouse).
> 
> [For Havana, it's even more complicated, introducing more elements
> in the backporting chain.]
> 
> I hope I've described the existing oral tradition correctly.
> Please comment on that, and if we're ok with the way it's written
> above, I'd like to update our wiki pages ([1] and [2]) with that.
> 
> [1]: 
> https://wiki.openstack.org/wiki/ReviewChecklist#Oslo_Syncing_Checklist
>
>  [2]: https://wiki.openstack.org/wiki/StableBranch
> 
> Cheers, /Ihar
> 
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUEyR2AAoJEC5aWaUY1u57QS4H+gPHfebOBKJJAdhSjTRkaR9/
cV6A/M+snCAmlL5YcdMNruwAqaotvXMmUiUL2Mdekne7GqLlTwAtSnDQxwvr7BYu
m1Hu1/eRwVQZLS33UzvZRdAHJMlgD7Mq5p6w21yNKOVa+3wrXY+Q/JTAVv5i/pJ9
UQWTJpbE3DGT8j8B6jFrPbaMnjYjVrbHdGyvxqEaSdS0259kDSgSwULRmAilPRBd
3gIwZC1obqePkby7amQEIYKkPa53aFz2mSEPsWpaT2nYNLILCOcN5OLGNkdo1ksu
5gJ1hXx4MBuKbGALUO7QcdXgquXGv6O1hMq1GSi8bRsbxKbVn4XktsnS/ULqRSE=
=lzyp
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Chris Dent]

On Fri, 12 Sep 2014, Doug Hellmann wrote:


We could use a git hook (see my earlier message in this thread) or we
could add a command to tox to remove them before starting the tests.
Neither of those solutions would affect the runtime behavior in a way
that makes our dev environments fundamentally different from a
devstack or production deployment.


For reference, I've always been in the habit of managing automated
tasks in code checkouts with a Makefile that has targets with
depencies: e.g 'make test' will always do the 'clean' target and
'clean' does something like find . -name "*.pyc" ...

My stumble into openstack to find tox being the one _visible_ source
of automation was quite a shock. Things like git hooks do not count
as visible, despite being useful.

The idea being that the systems we work with as developers need to
be both discoverable and introspectable. Or to be clear:
transparent.

tox, in general, is vaguely magical, and it would be a shame to add
some more. Yes, the pyc thing screwing with your stuff is bad and
bit me hard a few times but once I knew what it was what I really
wanted was a quick (and centrally approved) way to do 'make clean'.

I guess tox -eclean or something would be the same but it feels
wrong somehow.

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Daniel P. Berrange]

On Fri, Sep 12, 2014 at 04:23:09PM +, Jeremy Stanley wrote:
> On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote:
> [...]
> > Agreed, the problem with stale .pyc files is that it never occurs to
> > developers that .pyc files are causing the problem until after you've
> > wasted (potentially hours of) time debugging the problem. Avoiding
> > this pain for all developers out of the box is a clear win overall
> > and makes openstack development less painful.
> 
> I've been bitten by similar issues often enough that I regularly git
> clean -dfx my checkouts or at least pass -r to tox so that it will
> recreate its virtualenvs from scratch. Yes it does add some extra
> time to the next test run, but you can iterate fairly tightly after
> that as long as you're not actively moving stuff around while you
> troubleshoot (and coupled with a git hook like Doug described for
> cleaning on topic branch changes would be a huge boon as well).

I'm not debating whether there are ways to clean up your env to avoid
the problem /after/ it occurs. The point is to stop the problem occuring
in the first place to avoid placing this uneccessary clean up burden
on devs.  Intentionally leaving things setup so that contributors hit
bugs like stale .pyc files is just user hostile.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Chris Dent]

On Fri, 12 Sep 2014, Julien Danjou wrote:


I guess the problem is more likely that testrepository load the tests
From the source directory whereas maybe we could make it load them from
what's installed into the venv?


This rather ruins TDD doesn't it?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Jeremy Stanley]

On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote:
[...]
> Agreed, the problem with stale .pyc files is that it never occurs to
> developers that .pyc files are causing the problem until after you've
> wasted (potentially hours of) time debugging the problem. Avoiding
> this pain for all developers out of the box is a clear win overall
> and makes openstack development less painful.

I've been bitten by similar issues often enough that I regularly git
clean -dfx my checkouts or at least pass -r to tox so that it will
recreate its virtualenvs from scratch. Yes it does add some extra
time to the next test run, but you can iterate fairly tightly after
that as long as you're not actively moving stuff around while you
troubleshoot (and coupled with a git hook like Doug described for
cleaning on topic branch changes would be a huge boon as well).
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Tempest Bug triage

[Mauro S M Rodrigues]

On 09/11/2014 04:52 PM, David Kranz wrote:
So we had a Bug Day this week and the results were a bit disappointing 
due to lack of participation. We went from 124 New bugs to 75. There 
were also many cases where bugs referred to logs that no longer 
existed. This suggests that we really need to keep up with bug triage 
in real time. Since bug triage should involve the Core review team, we 
propose to rotate the responsibility of triaging bugs weekly. I put up 
an etherpad here 
https://etherpad.openstack.org/p/qa-bug-triage-rotation and I hope the 
tempest core review team will sign up. Given our size, this should 
involve signing up once every two months or so. I took next week.


 -David
+1, I'm not core team but I just assigned myself to the last week of 
September and first of December.


Also, given the bad quality of some reports we may want to come up with 
a template of "need to have" data on the bug reports. I really haven't 
look at it lately but we use to have several reports with just a bunch 
of traces, or just a link..


  --  mauro(sr)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Jeremy Stanley]

On 2014-09-12 12:07:41 -0400 (-0400), Mike Bayer wrote:
[...]
> corresponding to PYTHONHASHSEED, right?  That whole thing is
> Python 3 only.

See other reply, but I really don't understand where you got that
idea. Yes Python 2.x does not randomize the hash seed by default
like Py3K (you have to pass -R to get that behavior) but you can
still totally override the hash seed from the environment in 2.x
(and more recent versions of tox happily do this for you and print
out the hash seed which was chosen for a given test run).
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Daniel P. Berrange]

On Fri, Sep 12, 2014 at 12:03:38PM -0400, Sean Dague wrote:
> On 09/12/2014 11:52 AM, Doug Hellmann wrote:
> > 
> > On Sep 12, 2014, at 11:21 AM, Mike Bayer  wrote:
> > 
> > I have to agree with Mike here. Cleaning up our dev environments using a 
> > little automation is better than disabling a feature of the interpreter 
> > that may have unforeseen consequences in behavior or performance. The more 
> > we introduce unusual settings like this into our environments and tools, 
> > the more edge cases and weirdness we’re going to find in those tools that 
> > keep us from doing the work we really want to be doing.
> > 
> > We could use a git hook (see my earlier message in this thread) or we could 
> > add a command to tox to remove them before starting the tests. Neither of 
> > those solutions would affect the runtime behavior in a way that makes our 
> > dev environments fundamentally different from a devstack or production 
> > deployment.
> 
> You believe that unit tests are going to change in the way they run so
> dramatically with this change that it invalidates their use?
> 
> Do we have examples of what changes if you do and don't have pyc files
> there?
> 
> Remember, we're not changing integration testing with this. This is
> solely unit testing.
> 
> The reason I don't like "just fix it in your local env" is you are then
> exporting the complexity to developers. For something that they should
> really not have to get bitten by... a lot.

Agreed, the problem with stale .pyc files is that it never occurs to
developers that .pyc files are causing the problem until after you've
wasted (potentially hours of) time debugging the problem. Avoiding
this pain for all developers out of the box is a clear win overall
and makes openstack development less painful.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Jeremy Stanley]

On 2014-09-12 11:36:20 -0400 (-0400), Mike Bayer wrote:
[...]
> not to mention PYTHONHASHSEED only works on Python 3.  What is the
> issue in tox you’re referring to ?

Huh? The overrides we added to numerous projects' tox.ini files to
stem the breakage in Python 2.x unit tests from hash seed
randomization in newer tox releases would seem to contradict your
assertion. Also documentation...

https://docs.python.org/2.7/using/cmdline.html#envvar-PYTHONHASHSEED

(New in version 2.6.8.)
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Sean Dague]

On 09/12/2014 12:07 PM, Mike Bayer wrote:
> 
> On Sep 12, 2014, at 12:03 PM, Sean Dague  wrote:
> 
>> On 09/12/2014 11:33 AM, Mike Bayer wrote:
>>>
>>> On Sep 12, 2014, at 11:24 AM, Sean Dague  wrote:
>>>
 On 09/12/2014 11:21 AM, Mike Bayer wrote:
>
> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
>
>> I assume you, gentle OpenStack developers, often find yourself in a hair
>> tearing out moment of frustration about why local unit tests are doing
>> completely insane things. The code that it is stack tracing on is no
>> where to be found, and yet it fails.
>>
>> And then you realize that part of oslo doesn't exist any more
>> except there are still pyc files laying around. Gah!
>>
>> I've proposed the following to Nova and Python novaclient -
>> https://review.openstack.org/#/c/121044/
>>
>> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>
> my VPN was down and I didn’t get this thread just now, but I am strongly 
> -1 on this as added to tox.ini, my response is 
> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
>
> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
> *your* environment.  Don’t force it on our automated tests or on my 
> environment.   .pyc files make a difference in behavior, and if we banish 
> them from all testing, then our code is never tested within the 
> environment that it will normally be run in after shipment.
>
> I’d far prefer a simple script added to tox.ini which deletes orphaned 
> .pyc files only, if a change to tox.ini must be made.

 Your example in the other thread includes the random seed behavior,
 which is already addressed in new tox. So I don't see that as an issue.
>>>
>>> Will these patches all be accompanied by corresponding PYTHONHASHSEED 
>>> settings?   Also why don’t you want to place PYTHONDONTWRITEBYTECODE into 
>>> your own environment?I don’t want this flag on my machine.
>>
>> This was the set of tox changes that went in in August.
> 
> corresponding to PYTHONHASHSEED, right?  That whole thing is Python 3 only.

It very much is not only python 3. We have to pin in on a bunch of our
python 2 tests now until we clean them up. There was a giant cross
project effort on this all through July & August to bring this in.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 12:03 PM, Sean Dague  wrote:

> On 09/12/2014 11:33 AM, Mike Bayer wrote:
>> 
>> On Sep 12, 2014, at 11:24 AM, Sean Dague  wrote:
>> 
>>> On 09/12/2014 11:21 AM, Mike Bayer wrote:
 
 On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
 
> I assume you, gentle OpenStack developers, often find yourself in a hair
> tearing out moment of frustration about why local unit tests are doing
> completely insane things. The code that it is stack tracing on is no
> where to be found, and yet it fails.
> 
> And then you realize that part of oslo doesn't exist any more
> except there are still pyc files laying around. Gah!
> 
> I've proposed the following to Nova and Python novaclient -
> https://review.openstack.org/#/c/121044/
> 
> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
 
 my VPN was down and I didn’t get this thread just now, but I am strongly 
 -1 on this as added to tox.ini, my response is 
 http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
 
 Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
 *your* environment.  Don’t force it on our automated tests or on my 
 environment.   .pyc files make a difference in behavior, and if we banish 
 them from all testing, then our code is never tested within the 
 environment that it will normally be run in after shipment.
 
 I’d far prefer a simple script added to tox.ini which deletes orphaned 
 .pyc files only, if a change to tox.ini must be made.
>>> 
>>> Your example in the other thread includes the random seed behavior,
>>> which is already addressed in new tox. So I don't see that as an issue.
>> 
>> Will these patches all be accompanied by corresponding PYTHONHASHSEED 
>> settings?   Also why don’t you want to place PYTHONDONTWRITEBYTECODE into 
>> your own environment?I don’t want this flag on my machine.
> 
> This was the set of tox changes that went in in August.

corresponding to PYTHONHASHSEED, right?  That whole thing is Python 3 only.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 11:55 AM, Julien Danjou  wrote:

> On Fri, Sep 12 2014, Sean Dague wrote:
> 
>> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>> 
>> This prevents pyc files from being writen in your git tree (win!). It
>> doesn't seem to impact what pip installs... and if anyone knows how to
>> prevent those pyc files from getting created, that would be great.
>> 
>> But it's something which hopefully causes less perceived developer
>> fragility of the system.
> 
> I understand it's generating .pyc could be something, but I don't really
> like that patch.
> 
> I guess the problem is more likely that testrepository load the tests
> From the source directory whereas maybe we could make it load them from
> what's installed into the venv?

we do this in oslo.db by doing an install within tox.ini and then making sure 
we don’t set usedevelop.  However, oslo.db does this because there’s issues 
with using namespace packages (e.g. oslo/db, oslo/utils, etc.) when you mix up 
installs with develop installations.   I hate it, and I’d like to someday solve 
that problem differently (locally I will often manually craft a test 
environment with PYTHONPATH just to avoid it).  I run different test runners 
based on what I’m trying to do and I skip tox for running individual tests, so 
this behavior gets in my way constantly, whereas the .pyc file issue almost 
never.

So IMO this approach as a means to get around the infrequent .pyc annoyance 
introduces lots more inconvenience than it saves.












> 
> -- 
> Julien Danjou
> /* Free Software hacker
>   http://julien.danjou.info */
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Sean Dague]

On 09/12/2014 11:52 AM, Doug Hellmann wrote:
> 
> On Sep 12, 2014, at 11:21 AM, Mike Bayer  wrote:
> 
>>
>> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
>>
>>> I assume you, gentle OpenStack developers, often find yourself in a hair
>>> tearing out moment of frustration about why local unit tests are doing
>>> completely insane things. The code that it is stack tracing on is no
>>> where to be found, and yet it fails.
>>>
>>> And then you realize that part of oslo doesn't exist any more
>>> except there are still pyc files laying around. Gah!
>>>
>>> I've proposed the following to Nova and Python novaclient -
>>> https://review.openstack.org/#/c/121044/
>>>
>>> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>>
>> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
>> on this as added to tox.ini, my response is 
>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
>>
>> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
>> *your* environment.  Don’t force it on our automated tests or on my 
>> environment.   .pyc files make a difference in behavior, and if we banish 
>> them from all testing, then our code is never tested within the environment 
>> that it will normally be run in after shipment.
>>
>> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
>> files only, if a change to tox.ini must be made.
> 
> I have to agree with Mike here. Cleaning up our dev environments using a 
> little automation is better than disabling a feature of the interpreter that 
> may have unforeseen consequences in behavior or performance. The more we 
> introduce unusual settings like this into our environments and tools, the 
> more edge cases and weirdness we’re going to find in those tools that keep us 
> from doing the work we really want to be doing.
> 
> We could use a git hook (see my earlier message in this thread) or we could 
> add a command to tox to remove them before starting the tests. Neither of 
> those solutions would affect the runtime behavior in a way that makes our dev 
> environments fundamentally different from a devstack or production deployment.

You believe that unit tests are going to change in the way they run so
dramatically with this change that it invalidates their use?

Do we have examples of what changes if you do and don't have pyc files
there?

Remember, we're not changing integration testing with this. This is
solely unit testing.

The reason I don't like "just fix it in your local env" is you are then
exporting the complexity to developers. For something that they should
really not have to get bitten by... a lot.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Sean Dague]

On 09/12/2014 11:33 AM, Mike Bayer wrote:
> 
> On Sep 12, 2014, at 11:24 AM, Sean Dague  wrote:
> 
>> On 09/12/2014 11:21 AM, Mike Bayer wrote:
>>>
>>> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
>>>
 I assume you, gentle OpenStack developers, often find yourself in a hair
 tearing out moment of frustration about why local unit tests are doing
 completely insane things. The code that it is stack tracing on is no
 where to be found, and yet it fails.

 And then you realize that part of oslo doesn't exist any more
 except there are still pyc files laying around. Gah!

 I've proposed the following to Nova and Python novaclient -
 https://review.openstack.org/#/c/121044/

 Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>>>
>>> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
>>> on this as added to tox.ini, my response is 
>>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
>>>
>>> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
>>> *your* environment.  Don’t force it on our automated tests or on my 
>>> environment.   .pyc files make a difference in behavior, and if we banish 
>>> them from all testing, then our code is never tested within the environment 
>>> that it will normally be run in after shipment.
>>>
>>> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
>>> files only, if a change to tox.ini must be made.
>>
>> Your example in the other thread includes the random seed behavior,
>> which is already addressed in new tox. So I don't see that as an issue.
> 
> Will these patches all be accompanied by corresponding PYTHONHASHSEED 
> settings?   Also why don’t you want to place PYTHONDONTWRITEBYTECODE into 
> your own environment?I don’t want this flag on my machine.

This was the set of tox changes that went in in August.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [clients] [keystone] lack of retrying tokens leads to overall OpenStack fragility

[Steven Hardy]

On Thu, Sep 11, 2014 at 08:43:22PM -0400, Jamie Lennox wrote:
> 
> 
> - Original Message -
> > From: "Steven Hardy" 
> > To: "OpenStack Development Mailing List (not for usage questions)" 
> > 
> > Sent: Friday, 12 September, 2014 12:21:52 AM
> > Subject: Re: [openstack-dev] [all] [clients] [keystone] lack of retrying 
> > tokens leads to overall OpenStack fragility
> > 
> > On Wed, Sep 10, 2014 at 08:46:45PM -0400, Jamie Lennox wrote:
> > > 
> > > - Original Message -
> > > > From: "Steven Hardy" 
> > > > To: "OpenStack Development Mailing List (not for usage questions)"
> > > > 
> > > > Sent: Thursday, September 11, 2014 1:55:49 AM
> > > > Subject: Re: [openstack-dev] [all] [clients] [keystone] lack of retrying
> > > > tokens leads to overall OpenStack fragility
> > > > 
> > > > On Wed, Sep 10, 2014 at 10:14:32AM -0400, Sean Dague wrote:
> > > > > Going through the untriaged Nova bugs, and there are a few on a 
> > > > > similar
> > > > > pattern:
> > > > > 
> > > > > Nova operation in progress takes a while
> > > > > Crosses keystone token expiration time
> > > > > Timeout thrown
> > > > > Operation fails
> > > > > Terrible 500 error sent back to user
> > > > 
> > > > We actually have this exact problem in Heat, which I'm currently trying
> > > > to
> > > > solve:
> > > > 
> > > > https://bugs.launchpad.net/heat/+bug/1306294
> > > > 
> > > > Can you clarify, is the issue either:
> > > > 
> > > > 1. Create novaclient object with username/password
> > > > 2. Do series of operations via the client object which eventually fail
> > > > after $n operations due to token expiry
> > > > 
> > > > or:
> > > > 
> > > > 1. Create novaclient object with username/password
> > > > 2. Some really long operation which means token expires in the course of
> > > > the service handling the request, blowing up and 500-ing
> > > > 
> > > > If the former, then it does sound like a client, or usage-of-client bug,
> > > > although note if you pass a *token* vs username/password (as is 
> > > > currently
> > > > done for glance and heat in tempest, because we lack the code to get the
> > > > token outside of the shell.py code..), there's nothing the client can 
> > > > do,
> > > > because you can't request a new token with longer expiry with a token...
> > > > 
> > > > However if the latter, then it seems like not really a client problem to
> > > > solve, as it's hard to know what action to take if a request failed
> > > > part-way through and thus things are in an unknown state.
> > > > 
> > > > This issue is a hard problem, which can possibly be solved by
> > > > switching to a trust scoped token (service impersonates the user), but
> > > > then
> > > > you're effectively bypassing token expiry via delegation which sits
> > > > uncomfortably with me (despite the fact that we may have to do this in
> > > > heat
> > > > to solve the afforementioned bug)
> > > > 
> > > > > It seems like we should have a standard pattern that on token
> > > > > expiration
> > > > > the underlying code at least gives one retry to try to establish a new
> > > > > token to complete the flow, however as far as I can tell *no* clients
> > > > > do
> > > > > this.
> > > > 
> > > > As has been mentioned, using sessions may be one solution to this, and
> > > > AFAIK session support (where it doesn't already exist) is getting into
> > > > various clients via the work being carried out to add support for v3
> > > > keystone by David Hu:
> > > > 
> > > > https://review.openstack.org/#/q/owner:david.hu%2540hp.com,n,z
> > > > 
> > > > I see patches for Heat (currently gating), Nova and Ironic.
> > > > 
> > > > > I know we had to add that into Tempest because tempest runs can exceed
> > > > > 1
> > > > > hr, and we want to avoid random fails just because we cross a token
> > > > > expiration boundary.
> > > > 
> > > > I can't claim great experience with sessions yet, but AIUI you could do
> > > > something like:
> > > > 
> > > > from keystoneclient.auth.identity import v3
> > > > from keystoneclient import session
> > > > from keystoneclient.v3 import client
> > > > 
> > > > auth = v3.Password(auth_url=OS_AUTH_URL,
> > > >username=USERNAME,
> > > >password=PASSWORD,
> > > >project_id=PROJECT,
> > > >user_domain_name='default')
> > > > sess = session.Session(auth=auth)
> > > > ks = client.Client(session=sess)
> > > > 
> > > > And if you can pass the same session into the various clients tempest
> > > > creates then the Password auth-plugin code takes care of 
> > > > reauthenticating
> > > > if the token cached in the auth plugin object is expired, or nearly
> > > > expired:
> > > > 
> > > > https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/auth/identity/base.py#L120
> > > > 
> > > > So in the tempest case, it seems like it may be a case of migrating the
> > > > code creating the clients to use sessions instead of passing a token or
> > 

Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep

[Morgan Fainberg]

I do not see python-memcache library in either keystone client’s 
requirements.txt[0] or test-requirements.txt[1]. For purposes of ensuring that 
we do not break people deploying auth_token in keystoneclient (for older 
releases) I don’t see the soft dependency on python-memcache from going away.

Even for keystonemiddleware we do not have a hard-dependency on python-memcache 
in requirements.txt[2] or test-requirements.txt[3] as we gate on py33.

—Morgan 

[0] 
http://git.openstack.org/cgit/openstack/python-keystoneclient/tree/requirements.txt?id=0.10.1
[1] 
http://git.openstack.org/cgit/openstack/python-keystoneclient/tree/test-requirements.txt?id=0.10.1
[2] 
http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/requirements.txt?id=1.1.1
[3] 
http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/test-requirements.txt?id=1.1.1

—
Morgan Fainberg


-Original Message-
From: Brant Knudson 
Reply: OpenStack Development Mailing List (not for usage questions) 
>
Date: September 12, 2014 at 08:33:15
To: OpenStack Development Mailing List (not for usage questions) 
>
Subject:  Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive 
dep

> On Thu, Sep 11, 2014 at 2:17 AM, Thomas Goirand wrote:
>  
> >
> > On my side (as the Debian package maintainer of OpenStack), I was more
> > than happy to see that Ceilometer made the choice to use a Python module
> > for memcache which supports Python 3. Currently python-memcache does
> > *not* support Python 3. It's in fact standing in the way to add Python 3
> > compatibility to *a lot* of the OpenStack packages, because this
> > directly impact python-keystoneclient, which is a (build-)dependency of
> > almost everything.
> >
> >
> Thomas -
>  
> python-keystoneclient should no longer have a hard dependency on
> python-memcache(d). The auth_token middleware which can use memcache has
> been moved into the keystonemiddleware repo (a copy is left in
> keystoneclient only for backwards-compatibility). If python-keystoneclient
> still has a dependency on python-memcache then we're doing something wrong
> and should be able to fix it.
>  
> - Brant
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>  


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Mike Bayer]

On Sep 12, 2014, at 11:56 AM, Johannes Erdfelt  wrote:

> On Fri, Sep 12, 2014, Doug Hellmann  wrote:
>> I don’t think we will want to retroactively change the migration scripts
>> (that’s not something we generally like to do),
> 
> We don't allow semantic changes to migration scripts since people who
> have already run it won't get those changes. However, we haven't been
> shy about fixing bugs that prevent the migration script from running
> (which this change would probably fall into).

fortunately BEGIN/ COMMIT are not semantic directives. The migrations 
semantically indicated by the script are unaffected in any way by these 
run-environment settings.


> 
>> so we should look at changes needed to make sqlalchemy-migrate deal with
>> them (by ignoring them, or working around the errors, or whatever).
> 
> That said, I agree that sqlalchemy-migrate shouldn't be changing in a
> non-backwards compatible way.

on the sqlalchemy-migrate side, the handling of it’s ill-conceived “sql script” 
feature can be further mitigated here by parsing for the “COMMIT” line when it 
breaks out the SQL and ignoring it, I’d favor that it emits a warning also.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] battling stale .pyc files

[Julien Danjou]

On Fri, Sep 12 2014, Mike Bayer wrote:

> Just my 2.5c on this issue as to the approach I think is best. Leave
> the Python interpreter’s behavior as much as “normal” as possible in
> our default test environment.

I definitely agree with all of that. :)

-- 
Julien Danjou
// Free Software hacker
// http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Johannes Erdfelt]

On Fri, Sep 12, 2014, Doug Hellmann  wrote:
> I don’t think we will want to retroactively change the migration scripts
> (that’s not something we generally like to do),

We don't allow semantic changes to migration scripts since people who
have already run it won't get those changes. However, we haven't been
shy about fixing bugs that prevent the migration script from running
(which this change would probably fall into).

> so we should look at changes needed to make sqlalchemy-migrate deal with
> them (by ignoring them, or working around the errors, or whatever).

That said, I agree that sqlalchemy-migrate shouldn't be changing in a
non-backwards compatible way.

JE


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Julien Danjou]

On Fri, Sep 12 2014, Sean Dague wrote:

> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>
> This prevents pyc files from being writen in your git tree (win!). It
> doesn't seem to impact what pip installs... and if anyone knows how to
> prevent those pyc files from getting created, that would be great.
>
> But it's something which hopefully causes less perceived developer
> fragility of the system.

I understand it's generating .pyc could be something, but I don't really
like that patch.

I guess the problem is more likely that testrepository load the tests
From the source directory whereas maybe we could make it load them from
what's installed into the venv?

-- 
Julien Danjou
/* Free Software hacker
   http://julien.danjou.info */


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Thierry Carrez]

Anita Kuno wrote:
> My question involves third party discussions. Now I know at least
> Neutron is going to have a chat about drivers which involves third party
> ci accounts as a supportive aspect of that discussion, but I am
> wondering about the framework for a discussion which I can recommend
> attendees of the third party meetings to attend. They are shaping up to
> be an attentive, forward thinking group and are supporting each other
> which I was hoping for from the beginning so I am very heartened by our
> progress. I am feeling that as a group folks have questions and concerns
> they would appreciate the opportunity to air in a mutually constructive
> venue.
> 
> What day and where would be the mutually constructive venue?
> 
> I held off on Joe's thread since third party ci affects 4 or 5 programs,
> not enough to qualify in my mind as a topic that is OpenStack wide, but
> the programs it affects are quite affected, so I do feel it is time to
> mention it.

I think those discussions could happen in a cross-project workshop.
We'll run 2 or 3 of those in parallel all day Tuesday, so there is
definitely room there.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Doug Hellmann]

On Sep 12, 2014, at 11:21 AM, Mike Bayer  wrote:

> 
> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
> 
>> I assume you, gentle OpenStack developers, often find yourself in a hair
>> tearing out moment of frustration about why local unit tests are doing
>> completely insane things. The code that it is stack tracing on is no
>> where to be found, and yet it fails.
>> 
>> And then you realize that part of oslo doesn't exist any more
>> except there are still pyc files laying around. Gah!
>> 
>> I've proposed the following to Nova and Python novaclient -
>> https://review.openstack.org/#/c/121044/
>> 
>> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
> 
> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
> on this as added to tox.ini, my response is 
> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
> 
> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
> *your* environment.  Don’t force it on our automated tests or on my 
> environment.   .pyc files make a difference in behavior, and if we banish 
> them from all testing, then our code is never tested within the environment 
> that it will normally be run in after shipment.
> 
> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
> files only, if a change to tox.ini must be made.

I have to agree with Mike here. Cleaning up our dev environments using a little 
automation is better than disabling a feature of the interpreter that may have 
unforeseen consequences in behavior or performance. The more we introduce 
unusual settings like this into our environments and tools, the more edge cases 
and weirdness we’re going to find in those tools that keep us from doing the 
work we really want to be doing.

We could use a git hook (see my earlier message in this thread) or we could add 
a command to tox to remove them before starting the tests. Neither of those 
solutions would affect the runtime behavior in a way that makes our dev 
environments fundamentally different from a devstack or production deployment.

Doug


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Thierry Carrez]

Russell Bryant wrote:
> On 09/12/2014 07:37 AM, Thierry Carrez wrote:
>> If you think this is wrong and think the "design summit suggestion"
>> website is a better way to do it, let me know why! If some programs
>> really can't stand the 'etherpad/IRC' approach I'll see how we can spin
>> up a limited instance.
> 
> I think this is fine, especially if it's a better reflection of reality
> and lets the teams work more efficiently.
> 
> However, one of the benefits of the old submission system was the
> clarity of the process and openness to submissions from anyone.  We
> don't want to be in a situation where non-core folks feel like they have
> a harder time submitting a session.
> 
> Once this is settled, as long as the wiki pages [1][2] reflect the
> process and is publicized, it should be fine.
> 
> [1] https://wiki.openstack.org/wiki/Summit
> [2] https://wiki.openstack.org/wiki/Summit/Planning

Yes, I'll document the new process and heavily publicize it, once I'm
sure that's the way forward :)

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Thierry Carrez]

Eoghan Glynn wrote:
>> If you think this is wrong and think the "design summit suggestion"
>> website is a better way to do it, let me know why! If some programs
>> really can't stand the 'etherpad/IRC' approach I'll see how we can spin
>> up a limited instance.
> 
> +1 on a collaborative scheduling process within each project.
> 
> That's pretty much what we did within the ceilometer core group for
> the Juno summit, except that we used a googledocs spreadsheet instead
> of an etherpad.
> 
> So I don't think we need to necessarily mandate usage of an etherpad,
> just let every project decide whatever shared document format they
> want to use.
> 
> FTR the benefit of a googledocs spreadsheet in my view would include
> the ease of totalling votes & sessions slots, color-coding candidate
> sessions for merging etc.

Good point. I've replaced the wording in the wiki page -- just use
whatever suits you best, as long as it's a public document and you can
link to it.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Anita Kuno]

On 09/12/2014 07:37 AM, Thierry Carrez wrote:
> Hi everyone,
> 
> I visited the Paris Design Summit space on Monday and confirmed that it
> should be possible to split it in a way that would allow to have
> per-program "contributors meetups" on the Friday. The schedule would go
> as follows:
> 
> Tuesday: cross-project workshops
> Wednesday, Thursday: traditional "scheduled" slots
> Friday: contributors meetups
> 
> We'll also have "pods" available all 4 days for more ad-hoc small meetings.
> 
> In the mean time, we need to discuss how we want to handle the selection
> of session topics.
> 
> In past summits we used a Design-Summit-specific "session suggestion"
> website, and PTLs would approve/deny them. This setup grew less and less
> useful: session topics were selected collaboratively on etherpads,
> discussed in meetings, and finally filed/reorganized/merged on the
> website just before scheduling. Furthermore, with even less "scheduled"
> slots, we would have to reject most of the suggestions, which is more
> frustrating for submitters than the positive experience of joining team
> meetings to discuss which topics are the most important. Finally, topics
> will need to be split between "scheduled" sessions and the "contributors
> meetup" agenda, and that's easier to do on an Etherpad anyway.
> 
> This is why I'd like to suggest that all programs use etherpads to
> collect important topics, select which ones would get in the very few
> "scheduled" slots we'll have left, which will get discussed in the
> "contributors meetup", and which are better left for a "pod" discussion.
> I suggest we all use IRC team meetings to collaboratively discuss that
> content between interested contributors.
> 
> To simplify the communication around this, I tried to collect the
> already-announced etherpads on a single page at:
> 
> https://wiki.openstack.org/wiki/Summit/Planning
> 
> Please add any that I missed !
> 
> If you think this is wrong and think the "design summit suggestion"
> website is a better way to do it, let me know why! If some programs
> really can't stand the 'etherpad/IRC' approach I'll see how we can spin
> up a limited instance.
> 
> Regards,
> 
Thanks Thierry,

This looks like it should shape up to be a nice buffet of formats for us
to evaluate and then provide feedback on what works best for whom at the
wrap-up (which I believe will now be on the mailing list after the summit).

My question involves third party discussions. Now I know at least
Neutron is going to have a chat about drivers which involves third party
ci accounts as a supportive aspect of that discussion, but I am
wondering about the framework for a discussion which I can recommend
attendees of the third party meetings to attend. They are shaping up to
be an attentive, forward thinking group and are supporting each other
which I was hoping for from the beginning so I am very heartened by our
progress. I am feeling that as a group folks have questions and concerns
they would appreciate the opportunity to air in a mutually constructive
venue.

What day and where would be the mutually constructive venue?

I held off on Joe's thread since third party ci affects 4 or 5 programs,
not enough to qualify in my mind as a topic that is OpenStack wide, but
the programs it affects are quite affected, so I do feel it is time to
mention it.

Thanks,
Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-dev][Cinder] FFE request for adding Huawei SDSHypervisor driver and connector

[Thierry Carrez]

Zhangni wrote:
> I'd like to request an Juno feature freeze exception for this blueprint
> and Spec:
> 
> https://blueprints.launchpad.net/cinder/+spec/huawei-sdshypervisor-driver
> 
> https://review.openstack.org/#/c/101688/
> 
> as implemented by the following patch:
> 
> https://review.openstack.org/#/c/108609

I would say it's way too late at this point for a new driver in Juno. At
this point we should be focused on fixing what we already have, not add
more surface.

Cheers,

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 11:33 AM, Mike Bayer  wrote:

> 
> On Sep 12, 2014, at 11:24 AM, Sean Dague  wrote:
> 
>> On 09/12/2014 11:21 AM, Mike Bayer wrote:
>>> 
>>> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
>>> 
 I assume you, gentle OpenStack developers, often find yourself in a hair
 tearing out moment of frustration about why local unit tests are doing
 completely insane things. The code that it is stack tracing on is no
 where to be found, and yet it fails.
 
 And then you realize that part of oslo doesn't exist any more
 except there are still pyc files laying around. Gah!
 
 I've proposed the following to Nova and Python novaclient -
 https://review.openstack.org/#/c/121044/
 
 Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>>> 
>>> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
>>> on this as added to tox.ini, my response is 
>>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
>>> 
>>> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
>>> *your* environment.  Don’t force it on our automated tests or on my 
>>> environment.   .pyc files make a difference in behavior, and if we banish 
>>> them from all testing, then our code is never tested within the environment 
>>> that it will normally be run in after shipment.
>>> 
>>> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
>>> files only, if a change to tox.ini must be made.
>> 
>> Your example in the other thread includes the random seed behavior,
>> which is already addressed in new tox. So I don't see that as an issue.
> 
> Will these patches all be accompanied by corresponding PYTHONHASHSEED 
> settings?   Also why don’t you want to place PYTHONDONTWRITEBYTECODE into 
> your own environment?I don’t want this flag on my machine.

not to mention PYTHONHASHSEED only works on Python 3.  What is the issue in tox 
you’re referring to ?




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep

[Brant Knudson]

On Thu, Sep 11, 2014 at 2:17 AM, Thomas Goirand  wrote:

>
> On my side (as the Debian package maintainer of OpenStack), I was more
> than happy to see that Ceilometer made the choice to use a Python module
> for memcache which supports Python 3. Currently python-memcache does
> *not* support Python 3. It's in fact standing in the way to add Python 3
> compatibility to *a lot* of the OpenStack packages, because this
> directly impact python-keystoneclient, which is a (build-)dependency of
> almost everything.
>
>
Thomas -

python-keystoneclient should no longer have a hard dependency on
python-memcache(d). The auth_token middleware which can use memcache has
been moved into the keystonemiddleware repo (a copy is left in
keystoneclient only for backwards-compatibility). If python-keystoneclient
still has a dependency on python-memcache then we're doing something wrong
and should be able to fix it.

- Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 11:24 AM, Sean Dague  wrote:

> On 09/12/2014 11:21 AM, Mike Bayer wrote:
>> 
>> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
>> 
>>> I assume you, gentle OpenStack developers, often find yourself in a hair
>>> tearing out moment of frustration about why local unit tests are doing
>>> completely insane things. The code that it is stack tracing on is no
>>> where to be found, and yet it fails.
>>> 
>>> And then you realize that part of oslo doesn't exist any more
>>> except there are still pyc files laying around. Gah!
>>> 
>>> I've proposed the following to Nova and Python novaclient -
>>> https://review.openstack.org/#/c/121044/
>>> 
>>> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
>> 
>> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
>> on this as added to tox.ini, my response is 
>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
>> 
>> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
>> *your* environment.  Don’t force it on our automated tests or on my 
>> environment.   .pyc files make a difference in behavior, and if we banish 
>> them from all testing, then our code is never tested within the environment 
>> that it will normally be run in after shipment.
>> 
>> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
>> files only, if a change to tox.ini must be made.
> 
> Your example in the other thread includes the random seed behavior,
> which is already addressed in new tox. So I don't see that as an issue.

Will these patches all be accompanied by corresponding PYTHONHASHSEED settings? 
  Also why don’t you want to place PYTHONDONTWRITEBYTECODE into your own 
environment?I don’t want this flag on my machine.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Mike Bayer]

On Sep 12, 2014, at 10:40 AM, Ihar Hrachyshka  wrote:

> Signed PGP part
> On 12/09/14 16:33, Mike Bayer wrote:
>> I agree with this, changing the MySQL driver now is not an option.
> 
> That was not the proposal. The proposal was to introduce support to
> run against something different from MySQLdb + a gate job for that
> alternative. The next cycle was supposed to do thorough regression
> testing, benchmarking, etc. to decide whether we're ok to recommend
> that alternative to users.

ah, well that is a great idea.  But we can have that throughout Kilo anyway, 
why not ?





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Sean Dague]

On 09/12/2014 11:19 AM, Doug Hellmann wrote:
> 
> On Sep 12, 2014, at 9:23 AM, Ihar Hrachyshka  wrote:
> 
>> Signed PGP part
>> On 12/09/14 13:20, Sean Dague wrote:
>>> On 09/12/2014 06:41 AM, Ihar Hrachyshka wrote:
 Some updates/concerns/questions.

 The status of introducing a new driver to gate is:

 - all the patches for mysql-connector are merged in all
 projects; - all devstack patches to support switching the driver
 are merged; - new sqlalchemy-migrate library is released;

 - version bump is *not* yet done; - package is still *not* yet
 published on pypi; - new gate job is *not* yet introduced.

 The new sqlalchemy-migrate release introduced unit test failures
 in those three projects: nova, cinder, glance.

 On technical side of the failure: my understanding is that those
 projects that started to fail assume too much about how those
 SQL scripts are executed. They assume they are executed in one
 go, they also assume they need to open and commit transaction on
 their own. I don't think this is something to be fixed in
 sqlalchemy-migrate itself. Instead, simple removal of those
 'BEGIN TRANSACTION; ... COMMIT;' statements should just work and
 looks like a sane thing to do anyway. I've proposed the following
 patches for all three projects to handle it [1].

 That said, those failures were solved by pinning the version of
 the library in openstack/requirements and those projects. This is
 in major contrast to how we handled the new testtools release
 just several weeks ago, when the problem was solved by fixing
 three affected projects because of their incorrect usage of
 tearDown/setUp methods.

 Even more so, those failures seem to trigger the resolution to
 move the enable-mysql-connector oslo spec to Kilo, while the
 library version bump is the *only* change missing codewise (we
 will also need a gate job description, but that doesn't touch
 codebase at all). The resolution looks too prompt and ungrounded
 to me. Is it really that gate failure for three projects that
 resulted in it, or there are some other hidden reasons behind it?
 Was it discussed anywhere? If so, I wasn't given a chance to
 participate in that discussion; I suspect another supporter of
 the spec (Agnus Lees) was not involved either.

 Not allowing those last pieces of the spec in this cycle, we
 just postpone start of any realistic testing of the feature for
 another half a year.

 Why do we block new sqlalchemy-migrate and the spec for another
 cycle instead of fixing the affected projects with *primitive*
 patches like we did for new testtools?
>>>
>>> Because we are in Feature Freeze. Now is the time for critical bug
>>> fixes only, as we start to stabalize the tree. Releasing dependent
>>> libraries that can cause breaks, for whatever reason, should be
>>> soundly avoided.
>>>
>>> If this was August, fine. But it's feature freeze.
>>
>> I probably missed the fact that we are so strict now that we don't
>> allow tiny missing bits to go in. In my excuse, I was offline for
>> around three last weeks. I was a bit misled by the fact that I was
>> approached by an oslo core very recently on which remaining bits we
>> need to push before claiming the spec to be complete, and I assumed it
>> means that we are free to complete the work this cycle. Otherwise, I
>> wouldn't push for the new library version in the first place.
> 
> I suspect you’re referring to me, there. I believed the work was ready to be 
> wrapped up. I’m sorry my misunderstanding led to the issues.
> 
>>
>> Anyway, I guess there is no way now to get remaining bits in Juno,
>> even if small, and we're doomed to postpone them to Kilo.
> 
> I think we’re only looking at a couple of weeks delay. During that time we 
> can work on fixing the problem. I don’t think we will want to retroactively 
> change the migration scripts (that’s not something we generally like to do), 
> so we should look at changes needed to make sqlalchemy-migrate deal with them 
> (by ignoring them, or working around the errors, or whatever).

Yes, please, that would be highly appreciated. That kind of backwards
compat guaruntees is kind of why we took over migrate in the first place
as a project.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Sean Dague]

On 09/12/2014 11:21 AM, Mike Bayer wrote:
> 
> On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:
> 
>> I assume you, gentle OpenStack developers, often find yourself in a hair
>> tearing out moment of frustration about why local unit tests are doing
>> completely insane things. The code that it is stack tracing on is no
>> where to be found, and yet it fails.
>>
>> And then you realize that part of oslo doesn't exist any more
>> except there are still pyc files laying around. Gah!
>>
>> I've proposed the following to Nova and Python novaclient -
>> https://review.openstack.org/#/c/121044/
>>
>> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
> 
> my VPN was down and I didn’t get this thread just now, but I am strongly -1 
> on this as added to tox.ini, my response is 
> http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.
> 
> Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into 
> *your* environment.  Don’t force it on our automated tests or on my 
> environment.   .pyc files make a difference in behavior, and if we banish 
> them from all testing, then our code is never tested within the environment 
> that it will normally be run in after shipment.
> 
> I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
> files only, if a change to tox.ini must be made.

Your example in the other thread includes the random seed behavior,
which is already addressed in new tox. So I don't see that as an issue.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Mike Bayer]

On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:

> I assume you, gentle OpenStack developers, often find yourself in a hair
> tearing out moment of frustration about why local unit tests are doing
> completely insane things. The code that it is stack tracing on is no
> where to be found, and yet it fails.
> 
> And then you realize that part of oslo doesn't exist any more
> except there are still pyc files laying around. Gah!
> 
> I've proposed the following to Nova and Python novaclient -
> https://review.openstack.org/#/c/121044/
> 
> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.

my VPN was down and I didn’t get this thread just now, but I am strongly -1 on 
this as added to tox.ini, my response is 
http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html.

Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* 
environment.  Don’t force it on our automated tests or on my environment.   
.pyc files make a difference in behavior, and if we banish them from all 
testing, then our code is never tested within the environment that it will 
normally be run in after shipment.

I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc 
files only, if a change to tox.ini must be made.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Doug Hellmann]

On Sep 12, 2014, at 9:23 AM, Ihar Hrachyshka  wrote:

> Signed PGP part
> On 12/09/14 13:20, Sean Dague wrote:
> > On 09/12/2014 06:41 AM, Ihar Hrachyshka wrote:
> >> Some updates/concerns/questions.
> >>
> >> The status of introducing a new driver to gate is:
> >>
> >> - all the patches for mysql-connector are merged in all
> >> projects; - all devstack patches to support switching the driver
> >> are merged; - new sqlalchemy-migrate library is released;
> >>
> >> - version bump is *not* yet done; - package is still *not* yet
> >> published on pypi; - new gate job is *not* yet introduced.
> >>
> >> The new sqlalchemy-migrate release introduced unit test failures
> >> in those three projects: nova, cinder, glance.
> >>
> >> On technical side of the failure: my understanding is that those
> >> projects that started to fail assume too much about how those
> >> SQL scripts are executed. They assume they are executed in one
> >> go, they also assume they need to open and commit transaction on
> >> their own. I don't think this is something to be fixed in
> >> sqlalchemy-migrate itself. Instead, simple removal of those
> >> 'BEGIN TRANSACTION; ... COMMIT;' statements should just work and
> >> looks like a sane thing to do anyway. I've proposed the following
> >> patches for all three projects to handle it [1].
> >>
> >> That said, those failures were solved by pinning the version of
> >> the library in openstack/requirements and those projects. This is
> >> in major contrast to how we handled the new testtools release
> >> just several weeks ago, when the problem was solved by fixing
> >> three affected projects because of their incorrect usage of
> >> tearDown/setUp methods.
> >>
> >> Even more so, those failures seem to trigger the resolution to
> >> move the enable-mysql-connector oslo spec to Kilo, while the
> >> library version bump is the *only* change missing codewise (we
> >> will also need a gate job description, but that doesn't touch
> >> codebase at all). The resolution looks too prompt and ungrounded
> >> to me. Is it really that gate failure for three projects that
> >> resulted in it, or there are some other hidden reasons behind it?
> >> Was it discussed anywhere? If so, I wasn't given a chance to
> >> participate in that discussion; I suspect another supporter of
> >> the spec (Agnus Lees) was not involved either.
> >>
> >> Not allowing those last pieces of the spec in this cycle, we
> >> just postpone start of any realistic testing of the feature for
> >> another half a year.
> >>
> >> Why do we block new sqlalchemy-migrate and the spec for another
> >> cycle instead of fixing the affected projects with *primitive*
> >> patches like we did for new testtools?
> >
> > Because we are in Feature Freeze. Now is the time for critical bug
> > fixes only, as we start to stabalize the tree. Releasing dependent
> > libraries that can cause breaks, for whatever reason, should be
> > soundly avoided.
> >
> > If this was August, fine. But it's feature freeze.
> 
> I probably missed the fact that we are so strict now that we don't
> allow tiny missing bits to go in. In my excuse, I was offline for
> around three last weeks. I was a bit misled by the fact that I was
> approached by an oslo core very recently on which remaining bits we
> need to push before claiming the spec to be complete, and I assumed it
> means that we are free to complete the work this cycle. Otherwise, I
> wouldn't push for the new library version in the first place.

I suspect you’re referring to me, there. I believed the work was ready to be 
wrapped up. I’m sorry my misunderstanding led to the issues.

> 
> Anyway, I guess there is no way now to get remaining bits in Juno,
> even if small, and we're doomed to postpone them to Kilo.

I think we’re only looking at a couple of weeks delay. During that time we can 
work on fixing the problem. I don’t think we will want to retroactively change 
the migration scripts (that’s not something we generally like to do), so we 
should look at changes needed to make sqlalchemy-migrate deal with them (by 
ignoring them, or working around the errors, or whatever).

Doug

> 
> Thanks for the explanation,
> /Ihar
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] battling stale .pyc files

[Mike Bayer]

I’ve just found https://bugs.launchpad.net/nova/+bug/1368661, "Unit tests 
sometimes fail because of stale pyc files”.

The issue as stated in the report refers to the phenomenon of .pyc files that 
remain inappropriately, when switching branches or deleting files.

Specifically, the kind of scenario that in my experience causes this looks like 
this.  One version of the code has a setup like this:

   mylibrary/mypackage/somemodule/__init__.py

Then, a different version we switch to changes it to this:

   mylibrary/mypackage/somemodule.py

But somemodule/__init__.pyc will still be sitting around, and then things break 
- the Python interpreter skips the module (or perhaps the other way around. I 
just ran a test by hand and it seems like packages trump modules in Python 2.7).

This is an issue for sure, however the fix that is proposed I find alarming, 
which is to use the PYTHONDONTWRITEBYTECODE=1 flag written directly into the 
tox.ini file to disable *all* .pyc file writing, for all environments 
unconditionally, both human and automated.

I think that approach is a mistake.  .pyc files have a definite effect on the 
behavior of the interpreter.   They can, for example, be the factor that causes 
a dictionary to order its elements in one way versus another;  I’ve had many 
relying-on-dictionary-ordering issues (which make no mistake, are bugs) smoked 
out by the fact that a .pyc file would reveal the issue..pyc files also 
naturally have a profound effect on performance.   I’d hate for the Openstack 
community to just forget that .pyc files ever existed, our tox.ini’s safely 
protecting us from them, and then we start seeing profiling results getting 
published that forgot to run the Python interpreter in it’s normal state of 
operation.  If we put this flag into every tox.ini, it means the totality of 
openstack testing will not only run more slowly, it also means our code will 
never be run within the Python runtime environment that will actually be used 
when code is shipped.   The Python interpreter is incredibly stable and 
predictable and a small change like this is hardly something that we’d usually 
notice…until something worth noticing actually goes wrong, and automated 
testing is where that should be found, not after shipment.

The issue of the occasional unmatched .pyc file whose name happens to still be 
imported by the application is not that frequent, and can be solved by just 
making sure unmatched .pyc files are deleted ahead of time.I’d favor a 
utility such as in oslo.utils which performs this simple step of finding all 
unmatched .pyc files and deleting (taking care to be aware of __pycache__ / 
pep3147), and can be invoked from tox.ini as a startup command.

But guess what - suppose you totally disagree and you really want to not have 
any .pyc files in your dev environment.   Simple!  Put 
PYTHONDONTWRITEBYTECODE=1 into *your* environment - it doesn’t need to be in 
tox.ini, just stick it in your .profile.   Let’s put it up on the wikis, let’s 
put it into the dev guides, let’s go nuts.   Banish .pyc files from your 
machine all you like.   But let’s *not* do this on our automated test 
environments, and not force it to happen in *my* environment. 

I also want to note that the issue of stale .pyc files should only apply to 
within the library subject to testing as it lives in its source directory.  
This has nothing to do with the packages that are installed under .tox as those 
are full packages, unless there’s some use case I’m not aware of (possible), we 
don’t checkout code into .tox nor do we manipulate files there as a matter of 
course.

Just my 2.5c on this issue as to the approach I think is best.   Leave the 
Python interpreter’s behavior as much as “normal” as possible in our default 
test environment.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Doug Hellmann]

On Sep 12, 2014, at 7:39 AM, Sean Dague  wrote:

> I assume you, gentle OpenStack developers, often find yourself in a hair
> tearing out moment of frustration about why local unit tests are doing
> completely insane things. The code that it is stack tracing on is no
> where to be found, and yet it fails.
> 
> And then you realize that part of oslo doesn't exist any more
> except there are still pyc files laying around. Gah!
> 
> I've proposed the following to Nova and Python novaclient -
> https://review.openstack.org/#/c/121044/
> 
> Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.
> 
> This prevents pyc files from being writen in your git tree (win!). It
> doesn't seem to impact what pip installs... and if anyone knows how to
> prevent those pyc files from getting created, that would be great.
> 
> But it's something which hopefully causes less perceived developer
> fragility of the system.
> 
>   -Sean

I also use git-hooks with a post-checkout script to remove pyc files any time I 
change between branches, which is especially helpful if the different branches 
have code being moved around:

git-hooks: https://github.com/icefox/git-hooks

The script:

$ cat ~/.git_hooks/post-checkout/remove_pyc
#!/bin/sh
echo "Removing pyc files from `pwd`"
find . -name '*.pyc' | xargs rm -f
exit 0

> 
> -- 
> Sean Dague
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Flavio Percoco]

On 09/12/2014 01:56 PM, Flavio Percoco wrote:
> On 09/12/2014 11:36 AM, Thierry Carrez wrote:
>> Flavio Percoco wrote:
>>> On 09/12/2014 12:14 AM, Zane Bitter wrote:
 The final question is the one of arbitrary access to messages in the
 queue (or "queue" if you prefer). Flavio indicated that this effectively
 came for free with their implementation of Pub-Sub. IMHO it is
 unnecessary and limits the choice of potential back ends in the future.
 I would personally be +1 on removing it from the v2 API, and also +1 on
 the v2 API shipping in Kilo so that as few new adopters as possible get
 stuck with the limited choices of back-end. I hope that would resolve
 Clint's concerns that we need a separate, light-weight queue system; I
 personally don't believe we need two projects, even though I agree that
 all of the use cases I personally care about could probably be satisfied
 without Pub-Sub.
>>>
>>> Right, being able to support other backends is one of the reasons we're
>>> looking forward to remove the support for arbitrary access to messages.
>>> As of now, the plan is to remove that endpoint unless a very good use
>>> case comes up that makes supporting other backends not worth it, which I
>>> doubt. The feedback from Zaqar's early adopters is that the endpoint is
>>> indeed not useful.
>>
>> Thanks Zane, that was indeed useful. I agree with you it would be better
>> to avoid needing 2 separate projects for such close use cases.
> 
> +1
> 
>> Let's assume we remove arbitrary access to messages in v2. When you say
>> it would remove limits on the choice of potential backends, does that
>> mean we could have a pure queue backend (like RabbitMQ), at least in
>> theory ? Would a ZaqarV2 address all of Clint and Devananda's concerns
>> about queue semantics ? If yes, then the graduation question becomes,
>> how likely is that work to be completed early enough in Kilo.
>>
>> If it's a no-brainer and takes a week to sort out, I think we could
>> approve Zaqar's Kilo graduation, even if that stretches the "no major
>> API rewrite planned" requirement.
> 
> Let me break the above down into several points so we can discuss them
> separately:
> 
> - Removing that endpoint won't take more than a week. It's an API change
> and it won't affect the existing storage drivers.

For the sake of discussion and to provide more info on this point, I've
done this (there are still some tests to clean-up but that's basically
all that's required):

https://review.openstack.org/#/c/121141/

Flavio

> 
> - Removing that endpoint will certainly make the adoption of other
> messaging technologies easier but there are other things to consider
> besides that specific endpoint (some of them were stated here[0]). In
> any case, removing the endpoint definitely makes it easier.
> 
> - Besides the random access to messages, I'm not clear what other
> concerns there are with regards the current semantics. It'd be nice if
> we could recollect them in this section and discuss them. I took a look
> at the other emails in this thread and it seems to me that the concerns
> that have been raised are more oriented to the project scope and
> use-cases. I also looked at the meeting logs again[1] and the only
> concern related to the semantics I found is about the
> `get-message-by-id` endpoint. Please, correct me if I'm wrong.
> 
> 
> [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/
> [1]
> http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-09-20.01.log.html
> 
> Flavio
> 
>> But if we think this needs careful discussion so that the v2 API design
>> (and backend support) satisfies the widest set of users, then incubating
>> for another cycle while v2 is implemented seems like the right course of
>> action. We shouldn't graduate if there is any risk we would end up with
>> ZaqarV1 in Kilo, and then have to deprecate it for n cycles just because
>> it was shipped in the official release and therefore inherits its API
>> deprecation rules.
>>
>> Regards,
>>
> 
> 


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Russell Bryant]

On 09/12/2014 07:37 AM, Thierry Carrez wrote:
> If you think this is wrong and think the "design summit suggestion"
> website is a better way to do it, let me know why! If some programs
> really can't stand the 'etherpad/IRC' approach I'll see how we can spin
> up a limited instance.

I think this is fine, especially if it's a better reflection of reality
and lets the teams work more efficiently.

However, one of the benefits of the old submission system was the
clarity of the process and openness to submissions from anyone.  We
don't want to be in a situation where non-core folks feel like they have
a harder time submitting a session.

Once this is settled, as long as the wiki pages [1][2] reflect the
process and is publicized, it should be fine.

[1] https://wiki.openstack.org/wiki/Summit
[2] https://wiki.openstack.org/wiki/Summit/Planning

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][db] Need help resolving a strange error with db connections in tests

[Anna Kamyshnikova]

Kevin Benton

Thank you very much!!! I lost hope to find a way to fix these tests!

Regards,
Ann Kamyshnikova

On Fri, Sep 12, 2014 at 5:17 PM, Kevin Benton  wrote:

> This one was tricky. :-)
>
> This failure can be produced consistently by running the following two
> tests:
>
> neutron.tests.unit.brocade.test_brocade_plugin.TestBrocadePortsV2.test_delete_network_port_exists_owned_by_network
>
> neutron.tests.unit.db.test_migration.TestModelsMigrationsSyncMl2Psql.test_models_sync
>
> This failure behavior started after
> change I6f67bb430c50ddacb2d53398de75fb5f494964a0 to use oslo.db for all of
> neutron connection handling instead of SQLAlchemy.[1] The failure message
> returned from the DB layer is very misleading. If you remove the catch that
> converts to that generic error about username/pass and DB being
> wrong/missing, you will get the real following error:
>
> OperationalError: (OperationalError) database "dzmhwmgrou" is being
> accessed by other users
> DETAIL:  There are 1 other session(s) using the database.
>  'drop database dzmhwmgrou;' {}
>
>
> What is happening is that the oslo.db test case cleanup code is trying to
> destroy the db while a separate sqlalchemy engine (created from the alembic
> migration code) still has a connection to the db. The first test is
> required to help trigger the failure either because of imports or the run
> setting up database connections causing things to be cached in a module
> flag somewhere. I haven't looked into the exact source.
>
> Here is the diff to fix your patch to pass the same session into the
> alembic migration code that is setup and torn down by the test case. This
> should allow you to proceed forward with your work.
>
>
> ~/code/neutron$ git diff
> diff --git a/neutron/tests/unit/db/test_migration.py
> b/neutron/tests/unit/db/test_migration.py
> index 6db8ae0..c29ab67 100644
> --- a/neutron/tests/unit/db/test_migration.py
> +++ b/neutron/tests/unit/db/test_migration.py
> @@ -136,9 +136,12 @@ class ModelsMigrationsSyncMixin(object):
>  self.alembic_config.neutron_config = cfg.CONF
>
>  def db_sync(self, engine):
> -cfg.CONF.set_override('connection', engine.url, group='database')
> -migration.do_alembic_command(self.alembic_config, 'upgrade',
> 'head')
> -cfg.CONF.clear_override('connection', group='database')
> +with mock.patch(
> +'oslo.db.sqlalchemy.session.create_engine',
> +return_value=self.get_engine()
> +):
> +migration.do_alembic_command(self.alembic_config,
> + 'upgrade', 'head')
>
>  def get_engine(self):
>  return self.engine
>
>
>
>
> 1. https://review.openstack.org/#/c/110016/
>
> --
> Kevin Benton
>
> On Fri, Sep 12, 2014 at 2:15 AM, Anna Kamyshnikova <
> akamyshnik...@mirantis.com> wrote:
>
>> This is implementing ModelsMigrationsSync test from oslo [1]. For running
>> it locally on Postgres you have to do the following things (it is mentioned
>> in comments to test):
>>
>> For the opportunistic testing you need to set up a db named
>> 'openstack_citest' with user 'openstack_citest' and password
>> 'openstack_citest' on localhost.
>> The test will then use that db and user/password combo to run the
>> tests.
>>
>> For PostgreSQL on Ubuntu this can be done with the following commands::
>>
>> sudo -u postgres psql
>> postgres=# create user openstack_citest with createdb login
>> password
>>   'openstack_citest';
>> postgres=# create database openstack_citest with owner
>>openstack_citest;
>>
>> For MySQL on Ubuntu this can be done with the following commands::
>>
>> mysql -u root
>> >create database openstack_citest;
>> >grant all privilleges on openstack_citest.* to
>>  openstack_citest@localhost identified by 'opensatck_citest';
>>
>> As I said this error appeared only three weeks ago, although I'm working
>> on this test since 29 of April, it passed Jenkins in August without any
>> problems. Postgres is available there.
>>
>> [1] -
>> https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/test_migrations.py#L277
>>
>> On Fri, Sep 12, 2014 at 12:28 PM, Kevin Benton  wrote:
>>
>>> Can you explain a bit about that test? I'm having trouble reproducing it.
>>> On the system (upstream Jenkins) that it's failing on, is postgres
>>> available with that database?
>>>
>>> On Thu, Sep 11, 2014 at 7:07 AM, Anna Kamyshnikova <
>>> akamyshnik...@mirantis.com> wrote:
>>>
 Hello everyone!

 I'm working on implementing test in Neutron that checks that models are
 synchronized with database state [1] [2]. This is very important change as
 during Juno cycle big changes of database structure were done.

 I was working on it for rather long time but about three weeks ago
 strange error appeared [3], using AssertionPool shows [4]. The problem is
 that somehow 

Re: [openstack-dev] [metrics] Old reviews (2011) with strange uploaded dates in review.openstack.org

[Daniel Izquierdo]

On 12/09/14 15:38, Jeremy Stanley wrote:

On 2014-09-12 11:54:19 +0200 (+0200), Daniel Izquierdo wrote:
[...]

And a question: was there a migration around the 2012-12-16 of the
review system or some other noticeable event?. On such date there
was around 1,200 submitted reviews, while around those days, in
mean, there are some dozens of them.

We discovered that Gerrit configures datestamps in most of its
tables to reset on row updates (a particularly insane design choice
for things like a "created_on" column). Before we realized this,
intrusive maintenance activities--most notably project renames--were
mass-resetting the creation dates of changes and comments to the
date and time we ran the necessary update queries. Now we
special-case those fields in our update queries to forcibly reset
them to themselves so that they retain their original values, but at
this point there's no easy way to go back and fix the ones we did
before we noticed this unfortunate loss of date/time information.
That makes totally sense, thanks a lot for the info!. Then we should try 
to avoid those reviews when calculating time to review and other 
time-based metrics.




This maintenance notification looks relevant...

http://lists.openstack.org/pipermail/openstack-dev/2012-December/003934.html
Oops, thanks for the pointer. It's exactly that date (I didn't check the 
infra mailing list for that exactly date, my fault u_u).


Thanks a lot!

Regards,
Daniel.






--
Daniel Izquierdo Cortazar, PhD
Chief Data Officer
-
"Software Analytics for your peace of mind"
www.bitergia.com
@bitergia


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Victoria Martínez de la Cruz]

2014-09-12 10:44 GMT-03:00 Gordon Sim :

> On 09/11/2014 07:46 AM, Flavio Percoco wrote:
>
>> On 09/10/2014 03:18 PM, Gordon Sim wrote:
>>
>>> On 09/10/2014 09:58 AM, Flavio Percoco wrote:
>>>
  Other OpenStack components can integrate with Zaqar to surface
 events
 to end users and to communicate with guest agents that run in the
 "over-cloud" layer.

>>>
>>> I may be misunderstanding the last sentence, but I think *direct*
>>> integration of other OpenStack services with Zaqar would be a bad idea.
>>>
>>> Wouldn't this be better done through olso.messaging's notifications in
>>> some way? and/or through some standard protocol (and there's more than
>>> one to choose from)?
>>>
>>> Communicating through a specific, fixed messaging system, with its own
>>> unique protocol is actually a step backwards in my opinion, especially
>>> for things that you want to keep as loosely coupled as possible. This is
>>> exactly why various standard protocols emerged.
>>>
>>>
>> Yes and no. The answer is yes most of the time but there are use cases,
>> like the ones mentioned here[0], that make zaqar a good tool for the job.
>>
>
> I certainly wasn't saying that Zaqar is not a good tool. I was merely
> stating that - in my opinion - wiring it in as the only tool would be a
> mistake.
>

Fair enough. Zaqar is just one of the possibilities and it's crafted to
work with OpenStack. If users prefer to use a different tool, it's totally
fine. I guess that operators will choose what it best fit their needs.


>
>  [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases
>>
>
> Again, Zaqar might be great for those cases, but none of them describe
> features that are unique to Zaqar, so other solutions could also fit.


> All I'm saying is that if the channel between openstack services and users
> is configurable, that will give users more choice (as well as operators)
> and that - in my opinion - would be a good thing.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Propose adding StevenK to core reviewers

[Jiří Stránský]

On 9.9.2014 20:32, Gregory Haynes wrote:

Hello everyone!

I have been working on a meta-review of StevenK's reviews and I would
like to propose him as a new member of our core team.


+1



As I'm sure many have noticed, he has been above our stats requirements
for several months now. More importantly, he has been reviewing a wide
breadth of topics and seems to have a strong understanding of our code
base. He also seems to be doing a great job at providing valuable
feedback and being attentive to responses on his reviews.

As such, I think he would make a great addition to our core team. Can
the other core team members please reply with your votes if you agree or
disagree.

Thanks!
Greg



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 12/09/14 16:33, Mike Bayer wrote:
> I agree with this, changing the MySQL driver now is not an option.

That was not the proposal. The proposal was to introduce support to
run against something different from MySQLdb + a gate job for that
alternative. The next cycle was supposed to do thorough regression
testing, benchmarking, etc. to decide whether we're ok to recommend
that alternative to users.

/Ihar
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUEwX2AAoJEC5aWaUY1u576sEH/j1q6elB5jmt9AxInN77ei7E
dG80fol/E56UB+rtuTfrev2ceLYU6iTF7p11t/ABzXdvGHWwcfzD/zJrPUEu0+Bq
XbyATjNTEtjgBkcZr8R1Av2JwOgrny/3OeATQf8EfqDUKhjiUcAsPrYw14OebUyZ
HRyTA7QvC83aJQK28hMK+l2x7cYCPG5CGugUXd5BTXP/yMOQ60izvHd9B9vnx/5y
EgWDV3RwAXPiFQ41aeobIktlt9F+bl6y6S+mmJY3FgjsjqxKIJBlxmhCppKLcot5
9WhsBUa9uvgCAvOU7p7/B4pSo+9gaxJtXlCjzQBH6qWb07DItMLjsc8eF6uA5M0=
=xIP3
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Mike Bayer]

On Sep 12, 2014, at 7:20 AM, Sean Dague  wrote:

> 
> Because we are in Feature Freeze. Now is the time for critical bug fixes
> only, as we start to stabalize the tree. Releasing dependent libraries
> that can cause breaks, for whatever reason, should be soundly avoided.
> 
> If this was August, fine. But it's feature freeze.

I agree with this, changing the MySQL driver now is not an option.That 
train has left the station, I think it’s better we all take the whole Kilo 
cycle to get used to mysql-connector and its quirks before launching it on the 
world, as there will be many more.

However for Kilo, I think those “COMMIT” phrases should be removed and overall 
we need to make a very hard and fast rule that we *do not put multiple 
statements in an execute*.I’ve seen a bunch of these come through so far, 
and for some of them (more the in-Python ones) it seems like the underlying 
reason is a lack of understanding of what exactly a SQLAlchemy “Engine” is and 
what features it supports.

So first, let me point folks to the documentation for this, which anyone 
writing code involving Engine objects should read first:

http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html

Key to this is that while engine supports an “.execute()” method, in order to 
do anything that intends to work on a single connection and typically a single 
transaction, you procure a Connection and usually a Transaction from the 
Engine, most easily like this:

with engine.begin() as conn:
   conn.execute(statement 1)
   conn.execute(statement 2)
   conn.execute(statement 3)
   .. etc


Now let me apologize for the reason this misunderstanding exists in the first 
place:  it’s because in 2005 I put the “.execute()” convenience method on the 
Engine itself (well in fact we didn’t have the Engine/Connection dichotomy back 
then), and I also thought that “implicit execution”, e.g. statement.execute(), 
would be a great idea.Tons of other people still think it’s a great idea 
and even though I’ve buried this whole thing in the docs, they still use it 
like candy….until they have the need to control the scope of connectivity.  

*Huge* mistake, it’s my fault, but not something that can really be changed 
now.   Also, in 2005, Python didn’t have context managers.So we have all 
kinds of klunky patterns like “trans = conn.begin()”, kind of J2EE style, etc., 
but these days, the above pattern is your best bet when you want to invoke 
multiple statements.engine.execute() overall should just be avoided as it 
only leads to misunderstanding.   When we all move all of our migrate stuff to 
Alembic, there won’t be an Engine provided to a migration script, it will be a 
Connection to start with.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [solum] Consistency of development environment

[Paul Czarkowski]

>
>While I installed dnsmasq 2.63 manually, they used Ubuntu 14.04 to get
>around the problem. Is it maybe time to upgrade the base for the dev env
>to 14.04, or would that cause many other problems? Have you tried that
>out? If you have a pointer to an appropriate 14.04 image I can configure
>in Vagrant, I'd like to play with that a bit. Maybe that would also
>solve the problem with openvswitch.


We have actually recently upgraded our Vagrant environment to 14.04 so if
you pull from master from  https://github.com/rackerlabs/vagrant-solum-dev
you should get a working 14.04 instance.

We couldn¹t upgrade to 14.04 as quickly as we hoped as there is a bug that
we had to resolve with ubuntu system packages -
https://bugs.launchpad.net/solum/+bug/1365679


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Brad Topol]

+1!!! This is awesome.  I *always* ran into this was about to get   find . 
-name "*.pyc" -delete tattooed on the inside of my forearm. Now I don't 
have to.  Thanks!!!

--Brad



Brad Topol, Ph.D.
IBM Distinguished Engineer
OpenStack
(919) 543-0646
Internet:  bto...@us.ibm.com
Assistant: Kendra Witherspoon (919) 254-0680



From:   Sean Dague 
To: "openstack-dev@lists.openstack.org" 
, 
Date:   09/12/2014 07:40 AM
Subject:[openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in 
tox.ini



I assume you, gentle OpenStack developers, often find yourself in a hair
tearing out moment of frustration about why local unit tests are doing
completely insane things. The code that it is stack tracing on is no
where to be found, and yet it fails.

And then you realize that part of oslo doesn't exist any more
except there are still pyc files laying around. Gah!

I've proposed the following to Nova and Python novaclient -
https://review.openstack.org/#/c/121044/

Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.

This prevents pyc files from being writen in your git tree (win!). It
doesn't seem to impact what pip installs... and if anyone knows how to
prevent those pyc files from getting created, that would be great.

But it's something which hopefully causes less perceived developer
fragility of the system.

 -Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Eoghan Glynn]

> I visited the Paris Design Summit space on Monday and confirmed that it
> should be possible to split it in a way that would allow to have
> per-program "contributors meetups" on the Friday. The schedule would go
> as follows:
> 
> Tuesday: cross-project workshops
> Wednesday, Thursday: traditional "scheduled" slots
> Friday: contributors meetups
> 
> We'll also have "pods" available all 4 days for more ad-hoc small meetings.

Excellent :)

> In the mean time, we need to discuss how we want to handle the selection
> of session topics.
> 
> In past summits we used a Design-Summit-specific "session suggestion"
> website, and PTLs would approve/deny them. This setup grew less and less
> useful: session topics were selected collaboratively on etherpads,
> discussed in meetings, and finally filed/reorganized/merged on the
> website just before scheduling. Furthermore, with even less "scheduled"
> slots, we would have to reject most of the suggestions, which is more
> frustrating for submitters than the positive experience of joining team
> meetings to discuss which topics are the most important. Finally, topics
> will need to be split between "scheduled" sessions and the "contributors
> meetup" agenda, and that's easier to do on an Etherpad anyway.
> 
> This is why I'd like to suggest that all programs use etherpads to
> collect important topics, select which ones would get in the very few
> "scheduled" slots we'll have left, which will get discussed in the
> "contributors meetup", and which are better left for a "pod" discussion.
> I suggest we all use IRC team meetings to collaboratively discuss that
> content between interested contributors.
> 
> To simplify the communication around this, I tried to collect the
> already-announced etherpads on a single page at:
> 
> https://wiki.openstack.org/wiki/Summit/Planning
> 
> Please add any that I missed !
> 
> If you think this is wrong and think the "design summit suggestion"
> website is a better way to do it, let me know why! If some programs
> really can't stand the 'etherpad/IRC' approach I'll see how we can spin
> up a limited instance.

+1 on a collaborative scheduling process within each project.

That's pretty much what we did within the ceilometer core group for
the Juno summit, except that we used a googledocs spreadsheet instead
of an etherpad.

So I don't think we need to necessarily mandate usage of an etherpad,
just let every project decide whatever shared document format they
want to use.

FTR the benefit of a googledocs spreadsheet in my view would include
the ease of totalling votes & sessions slots, color-coding candidate
sessions for merging etc.

Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [solum] Consistency of development environment

[Alexander Vollschwitz]

Hello Adrian,

thanks for the quick reply, and sorry for my delayed response.

On 09/03/2014 04:42 PM, Adrian Otto wrote:
> We have noticed lately that our devstack setup does not always work.
>  [...] We have discussed ways to mitigate this. One idea was to
> select a particular devstack from a prior OpenStack release to help
> cut down on the rate of change. [...] We also considered additional
> functional tests for devstack to run when new code is submitted. I
> suppose we could run testing continuously in loops in attempts to
> detect non-determinism. [...] All of the above are opportunities for
> us to improve matters going forward. There are probably even better
> ideas we should consider as well.

I'm currently toying with the following approach: Clone all necessary
repos locally (staging repos), and configure devstack to use them (via
GIT_BASE). At fixed intervals, automatically update the staging repos,
and start the provisioning of a dev env. If that goes well, run the
Solum tests with it. If that is also successful, the state in the
staging repos gets pulled into a second set of local repos (setup
repos), which I can use for actual dev env provisioning. If things fail,
we could give it a couple of retries (for the non-determinism you
mentioned), or just wait for the next interval.

So there should (hopefully) always be a fairly recent and usable state
in the setup repos. How well this works will mostly depend on the
interval, I guess. I.e., the shorter the interval, the higher the chance
for filtering out a usable state. Of course, if things get broken
permanently, the state in the setup repos would no longer advance and
we'd need to look into the cause and take actions (see example below).

This approach doesn't really solve the root cause, but could make
setting up a dev env a bit more reliable. I got the first part in place,
i.e. provisioning from staging repos. I'll now experiment with the
second part, and let you know.


> For now, we would like to help you get past the friction you are
> experiencing so you can get a working environment up.

I could resolve the two problems I mentioned manually and get the dev
env working. However, this brings up a question:

>> after devstack provisioned OS, q-dhcp and q-l3 were not running.
>> The former refused to start due to an updated version requirement
>> for dnsmasq (see
>> https://bugs.launchpad.net/openstack-manuals/+bug/1347153) that
>> was not met

This problem is also described here:
http://netapp.github.io/openstack/2014/08/15/manila-devstack/

While I installed dnsmasq 2.63 manually, they used Ubuntu 14.04 to get
around the problem. Is it maybe time to upgrade the base for the dev env
to 14.04, or would that cause many other problems? Have you tried that
out? If you have a pointer to an appropriate 14.04 image I can configure
in Vagrant, I'd like to play with that a bit. Maybe that would also
solve the problem with openvswitch.

Kind regards,

Alex
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Jeremy Stanley]

On 2014-09-12 12:41:42 +0200 (+0200), Ihar Hrachyshka wrote:
[...]
> That said, those failures were solved by pinning the version of the
> library in openstack/requirements and those projects. This is in major
> contrast to how we handled the new testtools release just several
> weeks ago, when the problem was solved by fixing three affected
> projects because of their incorrect usage of tearDown/setUp methods.
[...]

This was of course different because it came during a period where
integrated projects are supposed to be focusing on stabilizing what
they have toward release, but also our behavior was somewhat altered
because we needed to perform some immediate damage control.

One of the side-effects of the failure mode this sqlalchemy-migrate
release induced was that each nova unit tests was generating ~0.5GiB
of log data, instantly overwhelming our test log analysis systems
and flooding our artifact archive (both in terms of bandwidth and
disk). The fastest way to stop this was to "roll back" what changed,
for which the options were either to introduce an exclusionary
version pin or convince the library authors to release an even newer
version tagged to the old one. We chose the first solution as it was
more directly under the control of the infrastructure and nova core
teams involved at that moment.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Gordon Sim]

On 09/11/2014 07:46 AM, Flavio Percoco wrote:

On 09/10/2014 03:18 PM, Gordon Sim wrote:

On 09/10/2014 09:58 AM, Flavio Percoco wrote:

 Other OpenStack components can integrate with Zaqar to surface events
to end users and to communicate with guest agents that run in the
"over-cloud" layer.


I may be misunderstanding the last sentence, but I think *direct*
integration of other OpenStack services with Zaqar would be a bad idea.

Wouldn't this be better done through olso.messaging's notifications in
some way? and/or through some standard protocol (and there's more than
one to choose from)?

Communicating through a specific, fixed messaging system, with its own
unique protocol is actually a step backwards in my opinion, especially
for things that you want to keep as loosely coupled as possible. This is
exactly why various standard protocols emerged.



Yes and no. The answer is yes most of the time but there are use cases,
like the ones mentioned here[0], that make zaqar a good tool for the job.


I certainly wasn't saying that Zaqar is not a good tool. I was merely 
stating that - in my opinion - wiring it in as the only tool would be a 
mistake.



[0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases


Again, Zaqar might be great for those cases, but none of them describe 
features that are unique to Zaqar, so other solutions could also fit.


All I'm saying is that if the channel between openstack services and 
users is configurable, that will give users more choice (as well as 
operators) and that - in my opinion - would be a good thing.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

On 09/12/2014 09:50 AM, Flavio Percoco wrote:

Zaqar supports once and only once delivery.


For the transfer from Zaqar to consumers it does (providing the claim id 
can be recovered). For transfer from producers to Zaqar I believe it is 
more limited.


If the connection to Zaqar fails during a post, the sender can't tell 
whether the message was successfully enqueued or not.


It could try to determine this is by browsing the entire queue looking 
for a matching body. However thatt would be awkward and in any case the 
absence of the message could mean that it wasn't enqueued or that it was 
already consumed and deleted.


One way of handling this is to have the post return a unique url to 
which the message(s) are put or posted. The sender can then repost 
(re-put) to this in the event of failure and the server can determine 
whether it already processed the publications. Alternatively the client 
can be required to generate a unique id on which the server can 
de-deduplicate.


The ActiveMQ REST interface supports both of these approaches: 
http://activemq.apache.org/restful-queue.html


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][neutron] tox -e py27 is not working in the latest neutron code

[Jeremy Stanley]

On 2014-09-12 10:02:49 + (+), Kelam, Koteswara Rao wrote:
> I am trying to run unit test cases in neutron code using “tox –e
> py27” but it is not working.
[...]
> 'TOX_INDEX_URL': 'http://pypi.openstack.org/openstack',
[...]
> 'PIP_INDEX_URL': 'http://pypi.openstack.org/openstack',
[...]
> Could not find any downloads that satisfy the requirement Paste
[...]

You have your environment misconfigured to use a mirror of PyPI
which is no longer maintained. Please use pypi.python.org or a
mirror you maintain for your own development work.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Tempest Bug triage

[David Kranz]

On 09/12/2014 05:11 AM, Kashyap Chamarthy wrote:

On Thu, Sep 11, 2014 at 03:52:56PM -0400, David Kranz wrote:

So we had a Bug Day this week and the results were a bit disappointing due
to lack of participation. We went from 124 New bugs to 75.

There were also many cases where bugs referred to logs that no longer
existed. This suggests that we really need to keep up with bug triage
in real time.

Alternatively, strongly recommend people to post *contextual* logs to
the bug, so they're there for reference forever and makes life less
painful while triaging bugs. Many times bugs are just filed in a hurry,
posting a quick bunch of logstash URLs which expires sooner or later.

Sure, posting contextual logs takes time, but as you can well imagine,
it results in higher quality reports (hopefully), and saves time for
others who have to take a fresh look at the bug and have to begin with
the maze of logs.
This would be "in addition to", not alternatively. Of course better bug 
reports with as much information as possible, with understanding of how 
long log files will be retained, etc. would always be better. But due to 
the sorry state we are now in, it is simply unrealistic to expect people 
to start investigating failures in code they do not understand that are 
obviously unrelated to the code they are trying to babysit through the 
gate. I wish it were otherwise, and believe this may change as  we 
achieve the goal of focusing our test time on tests that are related to 
the code being tested (in-project functional testing).


The purpose of rotating bug triage is that it was not happening at all. 
When there is a not-so-much-fun task for which every one is responsible, 
no one is responsible. It is better to share the load in a well 
understood way and know who has taken on responsibility at any point in 
time.


 -David


--
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [metrics] Old reviews (2011) with strange uploaded dates in review.openstack.org

[Jeremy Stanley]

On 2014-09-12 11:54:19 +0200 (+0200), Daniel Izquierdo wrote:
[...]
> And a question: was there a migration around the 2012-12-16 of the
> review system or some other noticeable event?. On such date there
> was around 1,200 submitted reviews, while around those days, in
> mean, there are some dozens of them.

We discovered that Gerrit configures datestamps in most of its
tables to reset on row updates (a particularly insane design choice
for things like a "created_on" column). Before we realized this,
intrusive maintenance activities--most notably project renames--were
mass-resetting the creation dates of changes and comments to the
date and time we ran the necessary update queries. Now we
special-case those fields in our update queries to forcibly reset
them to themselves so that they retain their original values, but at
this point there's no easy way to go back and fix the ones we did
before we noticed this unfortunate loss of date/time information.

This maintenance notification looks relevant...

http://lists.openstack.org/pipermail/openstack-dev/2012-December/003934.html

-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 12/09/14 13:20, Sean Dague wrote:
> On 09/12/2014 06:41 AM, Ihar Hrachyshka wrote:
>> Some updates/concerns/questions.
>> 
>> The status of introducing a new driver to gate is:
>> 
>> - all the patches for mysql-connector are merged in all
>> projects; - all devstack patches to support switching the driver
>> are merged; - new sqlalchemy-migrate library is released;
>> 
>> - version bump is *not* yet done; - package is still *not* yet
>> published on pypi; - new gate job is *not* yet introduced.
>> 
>> The new sqlalchemy-migrate release introduced unit test failures
>> in those three projects: nova, cinder, glance.
>> 
>> On technical side of the failure: my understanding is that those 
>> projects that started to fail assume too much about how those
>> SQL scripts are executed. They assume they are executed in one
>> go, they also assume they need to open and commit transaction on
>> their own. I don't think this is something to be fixed in
>> sqlalchemy-migrate itself. Instead, simple removal of those
>> 'BEGIN TRANSACTION; ... COMMIT;' statements should just work and
>> looks like a sane thing to do anyway. I've proposed the following
>> patches for all three projects to handle it [1].
>> 
>> That said, those failures were solved by pinning the version of
>> the library in openstack/requirements and those projects. This is
>> in major contrast to how we handled the new testtools release
>> just several weeks ago, when the problem was solved by fixing
>> three affected projects because of their incorrect usage of
>> tearDown/setUp methods.
>> 
>> Even more so, those failures seem to trigger the resolution to
>> move the enable-mysql-connector oslo spec to Kilo, while the
>> library version bump is the *only* change missing codewise (we
>> will also need a gate job description, but that doesn't touch
>> codebase at all). The resolution looks too prompt and ungrounded
>> to me. Is it really that gate failure for three projects that
>> resulted in it, or there are some other hidden reasons behind it?
>> Was it discussed anywhere? If so, I wasn't given a chance to
>> participate in that discussion; I suspect another supporter of
>> the spec (Agnus Lees) was not involved either.
>> 
>> Not allowing those last pieces of the spec in this cycle, we
>> just postpone start of any realistic testing of the feature for
>> another half a year.
>> 
>> Why do we block new sqlalchemy-migrate and the spec for another
>> cycle instead of fixing the affected projects with *primitive*
>> patches like we did for new testtools?
> 
> Because we are in Feature Freeze. Now is the time for critical bug
> fixes only, as we start to stabalize the tree. Releasing dependent
> libraries that can cause breaks, for whatever reason, should be
> soundly avoided.
> 
> If this was August, fine. But it's feature freeze.

I probably missed the fact that we are so strict now that we don't
allow tiny missing bits to go in. In my excuse, I was offline for
around three last weeks. I was a bit misled by the fact that I was
approached by an oslo core very recently on which remaining bits we
need to push before claiming the spec to be complete, and I assumed it
means that we are free to complete the work this cycle. Otherwise, I
wouldn't push for the new library version in the first place.

Anyway, I guess there is no way now to get remaining bits in Juno,
even if small, and we're doomed to postpone them to Kilo.

Thanks for the explanation,
/Ihar
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUEvPjAAoJEC5aWaUY1u57kPYIAMuTz5w8cmNLeXHSGpb0s0BT
4GPbTvLIvoTRXf2froozSxVo6B4oKgUFe7IkSI8nsBHP+dcDPotKwJEMgAKpLL1n
37ccFR+RuMCVMa6ZYHgz88o4dbTgv5XC5tBTnY78mX7WOoQHQ0ByRcBUZkIc9aoI
KF+SNRvHwVRT9qNPElcrfHKNPwROIe1Eml3aVaqnHWPWip5J7+E+/BU+YSxtDKIV
whrJzUpHgwph4NJ1lHddrzVCAjf8mWKj8EX1WWU2zTgUtfLi+xqvOBCnQ+1rBXA8
brIBpbUOObMjBqbemlymKuFvcuy6yHTXXvAfLcgGcRXSmvdjtfAIZCr5d9AjKhU=
=zPHu
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Zane Bitter]

On 12/09/14 07:37, Thierry Carrez wrote:

Hi everyone,

I visited the Paris Design Summit space on Monday and confirmed that it
should be possible to split it in a way that would allow to have
per-program "contributors meetups" on the Friday. The schedule would go
as follows:

Tuesday: cross-project workshops
Wednesday, Thursday: traditional "scheduled" slots
Friday: contributors meetups

We'll also have "pods" available all 4 days for more ad-hoc small meetings.

In the mean time, we need to discuss how we want to handle the selection
of session topics.

In past summits we used a Design-Summit-specific "session suggestion"
website, and PTLs would approve/deny them. This setup grew less and less
useful: session topics were selected collaboratively on etherpads,
discussed in meetings, and finally filed/reorganized/merged on the
website just before scheduling. Furthermore, with even less "scheduled"
slots, we would have to reject most of the suggestions, which is more
frustrating for submitters than the positive experience of joining team
meetings to discuss which topics are the most important. Finally, topics
will need to be split between "scheduled" sessions and the "contributors
meetup" agenda, and that's easier to do on an Etherpad anyway.

This is why I'd like to suggest that all programs use etherpads to
collect important topics, select which ones would get in the very few
"scheduled" slots we'll have left, which will get discussed in the
"contributors meetup", and which are better left for a "pod" discussion.
I suggest we all use IRC team meetings to collaboratively discuss that
content between interested contributors.


+1, this was #1 or #2 on my list of "things where the PTL becomes a 
single point of failure".


- ZB


To simplify the communication around this, I tried to collect the
already-announced etherpads on a single page at:

https://wiki.openstack.org/wiki/Summit/Planning

Please add any that I missed !

If you think this is wrong and think the "design summit suggestion"
website is a better way to do it, let me know why! If some programs
really can't stand the 'etherpad/IRC' approach I'll see how we can spin
up a limited instance.

Regards,




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][db] Need help resolving a strange error with db connections in tests

[Kevin Benton]

This one was tricky. :-)

This failure can be produced consistently by running the following two
tests:
neutron.tests.unit.brocade.test_brocade_plugin.TestBrocadePortsV2.test_delete_network_port_exists_owned_by_network
neutron.tests.unit.db.test_migration.TestModelsMigrationsSyncMl2Psql.test_models_sync

This failure behavior started after
change I6f67bb430c50ddacb2d53398de75fb5f494964a0 to use oslo.db for all of
neutron connection handling instead of SQLAlchemy.[1] The failure message
returned from the DB layer is very misleading. If you remove the catch that
converts to that generic error about username/pass and DB being
wrong/missing, you will get the real following error:

OperationalError: (OperationalError) database "dzmhwmgrou" is being
accessed by other users
DETAIL:  There are 1 other session(s) using the database.
 'drop database dzmhwmgrou;' {}


What is happening is that the oslo.db test case cleanup code is trying to
destroy the db while a separate sqlalchemy engine (created from the alembic
migration code) still has a connection to the db. The first test is
required to help trigger the failure either because of imports or the run
setting up database connections causing things to be cached in a module
flag somewhere. I haven't looked into the exact source.

Here is the diff to fix your patch to pass the same session into the
alembic migration code that is setup and torn down by the test case. This
should allow you to proceed forward with your work.


~/code/neutron$ git diff
diff --git a/neutron/tests/unit/db/test_migration.py
b/neutron/tests/unit/db/test_migration.py
index 6db8ae0..c29ab67 100644
--- a/neutron/tests/unit/db/test_migration.py
+++ b/neutron/tests/unit/db/test_migration.py
@@ -136,9 +136,12 @@ class ModelsMigrationsSyncMixin(object):
 self.alembic_config.neutron_config = cfg.CONF

 def db_sync(self, engine):
-cfg.CONF.set_override('connection', engine.url, group='database')
-migration.do_alembic_command(self.alembic_config, 'upgrade',
'head')
-cfg.CONF.clear_override('connection', group='database')
+with mock.patch(
+'oslo.db.sqlalchemy.session.create_engine',
+return_value=self.get_engine()
+):
+migration.do_alembic_command(self.alembic_config,
+ 'upgrade', 'head')

 def get_engine(self):
 return self.engine




1. https://review.openstack.org/#/c/110016/

--
Kevin Benton

On Fri, Sep 12, 2014 at 2:15 AM, Anna Kamyshnikova <
akamyshnik...@mirantis.com> wrote:

> This is implementing ModelsMigrationsSync test from oslo [1]. For running
> it locally on Postgres you have to do the following things (it is mentioned
> in comments to test):
>
> For the opportunistic testing you need to set up a db named
> 'openstack_citest' with user 'openstack_citest' and password
> 'openstack_citest' on localhost.
> The test will then use that db and user/password combo to run the
> tests.
>
> For PostgreSQL on Ubuntu this can be done with the following commands::
>
> sudo -u postgres psql
> postgres=# create user openstack_citest with createdb login
> password
>   'openstack_citest';
> postgres=# create database openstack_citest with owner
>openstack_citest;
>
> For MySQL on Ubuntu this can be done with the following commands::
>
> mysql -u root
> >create database openstack_citest;
> >grant all privilleges on openstack_citest.* to
>  openstack_citest@localhost identified by 'opensatck_citest';
>
> As I said this error appeared only three weeks ago, although I'm working
> on this test since 29 of April, it passed Jenkins in August without any
> problems. Postgres is available there.
>
> [1] -
> https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/test_migrations.py#L277
>
> On Fri, Sep 12, 2014 at 12:28 PM, Kevin Benton  wrote:
>
>> Can you explain a bit about that test? I'm having trouble reproducing it.
>> On the system (upstream Jenkins) that it's failing on, is postgres
>> available with that database?
>>
>> On Thu, Sep 11, 2014 at 7:07 AM, Anna Kamyshnikova <
>> akamyshnik...@mirantis.com> wrote:
>>
>>> Hello everyone!
>>>
>>> I'm working on implementing test in Neutron that checks that models are
>>> synchronized with database state [1] [2]. This is very important change as
>>> during Juno cycle big changes of database structure were done.
>>>
>>> I was working on it for rather long time but about three weeks ago
>>> strange error appeared [3], using AssertionPool shows [4]. The problem is
>>> that somehow there are more than one connection to database from each test.
>>> I tried to use locks from lockutils, but it didn’t help. On db meeting we
>>> decided to add TestCase just for one Ml2 plugin for starters, and then
>>> continue working on this strange error, that is why there are two change
>>> requests [1] and [2]. But I found out tha

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Zane Bitter]

On 12/09/14 04:50, Flavio Percoco wrote:

On 09/12/2014 12:14 AM, Zane Bitter wrote:

>However, Zaqar also supports the Pub-Sub model of messaging. I believe,
>but would like Flavio to confirm, that this is what is meant when the
>Zaqar team say that Zaqar is about messaging in general and not just
>queuing. That is to say, it is possible for multiple consumers to
>intentionally consume the same message, with each maintaining its own
>pointer in the queue. (Another way to think of this is that messages can
>be multicast to multiple virtual queues, with data de-duplication
>between them.) To a relative novice in the field like me, the difference
>between this and queuing sounds pretty academic :P. Call it what you
>will, it seems like a reasonable thing to implement to me.


Correct, this and other messaging patterns supported by Zaqar make it a
messaging service, which as Gordon mentioned in another email is just a
more generic term. Messages are the most important resource in Zaqar and
providing good, common and scalable patterns to access those messages is
what we strive for in Zaqar's API.


Thanks Flavio! I think we are more or less on the same page :)

Maybe you could clarify what the "other messaging patterns" are exactly, 
since that seems to be one of the points of confusion/contention.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Zane Bitter]

On 11/09/14 19:05, Jay Pipes wrote:

On 09/11/2014 04:09 PM, Zane Bitter wrote:

Swift is the current exception here, but one could argue, and people
have[2], that Swift is also the only project that actually conforms to
our stated design tenets for OpenStack. I'd struggle to tell the Zaqar
folks they've done the Wrong Thing... especially when abandoning the
RDBMS driver was done largely at the direction of the TC iirc.




[2] http://blog.linux2go.dk/2013/08/30/openstack-design-tenets-part-2/


No offense to Soren, who wrote some interesting and poignant things, nor
to the Swift developers, who continue to produce excellent work, but
Swift is object storage. It is a data plane system with a small API
surface, a very limited functional domain, and a small, inflexible
storage schema (which is perfectly fine for its use cases). It's needs
for a relational database are nearly non-existent. It replicates a
SQLite database around using rsync [1]. Try doing that with a schema of
any complexity and you will quickly find the limitations of such a
strategy.

If Nova was to take Soren's advice and implement its data-access layer
on top of Cassandra or Riak, we would just end up re-inventing SQL Joins
in Python-land. I've said it before, and I'll say it again. In Nova at
least, the SQL schema is complex because the problem domain is complex.
That means lots of relations, lots of JOINs, and that means the best way
to query for that data is via an RDBMS.

And I say that knowing just how *poor* some of the queries are in Nova!


I wasn't trying to suggest that Nova should change (if there was another 
project I had in mind while reading that it would have been Heat, not 
Nova). My point was that it's understandable that Zaqar, which is *also* 
a data-plane service with a small API surface and a limited functional 
domain, doesn't have the same architecture as Nova (just as Swift 
doesn't) and that it's probably counter-productive to force it into that 
architecture purely because a bunch of other things use it.



For projects like Swift, Zaqar, even Keystone, Glance and Cinder, a
non-RDBMS solution might be a perfectly reasonable solution for the
underlying data storage and access layer (and for the record, I never
said that Zaqar should or should not use an RDBMS for its storage). For
complex control plane software like Nova, though, an RDBMS is the best
tool for the job given the current lay of the land in open source data
storage solutions matched with Nova's complex query and transactional
requirements.


+1


Folks in these other programs have actually, you know, thought about
these kinds of things and had serious discussions about alternatives. It
would be nice to have someone acknowledge that instead of snarky
comments implying everyone else "has it wrong".


I didn't mean to imply that anybody else has it wrong (although FWIW I 
do think that Heat probably has it wrong), and I apologise to anyone who 
interpreted it that way.



Going back in my hole,
-jay


No! Let's talk about Zaqar :)

cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Server Groups - remove VM from group?

[Russell Bryant]

On 09/11/2014 05:01 PM, Jay Pipes wrote:
> On 09/11/2014 04:51 PM, Matt Riedemann wrote:
>> On 9/10/2014 6:00 PM, Russell Bryant wrote:
>>> On 09/10/2014 06:46 PM, Joe Cropper wrote:
 Hmm, not sure I follow the concern, Russell.  How is that any different
 from putting a VM into the group when it’s booted as is done today?
   This simply defers the ‘group insertion time’ to some time after
 initial the VM’s been spawned, so I’m not sure this creates anymore
 race
 conditions than what’s already there [1].

 [1] Sure, the to-be-added VM could be in the midst of a migration or
 something, but that would be pretty simple to check make sure its task
 state is None or some such.
>>>
>>> The way this works at boot is already a nasty hack.  It does policy
>>> checking in the scheduler, and then has to re-do some policy checking at
>>> launch time on the compute node.  I'm afraid of making this any worse.
>>> In any case, it's probably better to discuss this in the context of a
>>> more detailed design proposal.
>>>
>>
>> This [1] is the hack you're referring to right?
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py?id=2014.2.b3#n1297
>>
> 
> That's the hack *I* had in the back of my mind.

Yep.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Infra][Cinder] Coraid CI system

[Roman Bogorodskiy]

Hi,

Mykola has some problems sending emails to the list, so he asked me to post
a response, here it goes:

---
Remy, I have improved Coraid CI system and added logs of all components of
devstack. Please have a look:

http://38.111.159.9:8080/job/Coraid_CI/164/

According to the requirements from
http://ci.openstack.org/third_party.html#requesting-a-service-account ,
Gerrit plugin from Jenkins should be given the following options:

Successful: gerrit approve , --message 'Build
Successful ' --verified  --code-review

Failed: gerrit approve , --message 'Build Failed
' --verified  --code-review 
Unstable: gerrit approve , --message 'Build Unstable
' --verified  --code-review 

I configured gerrit plugin this way, so it sends the following comment
after checking patchset or comment with "recheck". For example,
https://review.openstack.org/#/c/120907/

Patch Set 1:

Build Successful

http://38.111.159.9:8080/job/Coraid_CI/164/ : SUCCESS


All logs are on this page. They are there as artifacts.

> I took a quick look and I don’t see which test cases are being run?
We test Coraid Cinder driver with standard tempest tests using
./driver_certs/cinder_driver_cert.sh script. Test cases are in the log of
job.

Please look at Coraid third-party system one more time and, please, show us
what we have to add or improve in order to get voting rights for gerrit
user coraid-ci.

Also I have set gerrit plugin on our Jenkins to the silent mode as you
suggested.

Thank you in advance.


On Fri, Sep 5, 2014 at 7:34 PM, Asselin, Ramy  wrote:

> -1 from me (non-cinder core)
>
> It very nice to see you're making progress. I, personally, was very
> confused about voting.
> Here's my understanding: "Voting": it is the ability to provide an
> official +1 -1 vote in the gerrit system.
>
> I don't see a "stable history" [1]. Before requesting voting, you should
> enable your system on the cinder project itself.
> Initially, you should disable ALL gerrit comments, i.e. run in silent
> mode, per request from cinder PTL [2]. Once stable there, you can enable
> gerrit comments. At this point, everyone can see pass/fail comments with a
> vote=0.
> Once stable there on real patches, you can request voting again, where the
> pass/fail would vote +1/-1.
>
> Ramy
> [1] http://38.111.159.9:8080/job/Coraid_CI/35/console
> [2]
> http://lists.openstack.org/pipermail/openstack-dev/2014-August/043876.html
>
>
> -Original Message-
> From: Duncan Thomas [mailto:duncan.tho...@gmail.com]
> Sent: Friday, September 05, 2014 7:55 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Infra][Cinder] Coraid CI system
>
> +1 from me (Cinder core)
>
> On 5 September 2014 15:09, Mykola Grygoriev 
> wrote:
> > Hi,
> >
> > My name is Mykola Grygoriev and I'm engineer who currently working on
> > deploying 3d party CI for Сoraid Сinder driver.
> >
> > Following instructions on
> >
> > http://ci.openstack.org/third_party.html#requesting-a-service-account
> >
> > asking for adding gerrit CI account (coraid-ci) to the Voting
> > Third-Party CI Gerrit group.
> >
> >
> >
> > We have already added description of Coraid CI system to wiki page -
> > https://wiki.openstack.org/wiki/ThirdPartySystems/Coraid_CI
> >
> > We used openstack-dev/sandbox project to test current CI
> > infrastructure with OpenStack Gerrit system. Please find our history
> there.
> >
> > Please have a look to results of Coraid CI system. it currently takes
> > updates from openstack/cinder project:
> > http://38.111.159.9:8080/job/Coraid_CI/32/
> > http://38.111.159.9:8080/job/Coraid_CI/33/
> >
> > Thank you in advance.
> >
> > --
> > Best regards,
> > Mykola Grygoriev
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Duncan Thomas
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Flavio Percoco]

On 09/12/2014 11:36 AM, Thierry Carrez wrote:
> Flavio Percoco wrote:
>> On 09/12/2014 12:14 AM, Zane Bitter wrote:
>>> The final question is the one of arbitrary access to messages in the
>>> queue (or "queue" if you prefer). Flavio indicated that this effectively
>>> came for free with their implementation of Pub-Sub. IMHO it is
>>> unnecessary and limits the choice of potential back ends in the future.
>>> I would personally be +1 on removing it from the v2 API, and also +1 on
>>> the v2 API shipping in Kilo so that as few new adopters as possible get
>>> stuck with the limited choices of back-end. I hope that would resolve
>>> Clint's concerns that we need a separate, light-weight queue system; I
>>> personally don't believe we need two projects, even though I agree that
>>> all of the use cases I personally care about could probably be satisfied
>>> without Pub-Sub.
>>
>> Right, being able to support other backends is one of the reasons we're
>> looking forward to remove the support for arbitrary access to messages.
>> As of now, the plan is to remove that endpoint unless a very good use
>> case comes up that makes supporting other backends not worth it, which I
>> doubt. The feedback from Zaqar's early adopters is that the endpoint is
>> indeed not useful.
> 
> Thanks Zane, that was indeed useful. I agree with you it would be better
> to avoid needing 2 separate projects for such close use cases.

+1

> Let's assume we remove arbitrary access to messages in v2. When you say
> it would remove limits on the choice of potential backends, does that
> mean we could have a pure queue backend (like RabbitMQ), at least in
> theory ? Would a ZaqarV2 address all of Clint and Devananda's concerns
> about queue semantics ? If yes, then the graduation question becomes,
> how likely is that work to be completed early enough in Kilo.
> 
> If it's a no-brainer and takes a week to sort out, I think we could
> approve Zaqar's Kilo graduation, even if that stretches the "no major
> API rewrite planned" requirement.

Let me break the above down into several points so we can discuss them
separately:

- Removing that endpoint won't take more than a week. It's an API change
and it won't affect the existing storage drivers.

- Removing that endpoint will certainly make the adoption of other
messaging technologies easier but there are other things to consider
besides that specific endpoint (some of them were stated here[0]). In
any case, removing the endpoint definitely makes it easier.

- Besides the random access to messages, I'm not clear what other
concerns there are with regards the current semantics. It'd be nice if
we could recollect them in this section and discuss them. I took a look
at the other emails in this thread and it seems to me that the concerns
that have been raised are more oriented to the project scope and
use-cases. I also looked at the meeting logs again[1] and the only
concern related to the semantics I found is about the
`get-message-by-id` endpoint. Please, correct me if I'm wrong.


[0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/
[1]
http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-09-20.01.log.html

Flavio

> But if we think this needs careful discussion so that the v2 API design
> (and backend support) satisfies the widest set of users, then incubating
> for another cycle while v2 is implemented seems like the right course of
> action. We shouldn't graduate if there is any risk we would end up with
> ZaqarV1 in Kilo, and then have to deprecate it for n cycles just because
> it was shipped in the official release and therefore inherits its API
> deprecation rules.
> 
> Regards,
> 


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

[Sean Dague]

I assume you, gentle OpenStack developers, often find yourself in a hair
tearing out moment of frustration about why local unit tests are doing
completely insane things. The code that it is stack tracing on is no
where to be found, and yet it fails.

And then you realize that part of oslo doesn't exist any more
except there are still pyc files laying around. Gah!

I've proposed the following to Nova and Python novaclient -
https://review.openstack.org/#/c/121044/

Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests.

This prevents pyc files from being writen in your git tree (win!). It
doesn't seem to impact what pip installs... and if anyone knows how to
prevent those pyc files from getting created, that would be great.

But it's something which hopefully causes less perceived developer
fragility of the system.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev