Re: [openstack-dev] [neutron] DVR Tunnel Design Question

2014-09-18 Thread Narasimhan, Vivekanandan
Hi Kevin,



Typically we noticed that the underlay switches maintained a table like this:



VLAN IDMAC Address  Learned-Interface



In the physical underlay, with the current architecture if we enable VLAN, the 
same DVR Unique

MAC will appear  on different VLANs as the packets get DVR Routed.



This will result in the rows of the above tables in the switch to be updated 
very frequently with new

VLANs noted in incoming packets for the same DVR MAC Address, even though they 
are from the

same physical port.



We are not sure if all the switches maintained the tables this way , but 
atleast we

saw Openvswitch implementations did.  So we consciously did not promote VLANs 
for

initial phase of DVR.



--

Thanks,



Vivek





From: Kevin Benton [mailto:blak...@gmail.com]
Sent: Thursday, September 18, 2014 3:01 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question



Can you clarify what you mean with the thrashing condition? MAC addresses only 
need to be unique per-VLAN so I don't see how the same MAC on multiple VLANs 
from the same physical port would lead to any issues.



On Wed, Sep 17, 2014 at 12:41 PM, Armando M. 
arma...@gmail.commailto:arma...@gmail.com wrote:

VLAN is on the radar, vxlan/gre was done to start with.

I believe Vivek mentioned the rationale in some other thread. The gist
of it below:

In the current architecture, we use a unique DVR MAC per compute node
to forward DVR Routed traffic directly to destination compute node.
The DVR routed traffic from the source compute node will carry
'destination VMs underlay VLAN' in the frame, but the Source Mac in
that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is
used for potentially a number of overlay network VMs that would exist
on that same source compute node.

The underlay infrastructure switches will see the same DVR Unique MAC
being associated with different VLANs on incoming frames, and so this
would result in VLAN Thrashing on the switches in the physical cloud
infrastructure. Since tunneling protocols carry the entire DVR routed
inner frames as tunnel payloads, there is no thrashing effect on
underlay switches.

There will still be thrashing effect on endpoints on CNs themselves,
when they try to learn that association between inner frame source MAC
and the TEP port on which the tunneled frame is received. But that we
have addressed in L2 Agent by having a 'DVR Learning Blocker' table,
which ensures that learning for DVR routed packets alone is
side-stepped.

As a result, VLAN was not promoted as a supported underlay for the
initial DVR architecture.

Cheers,
Armando


On 16 September 2014 20:35, 龚永生 
gong...@unitedstack.commailto:gong...@unitedstack.com wrote:
 I think the VLAN should also be supported later.  The tunnel should not be
 the prerequisite for the DVR feature.


 -- Original --
 From:  Steve Wormleyopenst...@wormley.commailto:openst...@wormley.com;
 Date:  Wed, Sep 17, 2014 10:29 AM
 To:  
 openstack-devopenstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org;
 Subject:  [openstack-dev] [neutron] DVR Tunnel Design Question

 In our environment using VXLAN/GRE would make it difficult to keep some of
 the features we currently offer our customers. So for a while now I've been
 looking at the DVR code, blueprints and Google drive docs and other than it
 being the way the code was written I can't find anything indicating why a
 Tunnel/Overlay network is required for DVR or what problem it was solving.

 Basically I'm just trying to see if I missed anything as I look into doing a
 VLAN/OVS implementation.

 Thanks,
 -Steve Wormley



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







--

Kevin Benton

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] Ipset, merge refactor for J?

2014-09-18 Thread Miguel Angel Ajo Pelayo

Ack, thank you for the feedback. I will put it in WIP until we reach Kilo.

We will track any new bugfixes or changes so the refactor is ready
for early kilo, 

then after this is merged I will tackle a second refactor to extract Ipset 
out as we planned, into a new driver which extends IptablesFirewallDriver,
here I will need to coordinate with  Juergen Brendel from cisco

https://review.openstack.org/#/c/119343/ (ebtables driver)
https://review.openstack.org/#/c/119352/ (ARP fix, based on ebtables)

I also know Jakub Libosvar had good plans to enhance the iptables firewall 
driver and testing code so may be it will be good to schedule some meetings
around this starting at Kilo.

I would also like to explore the ebtables driver to extend it with Ipset too.

Best regards,
Miguel Ángel.


- Original Message -
 On Tue, Sep 16, 2014 at 7:24 AM, Sean Dague s...@dague.net wrote:
  On 09/16/2014 03:57 AM, Thierry Carrez wrote:
  Miguel Angel Ajo Pelayo wrote:
  During the ipset implementatio, we designed a refactor [1] to cleanup
  the firewall driver a bit, and move all the ipset low-level knowledge
  down into the  IpsetManager.
 
  I'd like to see this merged for J, and, it's a bit of an urgent matter
  to decide, because we keep adding small changes [2] [3] fruit of the
  early testing which break the refactor, and will add extra work which
  needs to be refactored too.
 
  The advantage of merging now, vs in J, is having K  J share a more
  common
  code base, which would help us during bug backports/etc in the future.
 
  Shihanzhang and I, are happy to see this merge during K, as it doesn't
  incur in functional changes, just code blocks are moved from the iptables
  firewall driver to IpsetManager, and the corresponding tests are moved
  too.
  [...]
 
  IMHO code refactoring should be considered a superfluous change at this
  point in the cycle. The risk/benefit ratio is too high, and focus should
  be on bugfixing at this point.
 
  +1.
 
  Hold the refactoring until Kilo.
 
 +1
 
 At this point, we should be focusing on bugs which stabilize the
 release. Lets hold this for Kilo.
 
 Thanks,
 Kyle
 
  -Sean
 
  --
  Sean Dague
  http://dague.net
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [MagnetoDB] Core developer nomination

2014-09-18 Thread Illia Khudoshyn
Congrats, Charles! Great job!

On Thu, Sep 18, 2014 at 12:05 AM, Ilya Sviridov isviri...@mirantis.com
wrote:

 Hello magnetodb contributors,

 I'm glad to nominate Charles Wang to core developers of MagnetoDB.

 He is top non-core reviewer [1], implemented notifications [2] in mdb and
 made a great progress with performance, stability and scalability testing
  of MagnetoDB

 [1] http://stackalytics.com/report/contribution/magnetodb/90
 [2]
 https://blueprints.launchpad.net/magnetodb/+spec/magnetodb-notifications

 Welcome to team, Charles!
 Looking forward for your contribution

 --
 Ilya Sviridov
 isviridov @ FreeNode




-- 

Best regards,

Illia Khudoshyn,
Software Engineer, Mirantis, Inc.



38, Lenina ave. Kharkov, Ukraine

www.mirantis.com http://www.mirantis.ru/

www.mirantis.ru



Skype: gluke_work

ikhudos...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] DVR Tunnel Design Question

2014-09-18 Thread Kevin Benton
This sounds like a bug in Openvswitch. The unique constraint should be
VLAN+MAC instead of just VLAN, so observing the same MAC on a new VLAN
should just result in a new entry without the old one being ejected.

Without this behavior, it will also break transparent L2 firewalls.
For example (diagram below), if you divide a switch into two VLANs and
then connect the sides together with a firewall in the middle that
passes frames without modifying the MAC, the switch will see the same
MAC on different VLANs. Based on the behavior you described, this
wouldn't work. Is that correct?


-
|  x  x  x  x   |   x  x  x  x  |
|---|-
 VLAN 1 | |  VLAN2
  
  |   Firewall  |
  

On Wed, Sep 17, 2014 at 11:19 PM, Narasimhan, Vivekanandan
vivekanandan.narasim...@hp.com wrote:
 Hi Kevin,



 Typically we noticed that the underlay switches maintained a table like
 this:



 VLAN IDMAC Address  Learned-Interface



 In the physical underlay, with the current architecture if we enable VLAN,
 the same DVR Unique

 MAC will appear  on different VLANs as the packets get DVR Routed.



 This will result in the rows of the above tables in the switch to be updated
 very frequently with new

 VLANs noted in incoming packets for the same DVR MAC Address, even though
 they are from the

 same physical port.



 We are not sure if all the switches maintained the tables this way , but
 atleast we

 saw Openvswitch implementations did.  So we consciously did not promote
 VLANs for

 initial phase of DVR.



 --

 Thanks,



 Vivek





 From: Kevin Benton [mailto:blak...@gmail.com]
 Sent: Thursday, September 18, 2014 3:01 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question



 Can you clarify what you mean with the thrashing condition? MAC addresses
 only need to be unique per-VLAN so I don't see how the same MAC on multiple
 VLANs from the same physical port would lead to any issues.



 On Wed, Sep 17, 2014 at 12:41 PM, Armando M. arma...@gmail.com wrote:

 VLAN is on the radar, vxlan/gre was done to start with.

 I believe Vivek mentioned the rationale in some other thread. The gist
 of it below:

 In the current architecture, we use a unique DVR MAC per compute node
 to forward DVR Routed traffic directly to destination compute node.
 The DVR routed traffic from the source compute node will carry
 'destination VMs underlay VLAN' in the frame, but the Source Mac in
 that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is
 used for potentially a number of overlay network VMs that would exist
 on that same source compute node.

 The underlay infrastructure switches will see the same DVR Unique MAC
 being associated with different VLANs on incoming frames, and so this
 would result in VLAN Thrashing on the switches in the physical cloud
 infrastructure. Since tunneling protocols carry the entire DVR routed
 inner frames as tunnel payloads, there is no thrashing effect on
 underlay switches.

 There will still be thrashing effect on endpoints on CNs themselves,
 when they try to learn that association between inner frame source MAC
 and the TEP port on which the tunneled frame is received. But that we
 have addressed in L2 Agent by having a 'DVR Learning Blocker' table,
 which ensures that learning for DVR routed packets alone is
 side-stepped.

 As a result, VLAN was not promoted as a supported underlay for the
 initial DVR architecture.

 Cheers,
 Armando


 On 16 September 2014 20:35, 龚永生 gong...@unitedstack.com wrote:
 I think the VLAN should also be supported later.  The tunnel should not be
 the prerequisite for the DVR feature.


 -- Original --
 From:  Steve Wormleyopenst...@wormley.com;
 Date:  Wed, Sep 17, 2014 10:29 AM
 To:  openstack-devopenstack-dev@lists.openstack.org;
 Subject:  [openstack-dev] [neutron] DVR Tunnel Design Question

 In our environment using VXLAN/GRE would make it difficult to keep some of
 the features we currently offer our customers. So for a while now I've
 been
 looking at the DVR code, blueprints and Google drive docs and other than
 it
 being the way the code was written I can't find anything indicating why a
 Tunnel/Overlay network is required for DVR or what problem it was solving.

 Basically I'm just trying to see if I missed anything as I look into doing
 a
 VLAN/OVS implementation.

 Thanks,
 -Steve Wormley



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





 --

 Kevin Benton


 ___
 

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread mar...@redhat.com
On 17/09/14 16:40, Charles Crouch wrote:
 
 
 - Original Message -
 Hi,

 as part of general housekeeping on our reviews, it was discussed at last
 week's meeting [1] that we should set workflow -1 for stale reviews
 (like gerrit used to do when I were a lad).

 The specific criteria discussed was 'items that have a -1 from a core
 but no response from author for 14 days'. This topic came up again
 during today's meeting and it wasn't clear if the intention was for
 cores to start enforcing this? So:

 Do we start setting WIP/workflow -1 for those reviews that have a -1
 from a core but no response from author for 14 days

 thanks, marios

 [1]
 http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html
 
 So it looks like this has already started..
 
 https://review.openstack.org/#/c/105275/
 
 I think we need to document on the wiki *precisely* the criteria for setting 
 WIP/workflow -1. For example that review above has a Jenkins failure but no
 core reviews at all.

+1 on being precise - another case in point:

https://review.openstack.org/#/c/102304/

this review has a *-2* from core, which seems to 'stick' for future
revisions; the last revision is from  14 days ago, so does this fulfill
the criteria (I'd say it doesn't)?

Not sure if you were also suggesting that -1 from Jenkins also counts,
which imo makes sense...

'items that have a -1 from core or jenkins, or a -2 from core on the
latest revision, and no response from author for 14 days'

Really this needs to be put as a motion and voted on in the next meeting,

thanks, marios


 

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread mar...@redhat.com
On 18/09/14 00:29, James Polley wrote:
 
 
 On Wed, Sep 17, 2014 at 6:26 PM, mar...@redhat.com
 mailto:mar...@redhat.com mandr...@redhat.com
 mailto:mandr...@redhat.com wrote:
 
 Hi,
 
 as part of general housekeeping on our reviews, it was discussed at last
 week's meeting [1] that we should set workflow -1 for stale reviews
 (like gerrit used to do when I were a lad).
 
 The specific criteria discussed was 'items that have a -1 from a core
 but no response from author for 14 days'. This topic came up again
 during today's meeting and it wasn't clear if the intention was for
 cores to start enforcing this? So:
 
 Do we start setting WIP/workflow -1 for those reviews that have a -1
 from a core but no response from author for 14 days
 
 
 I'm in favour of doing this; as long as we make it clear that we're
 doing it to help us focus review effort on things that are under active
 development - it doesn't mean we think the patch shouldn't land, it just
 means we know it's not ready yet so we don't want reviewers to be
 looking at it until it moves forward.
 
 For the sake of making sure new developers don't get put off, I'd like
 to see us leaving a comment explaining why we're WIPing the change and
 noting that uploading a new revision will remove the WIP automatically
 

+1 - indeed, I'd say as part of this discussion, or if/when it comes up
as a motion for a vote in the weekly meeting, we should also put out and
agree on the 'standard' text to be used for this and stick it on the
wiki (regardless of whether this is to be implemented manually at first
and perhaps automated later),

thanks, marios

setting workflow -1 as this review has been inactive for two weeks
following a negative review. Please see the wiki @ foo for more
information. Note that once you upload a new revision the workflow is
expected to be reset (feel free to shout on freenode/#tripleo if it isn't).



 
 thanks, marios
 
 [1]
 
 http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 mailto:OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Chmouel Boudjnah
Ian Cordasco ian.corda...@rackspace.com writes:

 urllib3 do that automatically. I haven’t started to investigate exactly
 why they do this. Likewise, glance client has custom certificate
 verification in glanceclient.common.https. Why? I’m not exactly certain

this probably come from pre-requests port uses when it was using httplib
which didn't have SSL verification. There is a old wiki page about it
here https://wiki.openstack.org/wiki/SecureClientConnections

Slightly off-topic, speaking about requests and OpenStack client, it
would be nice to have Expect/100-Continue feature tackled for
glanceclient and swiftclient :

https://github.com/kennethreitz/requests/issues/713

Chmouel

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] Convergence - persistence desired and observed state

2014-09-18 Thread Murugan, Visnusaran
Yeah. Unmesh, we need to have ResourcePropertiesObserved. The columns too need 
to be as mentioned in blueprint. Having all properties under a single json will 
cause concurrency issues.

-Vishnu

-Original Message-
From: Qiming Teng [mailto:teng...@linux.vnet.ibm.com] 
Sent: Wednesday, September 17, 2014 6:52 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [heat] Convergence - persistence desired and 
observed state

On Wed, Sep 17, 2014 at 12:27:34PM +, Gurjar, Unmesh wrote:
 Hi All,
 
 The convergence blueprint (https://review.openstack.org/#/c/95907/) 
 introduces two new database tables (resource_observed and 
 resource_properties_observed ) for storing the observed state of a resource 
 (currently under review: https://review.openstack.org/#/c/109012/).
 
 However, it can be simplified by storing the desired and observed state of a 
 resource in the resource table itself (two columns in the form of a blob 
 storing a JSON). Please let me know your concerns or suggestions about this 
 approach.
 
 Thanks,
 Unmesh G.

It doesn't sounds like a good idea to me unless we have some plans to handle 
potential concurrency and compatibility issues.

Regards,
Qiming


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] zeromq work for kilo

2014-09-18 Thread James Page
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 17/09/14 15:34, Doug Hellmann wrote:
 This thread [1] has turned more “future focused, so I’m moving the
 conversation to the -dev list where we usually have those sorts of
 discussions.
[...]
 I'd also like to document the current design of the ZMQ driver -
 Doug - where is the best place todo this? I thought in the source
 tree somewhere.
 
 The documentation in the oslo.messaging repository [2] would be a
 good place to start for that. If we decide deployers/operators need
 the information we can either refer to it from the guides managed
 by the documentation team or we can move/copy the information. How
 about if you start a new drivers subdirectory there, and add
 information about zmq. We can have other driver authors provide
 similar detail about their drivers in the same directory.
 
 [2]
 http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source

OK
 
- - I'll raise a review with some appropriate design documentation
sometime in the next few weeks.

 2) a list of the critical bugs that need to be fixed + any
 existing patches associated with those bugs, so they can be
 reviewed early in kilo
[...]
 That’s a good list, and shorter than I expected. I have added these
 bugs to the next-kilo milestone.

Thanks

 3) an analysis of what it would take to be able to run
 functional tests for zeromq on our CI infrastructure, not
 necessarily the full tempest run or devstack-gate job, probably
 functional tests we place in the tree with the driver (we will be
 doing this for all of the drivers) + besides writing new
 functional tests, we need to bring the unit tests for zeromq into
 the oslo.messaging repository
 
 Kapil Thangavelu started work on both functional tests for the
 ZMQ driver last week; the output from the sprint is here:
 
 https://github.com/ostack-musketeers/oslo.messaging
 
 it covers the ZMQ driver (including messaging through the
 zmq-receiver proxy) and the associated MatchMakers (local, ring,
 redis) at a varying levels of coverage, but I feel it moves
 things in the right direction - Kapil's going to raise a review
 for this in the next couple of days.
 
 Doug - has any structure been agreed within the oslo.messaging
 tree for unit/functional test splits? Right now we have them all
 in one place.
 
 I think we will want them split up, but we don’t have an agreed
 existing structure for that. I would like to see a test framework
 of some sort that defines the tests in a way that can be used to
 run the same functional for all of the drivers as separate jobs
 (with appropriate hooks for ensuring the needed services are
 running, etc.). Setting that up warrants its own spec, because
 there are going to be quite a few details to work out. We will also
 need to participate in the larger conversation about how to set up
 those functional test jobs to be consistent with the other
 projects.

I guess we can put then up for review as is and then refactor to
support any future framework changes that come along.

 4) and some improvements that we would like to make longer term
 
 a) Connection re-use on outbound messaging avoiding the current
 tcp setup overhead for every sent message.  This may also bring
 further performance benefits due to underlying messaging batching
 in ZMQ.
 
 This sounds like it would be a good thing to do, but making what we
 have work correctly and testing it feels more important for now.

Agreed - these are all futures.

 b) Moving from tcp PUSH/PULL sockets between servers to
 DEALER/DEALER (or something similar) to allow for heartbeating
 and more immediate failure detection
 
 I would need to understand how much of a rewrite that represents
 before commenting further.

Sorry - that item was a little hand-wavey - it could be minimal in
terms of impact but I would probably see it tied to a) so it might
involve the zmq-receiver moving towards zmq-proxy for both inbound and
outbound tcp connections.

 
 c) Crypto support
 
 There are some other discussions about adding crypto to messaging,
 and I hope we can do that without having to touch each driver, if
 possible.

That would be even better IMHO.

- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUGo8jAAoJEL/srsug59jDAMMP/0RCc3ElFC/E/UqZJrY+X2oE
YMB87TM06AX26wS2uKe7F7xEAsv+vMTcSK/FC13xGfd+xBTT13pHquBQjeHwUx2C
+2Fbp6QoLwbvZF69IWL0E/8M/KueGth+JrHRpPqQqK5OcuwmDbDOeq2eychDMg1w
pkpXF0PPsa0tBRo3VQUHc43po9fAqT9Vpf8nG1M5vNTuOcCnrrmGg3DOFCzjVlG1
u2Mfo3HNIL48ZdkrSLHMk+7V1j4qHdU8ADxKaE3aPbxAaSieTlsKtf4pxJSM9Ikg
6WBsCChIVZA62Ox9xRZntidWdHTKcc4Lsv/K9ZhnFzST6OoWvEkpNxol1g/YX9lV
30TUdVBhNGm+ZU+2J8vxJ5suyVD0oW/J2hZGwhomi3tpKk9euKb5HMe5Ln0Fy1k9
63//XE8IzGIkNzwnBd4qnEubo1hg7Jfuw30uij9jJ85IdWRrpXiiei4XKZNJsAgD
/RsKzQt+BwlcztcWoQ0RX3OuOonQXb+5EgfqBce8DDHs+xiGJ6ven8gUi744Mpjc
XzsyfuHLPFsQNS7vO0Qkdgw4Q5o9Es/HliMeVdSRNHn+YraAmJHFTZMIu12wcRhT

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Flavio Percoco
On 09/18/2014 04:43 AM, Donald Stufft wrote:
 
 On Sep 17, 2014, at 10:24 PM, Thomas Goirand z...@debian.org
 mailto:z...@debian.org wrote:

 On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
 I think that all of the conversation to this point has been valuable,
 the general consensus is vendoring a library is not as desirable as
 using it strictly as a dependency. It would be nice in a perfect
 world if vendoring wasn’t and issue, but in this case I think the
 root of the matter is that Debian un-vendors urllib3 and we have
 referenced the vendored urllib3 instead of installing and utilizing
 urllib3 directly.

 This poses at least one problem for us: we are not able to guarantee
 we’re using the same urllib3 library as requests is. I am unsure how
 big of a deal this ends up being, but it is a concern and has brought
 up a question of how to handle this in the most appropriate and
 consistent way across all of the distributions we as OpenStack support.

 Does this make requests a bad library we should toss aside for
 something else? Instead of being concerned with the reasons for
 vendoring urllib3 (or un-vendoring it) we should shift the conversation
 towards two questions:

 1. Is it a real issue if the version of urllib3 is mismatched between
 our client libraries and requests?
 2. If it is a real issue how are we solving it?

 The main issue is that urllib3 in requests, as other pointed out, is not
 up-to-date, and will not be updated. In fact, that's the main reason why
 the upstream authors of requests are vendorizing: it's because they
 don't want to carry the burden of staying up-to-date.
 
 I don’t think this is remotely true, often times requests updates itself
 to versions of urllib3 which aren’t even released yet. Sometimes urllib3
 might make commits and do a release that happens between requests
 versions though. I mean technically they might be not up to date until
 their next version release though.
 

 And then, there's incompatibilities and divergences that appear, leading
 to all sorts of unexpected issues, like one thing working with pip, but
 not with the packages. This kind of issues are very hard to understand
 and debug. Distributions may report the issue upstream, then upstream
 will say but it's working for me, and then we may loose a lot of time.
 This happened already, and may happen again if we don't care enough.
 
 I think this is bound to happen anytime you have downstream modifying
 things. It happens in pip (pip vendors things too) and yea it’s quite
 annoying
 but part of PEP 440 is providing ways for downstream to signal they’ve
 modified things so that instead of “foo 1.0” you have “foo 1.0+ubuntu1” or
 whatever.
 

 Obviously we can work with the requests team to figure out the best
 approach.

 There's only a single approach that works: have the requests upstream
 authors to stop embedding foreign code, and use the dependency instead.
 
 There are legitimate reasons for a project to vendor things. Linux
 distributions
 are not the end be all of distribution models and they don’t get to
 dictate to
 upstream.
 
 Generally I agree that requests should not vendor urllib3, but it’s not
 a clear
 cut thing where there is one right way to do it.
 

 We should focus on the solution here rather than continuing
 down the path of whether requests should/shouldn’t be vendoring it’s
 dependencies since it is clear that the team has their reasons and
 does not want to switch to the dependency model again.

 I'm sure they have tons of wrong reasons. If they don't want to change
 anything, then we can only try to work around the issue, and never use
 the embedded version.
 
 Generally you either work with the embedded versions or you don’t
 use requests. You’re going to get very strange incompatibility problems
 if you try to mis requests.packages.urllib3 and urllib3 in one codebase
 and if you’re using requests at all it’s going to be expecting to use
 the embedded copy of urllib3.


After having gone through the whole thread and read all the concerns,
problems and reasonings, I think we should stick to requests as-is for
now and deal with this particular issue.

Regardless of the vendorized urllib3 package, I believe requests is a
good library with an easy-to-consume API and it has solved several
problems throughout OpenStack. Not to mention it's also helpped with
making OpenStack more consistent. We've put a lot of effort to get to
this point and I don't think we should revert all that because of the
vendorized `urllib3` package.

Cheers,
Flavio


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Design Summit planning

2014-09-18 Thread Thierry Carrez
Maish Saidel-Keesing wrote:
 On 17/09/2014 23:12, Anita Kuno wrote:
 On 09/17/2014 04:01 PM, Maish Saidel-Keesing wrote:
 This looks great - but I am afraid that something might be missing.

 As part of the Design summit in Atlanta there was an Ops Meetup track.
 [1] I do not see where this fits into the current planning process that
 has been posted.
 I would like to assume that part of the purpose of the summit is to also
 collect feedback from Enterprise Operators and also from smaller ones as
 well.

 If that is so then I would kindly request that there be some other way
 of allowing that part of the community to voice their concerns, and
 provide feedback.

 Perhaps a track that is not only Operator centric - but also an End-user
 focused one as well (mixing the two would be fine as well)

 Most of them are not on the openstack-dev list and they do not
 participate in the IRC team meetings, simply because they have no idea
 that these exist or maybe do not feel comfortable there. So they will
 not have any exposure to the process.

 My 0.02 Shekels.

 [1] - http://junodesignsummit.sched.org/overview/type/ops+meetup

 Hi Maish:

 This thread is about the Design Summit, the Operators Track is a
 different thing.

 In Atlanta the Operators Track was organized by Tom Fifield and I have
 every confidence he is working hard to ensure the operators have a voice
 in Paris and that those interested can participate.

 Last summit the Operators Track ran on the Monday and the Friday giving
 folks who usually spend most of the time at the Design summit to
 participate and hear the operator's voices. I know I did and I found it
 highly educational.

 Thanks,
 Anita.
 Thanks for the clarification Anita :)

I think the plan is to have the Ops summit run on Monday, with an extra
day on Thursday to continue the discussions. I CCed Tom for more input
on that.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread James Polley
On Thu, Sep 18, 2014 at 5:21 PM, mar...@redhat.com mandr...@redhat.com
wrote:

 On 17/09/14 16:40, Charles Crouch wrote:
 
 
  - Original Message -
  Hi,
 
  as part of general housekeeping on our reviews, it was discussed at last
  week's meeting [1] that we should set workflow -1 for stale reviews
  (like gerrit used to do when I were a lad).
 
  The specific criteria discussed was 'items that have a -1 from a core
  but no response from author for 14 days'. This topic came up again
  during today's meeting and it wasn't clear if the intention was for
  cores to start enforcing this? So:
 
  Do we start setting WIP/workflow -1 for those reviews that have a -1
  from a core but no response from author for 14 days
 
  thanks, marios
 
  [1]
 
 http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html
 
  So it looks like this has already started..
 
  https://review.openstack.org/#/c/105275/
 
  I think we need to document on the wiki *precisely* the criteria for
 setting
  WIP/workflow -1. For example that review above has a Jenkins failure but
 no
  core reviews at all.

 +1 on being precise - another case in point:

 https://review.openstack.org/#/c/102304/

 this review has a *-2* from core, which seems to 'stick' for future
 revisions; the last revision is from  14 days ago, so does this fulfill
 the criteria (I'd say it doesn't)?

 Not sure if you were also suggesting that -1 from Jenkins also counts,
 which imo makes sense...

 'items that have a -1 from core or jenkins, or a -2 from core on the
 latest revision, and no response from author for 14 days'

 Really this needs to be put as a motion and voted on in the next meeting,


My understanding has always been that we don't make decisions based on
votes on IRC meetings, because it's hard to get a time for the meeting that
allows the whole team to be easily present. I wouldn't feel comfortable
doing this at the US-timezone meeting as it excludes most of APAC; nor at
the EU-timezone meeting as it excludes most of the US.

If we're looking for consensus, I'd suggest we use Gerrit to collect votes.
We could model this is a change to the CONTRIBUTING.rst[1], or perhaps we
could draft a spec around the expected workflow (perhaps formalising some
of our expectations around cores averaging 3 reviews/workday + 1 spec
review/workday at the same time?

But perhaps I'm overthinking and making this far more formal than it has to
be. We've had a fair bit of discussion on the list and we seem to be
broadly in agreement; perhaps we just need someone to propose some details
and see if we can get consensus here.

[1] What's that you say? We don't have a CONTRIBUTING.rst? One second, let
me fix that *tap tap tap* We will as soon as
https://review.openstack.org/#/c/122350/ lands!


 thanks, marios


 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread Sullivan, Jon Paul
 -Original Message-
 From: James E. Blair [mailto:cor...@inaugust.com]
 Sent: 17 September 2014 23:24
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [TripleO] Set WIP for stale patches?
 
 Sullivan, Jon Paul jonpaul.sulli...@hp.com writes:
 
  I think this highlights exactly why this should be an automated
  process.  No errors in application, and no errors in interpretation of
  what has happened.
 
  So the -1 from Jenkins was a reaction to the comment created by adding
  the workflow -1.  This is going to happen on all of the patches that
  have their workflow value altered (tests will run, result would be
  whatever the result of the test was, of course).
 
 Jenkins only runs tests in reaction to comments if they say recheck.

This is not my experience.

In my experience if the check results are not fresh enough the recheck is 
automatically run.  I am not on the infra team, so without looking up code I am 
just guessing, but my guess is that the workflow score change triggers the 
check on the presumption that it has been approved, not accounting for the 
recent(ish) update that move wip to the workflow score.

 
  But I also agree that the Jenkins vote should not be included in the
  determination of marking a patch WIP, but a human review should (So
  Code-Review and not Verified column).
 
  And in fact, for the specific example to hand, the last Jenkins vote
  was actually a +1, so as I understand it should not have been marked
  WIP.
 
 I'd like to help you see the reviews you want to see without
 externalizing your individual workflow onto contributors.  What tool do
 you use to find reviews?
 
 If it's gerrit's webui, have you tried using the Review Inbox dashboard?
 Here it is for the tripleo-image-elements project:
 
   https://review.openstack.org/#/projects/openstack/tripleo-image-
 elements,dashboards/important-changes:review-inbox-dashboard
 
 If you would prefer something else, we can customize those dashboards to
 do whatever you want, including ignoring changes that have not been
 updated in 2 weeks.

This is not solely about finding reviews.  It is about pruning stale reviews.  
I think the auto-abandon code was excellent at doing this, but alas, it is no 
more. 

 
 -Jim
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Thanks, 
Jon-Paul Sullivan ☺ Cloud Services - @hpcloud

Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway.
Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's 
Quay, Dublin 2. 
Registered Number: 361933
 
The contents of this message and any attachments to it are confidential and may 
be legally privileged. If you have received this message in error you should 
delete it from your system immediately and advise the sender.

To any recipient of this message within HP, unless otherwise stated, you should 
consider this message and attachments as HP CONFIDENTIAL.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] Persist graph for convergence

2014-09-18 Thread Murugan, Visnusaran
Hi All,

Below is the model proposal for persisting ResourceGraph.

Column name

Type

Constraint

resource_id

varchar(36)

NOT NULL ForeignKey resource.id

needed_by

varchar(36)

NULL ForeignKey resource.id

stack_id

varchar(36)

NOT NULL ForeignKey stack.id

retry_count

Integer

Default (0)

observed_ready

Boolean

Default False



Create cycle:

1.   Persist graph

2.   Mark desired states for all resources in the stack. (table: resources)

3.   Initiate stack create by calculating leaf nodes (All 
observed_ready=False nodes in resource_id column but not in needed_by column)

4.   Post create resource jobs to engine worker

5.   Engine worker upon task initiation posts observe task to 
convergence-observer and complete tack execution (return true)

6.   Mark observed_ready upon receiving resource completion notification 
from observer. Repeat step 3 through 6 till all nodes (resources) are marked 
observed_ready.

7.   If not more nodes, then mark stack create complete and return happy

Update cycle:

1.   Persist new graph for new stack created

2.   Mark desired state for all resources in both old and new stacks.

3.   Perform steps 3 to 6

4.   Delete old stack, garbage collect unwanted resources

5.   Mark stack update complete and return happy.


Clint had in fact proposed to have a version Integer default(0). To achieve 
this I feel Table:Stack should also have versioning instead of creating a 
backup_stack.

Currently backup and nested stack relation is achieved using owner_id. In case 
if update, versioning makes more sense instead on creating a backup stack and 
setting owner_id as current stack.

-Vishnu





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] zeromq work for kilo

2014-09-18 Thread Flavio Percoco
On 09/17/2014 04:34 PM, Doug Hellmann wrote:
 This thread [1] has turned more “future focused, so I’m moving the 
 conversation to the -dev list where we usually have those sorts of 
 discussions.
 
 [1] http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
 


I saw this mentioned in the `openstack` thread but I'd like us to
reconsider it.

Since we've gone through the hey please don't deprecate it, I'll help
process a couple of times with the zmq driver, I'm thinking we should
pull it out of the code base anyway.

Here's a rough plan of what I think we should do until the zmq is
updated and has a proper gate working:

1. Pull it out the code base into its own repo (stackforge?)
2. Setup the basic `oslo.messaging` gates for it
3. Release the driver on pypi as: `oslo.messaging.zmq`
4. Make lots of noise and make sure people using zmq knows that they
have to install a separate package now. (I know this means creating a
new package for many distros).

If there are folks interested in maintaining this driver, they can still
do it with the driver outside the code base. If it ever gets enough
attention and maturity, it could be re-proposed for inclusion into the
code base.

I know there are folks interested in a broker-less solution and we now
have the not-tested-in-production-yet amqp1.0 driver which I think is a
very good solution and also shares most of the semantics we rely on
protocol-wise.

I didn't go through the whole thread so, I'm sorry if I'm repeating
something that has already been said/proposed/discussed.

Flavio

 On Sep 17, 2014, at 7:54 AM, James Page james.p...@ubuntu.com wrote:
 
 Signed PGP part
 Hi Li

 On 17/09/14 11:58, Li Ma wrote:
 The scale potential is very appealing and is something I want to
 test - - hopefully in the next month or so.

 Canonical are interested in helping to maintain this driver and
 hopefully we help any critical issues prior to Juno release.


 That sounds good. I just went through all the bugs reported in the
 community.

 The only critical bug which makes ZeroMQ malfunction is
 https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
 corresponding review is under way:
 https://review.openstack.org/#/c/84938/

 Agreed

 Others are tagged to 'zmq' in
 https://bugs.launchpad.net/oslo.messaging

 Looking through Doug's suggested list of information and collating
 what I know of from our work last week:

 1) documentation for how to configure and use zeromq with
 oslo.messaging (note, not the version in oslo-incubator, the version
 in the messaging library repository)

 As part of our sprint, I worked on automating deployment of OpenStack
 + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
 that work into some general documentation on how best to configure
 ZeroMQ with OpenStack - the current documentation is a bit raw and
 does not talk about how to configure the oslo-messaging-zmq-receiver
 at all.

 I also plan some packaging updates for Debian/Ubuntu in our next dev
 cycle to make this a little easier to configure and digest - for
 example, right now no systemd unit/upstart configuration/sysv init
 script is provided to manage the zmq-receiver.

 I'd also like to document the current design of the ZMQ driver - Doug
 - where is the best place todo this? I thought in the source tree
 somewhere.
 
 The documentation in the oslo.messaging repository [2] would be a good place 
 to start for that. If we decide deployers/operators need the information we 
 can either refer to it from the guides managed by the documentation team or 
 we can move/copy the information. How about if you start a new drivers 
 subdirectory there, and add information about zmq. We can have other driver 
 authors provide similar detail about their drivers in the same directory.
 
 [2] http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
 

 2) a list of the critical bugs that need to be fixed + any existing
 patches associated with those bugs, so they can be reviewed early in kilo

 This blocks operation of nova+neutron environements:

 https://bugs.launchpad.net/oslo.messaging/+bug/1301723
  Summary: Message was sent to wrong node with zmq as rpc_backend
  Patch: https://review.openstack.org/84938

 Also notifcations are effectively unimplemented which prevents use
 with Ceilometer so I'd also add:

 https://bugs.launchpad.net/oslo.messaging/+bug/1368154
  Summary: https://bugs.launchpad.net/oslo.messaging/+bug/
  Patch: https://review.openstack.org/120745
 
 That’s a good list, and shorter than I expected. I have added these bugs to 
 the next-kilo milestone.
 

 3) an analysis of what it would take to be able to run functional
 tests for zeromq on our CI infrastructure, not necessarily the full
 tempest run or devstack-gate job, probably functional tests we place
 in the tree with the driver (we will be doing this for all of the
 drivers) + besides writing new functional tests, we need to bring the
 unit tests 

Re: [openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

2014-09-18 Thread Rajdeep Dua
+1

On Fri, Sep 12, 2014 at 12:14 PM, Tim Hinrichs thinri...@vmware.com wrote:

 +1

 Tim

 On Sep 12, 2014, at 11:47 AM, Sean Roberts seanrobert...@gmail.com
 wrote:

  +1
 
  ~sean
 
  On Sep 12, 2014, at 11:28 AM, Peter Balland pball...@vmware.com
 wrote:
 
  Hello,
 
  I would like to nominate Aaron Rosen for the congress-core team.
 
  Aaron has been involved in congress for the last few months providing
  valuable reviews as well as leading the keystone integration and
  congressclient work.
 
  References:
 
 
 https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress
 ,
  n,z
 
 
 https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
  ngressclient,n,z
 
 
 https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
  ss,n,z
 
  Please respond with +1's or any concerns.
 
  Thanks,
 
  Peter
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] Persist graph for convergence

2014-09-18 Thread Gurjar, Unmesh
Additions/comments inline below.

Thanks,
Unmesh G.
From: Murugan, Visnusaran
Sent: Thursday, September 18, 2014 1:46 PM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [heat] Persist graph for convergence

Hi All,

Below is the model proposal for persisting ResourceGraph.

Column name

Type

Constraint

resource_id

varchar(36)

NOT NULL ForeignKey resource.id

needed_by

varchar(36)

NULL ForeignKey resource.id

stack_id

varchar(36)

NOT NULL ForeignKey stack.id

retry_count

Integer

Default (0)

observed_ready

Boolean

Default False



Create cycle:

1.   Persist graph
[Unmesh] - So, it is a pre-requisite to persist all the resources of a stack 
(in the resource table) to be able to persist a graph (i.e step #0).

2.   Mark desired states for all resources in the stack. (table: resources)
[Unmesh] - This I believe should be done as a part of persisting a stack's 
resources (in step #0).

3.   Initiate stack create by calculating leaf nodes (All 
observed_ready=False nodes in resource_id column but not in needed_by column)

4.   Post create resource jobs to engine worker

5.   Engine worker upon task initiation posts observe task to 
convergence-observer and complete tack execution (return true)

6.   Mark observed_ready upon receiving resource completion notification 
from observer. Repeat step 3 through 6 till all nodes (resources) are marked 
observed_ready.
[Unmesh] - In case resource creation fails, the worker will increment 
retry_count and try creating the resource for a maximum of 'action_retry_limit' 
configured value.

7.   If not more nodes, then mark stack create complete and return happy

Update cycle:

1.   Persist new graph for new stack created

2.   Mark desired state for all resources in both old and new stacks.

3.   Perform steps 3 to 6

4.   Delete old stack, garbage collect unwanted resources

5.   Mark stack update complete and return happy.


Clint had in fact proposed to have a version Integer default(0). To achieve 
this I feel Table:Stack should also have versioning instead of creating a 
backup_stack.

Currently backup and nested stack relation is achieved using owner_id. In case 
if update, versioning makes more sense instead on creating a backup stack and 
setting owner_id as current stack.

-Vishnu





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][Architecture]Suggestions for the third vendors' plugin and driver

2014-09-18 Thread Germy Lure
Hi Trinath,

I think the vendor company has many experts to review their codes. They can
do it well.

But I still have some comments inline.

Germy

On Thu, Sep 18, 2014 at 1:42 PM, trinath.soman...@freescale.com 
trinath.soman...@freescale.com wrote:

  Though Code reviews for vendor code takes more time, I feel it must go
 through Core reviews.



 Since, Vendors might submit the code that is working fine within their
 third party CI environment but the Code review make it more efficient with
 respect to the coding standards followed in the community.



 Also, for all the vendor plugins/drivers the code reviews (+1s and +2s)
 give a feedback on the quality they must be in to be with Neutron.

I think the quality of a software mainly lies on developers, otherwise
reviewers will be very very busy.
We suppose that all core members reviewed your plugin and gave feedback
many +, so can you guarantee the plugin high quality? even no BUGs?
I think only the vendor, cooperating with customer and providing plugin and
driver, can and must guarantee the quality. But those *private* releases
only exist in vendor's disk and running in customer's machine. It cannot be
updated to community because of approving waiting, because of not efficient
enough, because of the coding standards, 



 But one suggestion I want to put forward, when an -1 or -2 is given to the
 code, Reviewers might give a brief comment on why this was given, what
 might be preferred solution and Is there any reference implementation that
 can be considered for the code in review to move away from these errors.
 This can help the developers.

If core members prefer Cisco's implementation, all the other vendors follow
it? Why different plugins? Only one is enough.
Of course, this is a very extreme assumption. We just discuss a problem.









 --

 Trinath Somanchi - B39208

 trinath.soman...@freescale.com | extn: 4048
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] DVR Tunnel Design Question

2014-09-18 Thread Narasimhan, Vivekanandan
Yes... If I recollect, we were using 1.10.x version during that time (wherein 
discovered this 
as output of ovsapp-ctl fdb-show). After that I didn’t get time to re-verify on 
later
versions of openvswitch.

BTW, if this is not intended behaviour, then I donot see any particular reason 
why VLANs
need not be enabled in the current DVR architecture.  

To enable dvr, I need to post a minor patch in L2 agent to bring-in 
DVR rules into Phys bridges (as VLANs are driven by phys bridges 
in OVS L2 Agent).

--
Thanks,

Vivek

-Original Message-
From: Kevin Benton [mailto:blak...@gmail.com] 
Sent: Thursday, September 18, 2014 12:44 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question

This sounds like a bug in Openvswitch. The unique constraint should be
VLAN+MAC instead of just VLAN, so observing the same MAC on a new VLAN
should just result in a new entry without the old one being ejected.

Without this behavior, it will also break transparent L2 firewalls.
For example (diagram below), if you divide a switch into two VLANs and then 
connect the sides together with a firewall in the middle that passes frames 
without modifying the MAC, the switch will see the same MAC on different VLANs. 
Based on the behavior you described, this wouldn't work. Is that correct?


-
|  x  x  x  x   |   x  x  x  x  |
|---|-
 VLAN 1 | |  VLAN2
  
  |   Firewall  |
  

On Wed, Sep 17, 2014 at 11:19 PM, Narasimhan, Vivekanandan 
vivekanandan.narasim...@hp.com wrote:
 Hi Kevin,



 Typically we noticed that the underlay switches maintained a table 
 like
 this:



 VLAN IDMAC Address  Learned-Interface



 In the physical underlay, with the current architecture if we enable 
 VLAN, the same DVR Unique

 MAC will appear  on different VLANs as the packets get DVR Routed.



 This will result in the rows of the above tables in the switch to be 
 updated very frequently with new

 VLANs noted in incoming packets for the same DVR MAC Address, even 
 though they are from the

 same physical port.



 We are not sure if all the switches maintained the tables this way , 
 but atleast we

 saw Openvswitch implementations did.  So we consciously did not 
 promote VLANs for

 initial phase of DVR.



 --

 Thanks,



 Vivek





 From: Kevin Benton [mailto:blak...@gmail.com]
 Sent: Thursday, September 18, 2014 3:01 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question



 Can you clarify what you mean with the thrashing condition? MAC 
 addresses only need to be unique per-VLAN so I don't see how the same 
 MAC on multiple VLANs from the same physical port would lead to any issues.



 On Wed, Sep 17, 2014 at 12:41 PM, Armando M. arma...@gmail.com wrote:

 VLAN is on the radar, vxlan/gre was done to start with.

 I believe Vivek mentioned the rationale in some other thread. The gist 
 of it below:

 In the current architecture, we use a unique DVR MAC per compute node 
 to forward DVR Routed traffic directly to destination compute node.
 The DVR routed traffic from the source compute node will carry 
 'destination VMs underlay VLAN' in the frame, but the Source Mac in 
 that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is 
 used for potentially a number of overlay network VMs that would exist 
 on that same source compute node.

 The underlay infrastructure switches will see the same DVR Unique MAC 
 being associated with different VLANs on incoming frames, and so this 
 would result in VLAN Thrashing on the switches in the physical cloud 
 infrastructure. Since tunneling protocols carry the entire DVR routed 
 inner frames as tunnel payloads, there is no thrashing effect on 
 underlay switches.

 There will still be thrashing effect on endpoints on CNs themselves, 
 when they try to learn that association between inner frame source MAC 
 and the TEP port on which the tunneled frame is received. But that we 
 have addressed in L2 Agent by having a 'DVR Learning Blocker' table, 
 which ensures that learning for DVR routed packets alone is 
 side-stepped.

 As a result, VLAN was not promoted as a supported underlay for the 
 initial DVR architecture.

 Cheers,
 Armando


 On 16 September 2014 20:35, 龚永生 gong...@unitedstack.com wrote:
 I think the VLAN should also be supported later.  The tunnel should 
 not be the prerequisite for the DVR feature.


 -- Original --
 From:  Steve Wormleyopenst...@wormley.com;
 Date:  Wed, Sep 17, 2014 10:29 AM
 To:  openstack-devopenstack-dev@lists.openstack.org;
 Subject:  [openstack-dev] [neutron] DVR Tunnel Design Question

 In our environment using VXLAN/GRE would make it difficult to keep 
 some of the features we currently offer our customers. So for a while 
 now 

[openstack-dev] [nova] Nova API meeting

2014-09-18 Thread Christopher Yeoh
Hi,

Just a reminder that the weekly Nova API meeting is being held tomorrow
Friday UTC . 

We encourage cloud operators and those who use the REST API such as
SDK developers and others who and are interested in the future of the
API to participate.

In other timezones the meeting is at:

EST 20:00 (Thu)
Japan 09:00 (Fri)
China 08:00 (Fri)
ACDT 9:30 (Fri)

The proposed agenda and meeting details are here: 

https://wiki.openstack.org/wiki/Meetings/NovaAPI

Please feel free to add items to the agenda. 

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Heat][Zaqar] Integration plan moving forward

2014-09-18 Thread Flavio Percoco
Greetings,

If I recall correctly, Heat was planning to adopt Zaqar regardless of
the result of the graduation attempt (please correct me if I'm wrong).
Based on this assumption, I'd like to start working on a plan forward to
make this integration happen.

So far, these are the use cases I've collected from past discussions:

* Notify  heat user before an action is taken, and after - Heat may want
to wait  for a response before proceeding - notifications not
necessarily needed  and signed read-only queues might help, but not
necessary
* For integrating with user's tools
* Monitoring
* Control surface
* Config management tools
* Does not require notifications and/or read-only/signed queue endpoints
*[These may be helpful, but were not brought up in the discussion]
* Subscribe to an aggregate feed of interesting events from other
open-stack components (such as Nova)
* Heat is often deployed in a different place than other
components and doesn't have access to the AMQP bus
* Large  deployments consist of multiple AMQP brokers, and there
doesn't seem to  be a nice way to aggregate all those events [need to
confirm]
* Push metadata updates to os-collect-config agent running in
servers, instead of having them poll Heat


Few questions that I think we should start from:

- Does the above list cover Heat's needs?
- Which of the use cases listed above should be addressed first?
- Can we split the above into milestones w/ due dates?


Thanks,
Flavio

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [MagnetoDB] Core developer nomination

2014-09-18 Thread Dmitriy Ukhlov
+1 from me, Charles is very active contributor and I guess if we have such
developer in core team
MagnetoDB project will become much better that it is now.

On Thu, Sep 18, 2014 at 10:09 AM, Illia Khudoshyn ikhudos...@mirantis.com
wrote:

 Congrats, Charles! Great job!

 On Thu, Sep 18, 2014 at 12:05 AM, Ilya Sviridov isviri...@mirantis.com
 wrote:

 Hello magnetodb contributors,

 I'm glad to nominate Charles Wang to core developers of MagnetoDB.

 He is top non-core reviewer [1], implemented notifications [2] in mdb and
 made a great progress with performance, stability and scalability testing
  of MagnetoDB

 [1] http://stackalytics.com/report/contribution/magnetodb/90
 [2]
 https://blueprints.launchpad.net/magnetodb/+spec/magnetodb-notifications

 Welcome to team, Charles!
 Looking forward for your contribution

 --
 Ilya Sviridov
 isviridov @ FreeNode




 --

 Best regards,

 Illia Khudoshyn,
 Software Engineer, Mirantis, Inc.



 38, Lenina ave. Kharkov, Ukraine

 www.mirantis.com http://www.mirantis.ru/

 www.mirantis.ru



 Skype: gluke_work

 ikhudos...@mirantis.com

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Best regards,
Dmitriy Ukhlov
Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] DVR Tunnel Design Question

2014-09-18 Thread Kevin Benton
To enable dvr, I need to post a minor patch in L2 agent to bring-in
DVR rules into Phys bridges (as VLANs are driven by phys bridges
in OVS L2 Agent).

Great! You read my mind and answered my follow-up question. :-)
Let me know if there is anything I can help with.

On Thu, Sep 18, 2014 at 1:51 AM, Narasimhan, Vivekanandan
vivekanandan.narasim...@hp.com wrote:
 Yes... If I recollect, we were using 1.10.x version during that time (wherein 
 discovered this
 as output of ovsapp-ctl fdb-show). After that I didn’t get time to re-verify 
 on later
 versions of openvswitch.

 BTW, if this is not intended behaviour, then I donot see any particular 
 reason why VLANs
 need not be enabled in the current DVR architecture.

 To enable dvr, I need to post a minor patch in L2 agent to bring-in
 DVR rules into Phys bridges (as VLANs are driven by phys bridges
 in OVS L2 Agent).

 --
 Thanks,

 Vivek

 -Original Message-
 From: Kevin Benton [mailto:blak...@gmail.com]
 Sent: Thursday, September 18, 2014 12:44 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question

 This sounds like a bug in Openvswitch. The unique constraint should be
 VLAN+MAC instead of just VLAN, so observing the same MAC on a new VLAN
 should just result in a new entry without the old one being ejected.

 Without this behavior, it will also break transparent L2 firewalls.
 For example (diagram below), if you divide a switch into two VLANs and then 
 connect the sides together with a firewall in the middle that passes frames 
 without modifying the MAC, the switch will see the same MAC on different 
 VLANs. Based on the behavior you described, this wouldn't work. Is that 
 correct?


 -
 |  x  x  x  x   |   x  x  x  x  |
 |---|-
  VLAN 1 | |  VLAN2
   
   |   Firewall  |
   

 On Wed, Sep 17, 2014 at 11:19 PM, Narasimhan, Vivekanandan 
 vivekanandan.narasim...@hp.com wrote:
 Hi Kevin,



 Typically we noticed that the underlay switches maintained a table
 like
 this:



 VLAN IDMAC Address  Learned-Interface



 In the physical underlay, with the current architecture if we enable
 VLAN, the same DVR Unique

 MAC will appear  on different VLANs as the packets get DVR Routed.



 This will result in the rows of the above tables in the switch to be
 updated very frequently with new

 VLANs noted in incoming packets for the same DVR MAC Address, even
 though they are from the

 same physical port.



 We are not sure if all the switches maintained the tables this way ,
 but atleast we

 saw Openvswitch implementations did.  So we consciously did not
 promote VLANs for

 initial phase of DVR.



 --

 Thanks,



 Vivek





 From: Kevin Benton [mailto:blak...@gmail.com]
 Sent: Thursday, September 18, 2014 3:01 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question



 Can you clarify what you mean with the thrashing condition? MAC
 addresses only need to be unique per-VLAN so I don't see how the same
 MAC on multiple VLANs from the same physical port would lead to any issues.



 On Wed, Sep 17, 2014 at 12:41 PM, Armando M. arma...@gmail.com wrote:

 VLAN is on the radar, vxlan/gre was done to start with.

 I believe Vivek mentioned the rationale in some other thread. The gist
 of it below:

 In the current architecture, we use a unique DVR MAC per compute node
 to forward DVR Routed traffic directly to destination compute node.
 The DVR routed traffic from the source compute node will carry
 'destination VMs underlay VLAN' in the frame, but the Source Mac in
 that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is
 used for potentially a number of overlay network VMs that would exist
 on that same source compute node.

 The underlay infrastructure switches will see the same DVR Unique MAC
 being associated with different VLANs on incoming frames, and so this
 would result in VLAN Thrashing on the switches in the physical cloud
 infrastructure. Since tunneling protocols carry the entire DVR routed
 inner frames as tunnel payloads, there is no thrashing effect on
 underlay switches.

 There will still be thrashing effect on endpoints on CNs themselves,
 when they try to learn that association between inner frame source MAC
 and the TEP port on which the tunneled frame is received. But that we
 have addressed in L2 Agent by having a 'DVR Learning Blocker' table,
 which ensures that learning for DVR routed packets alone is
 side-stepped.

 As a result, VLAN was not promoted as a supported underlay for the
 initial DVR architecture.

 Cheers,
 Armando


 On 16 September 2014 20:35, 龚永生 gong...@unitedstack.com wrote:
 I think the VLAN should also be supported later.  The tunnel should
 not be the prerequisite for the DVR feature.


 -- 

[openstack-dev] [Neutron] Gate bugs for RC-1

2014-09-18 Thread Salvatore Orlando
Bug 1357055 [1] and 1323658 [2] affect neutron jobs and are among the top
gate offenders.
With this kind of bugs, it's hard to tell whether the root cause lies with
neutron, nova, tempest, or even cirros.
However, it is not ok that these bugs are not assigned in neutron. We need
to have some neutron developers' eyes on them and root cause them as soon
as possible.

If you wish to volunteer, please assign one of these bugs (or both of them)
to yourself. Please note that this might be a rather time consuming
activity.

Salvatore

[1] https://bugs.launchpad.net/neutron/+bug/1357055
[2] https://bugs.launchpad.net/neutron/+bug/1323658
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron][db] Ensuring db isolation between tests

2014-09-18 Thread Maru Newby
For legacy reasons, the Neutron test suite creates and destroys a db for each 
test.  There is a patch proposed to create the tables once and then ensure the 
tables are wiped at the end of each test [1], providing a performance 
improvement of ~10%.  I was wondering if this is the best way to provide 
isolation, since I’ve heard that isolation via per-test transactions should 
also work.  The patch author reported problems with this approach - apparently 
nested commits were not being rolled back.  Is there some trick to isolating 
with transactions that wouldn’t be immediately obvious?

Thanks,


Maru

1: https://review.openstack.org/#/c/122028/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat][Zaqar] Integration plan moving forward

2014-09-18 Thread Angus Salkeld
On 18/09/2014 7:11 PM, Flavio Percoco fla...@redhat.com wrote:

 Greetings,

 If I recall correctly, Heat was planning to adopt Zaqar regardless of
 the result of the graduation attempt (please correct me if I'm wrong).
 Based on this assumption, I'd like to start working on a plan forward to
 make this integration happen.

 So far, these are the use cases I've collected from past discussions:

 * Notify  heat user before an action is taken, and after - Heat may want
 to wait  for a response before proceeding - notifications not
 necessarily needed  and signed read-only queues might help, but not
 necessary
 * For integrating with user's tools
 * Monitoring
 * Control surface
 * Config management tools
 * Does not require notifications and/or read-only/signed queue
endpoints
 *[These may be helpful, but were not brought up in the discussion]
 * Subscribe to an aggregate feed of interesting events from other
 open-stack components (such as Nova)
 * Heat is often deployed in a different place than other
 components and doesn't have access to the AMQP bus
 * Large  deployments consist of multiple AMQP brokers, and there
 doesn't seem to  be a nice way to aggregate all those events [need to
 confirm]
 * Push metadata updates to os-collect-config agent running in
 servers, instead of having them poll Heat


 Few questions that I think we should start from:

 - Does the above list cover Heat's needs?
 - Which of the use cases listed above should be addressed first?

IMHO it would be great to simply replace the event store we have currently,
so that the user can get a stream of progress messages during the
deployment.

-Angus

 - Can we split the above into milestones w/ due dates?


 Thanks,
 Flavio

 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron][db] Ensuring db isolation between tests

2014-09-18 Thread Salvatore Orlando
Nested commits in sqlalchemy should be seen as a single transaction on the
backend, shouldn't they?
I don't know anything about this specific problem, but the fact that unit
tests use sqlite might be a reason, since it's not really a full DBMS...

I think that wrapping tests in transaction also will require some changes
in the architecture of the tests themselves, as many tests call the API
router or the plugin which then gets a db session and open a new
transaction. Furthermore, it will change the test behaviour possibly hiding
errors; some operations indeed perform several distinct transactions, which
in this case will be seen a single transaction.

What Kevin is doing here I think was the original way we used to do that in
Neutron (Folsom). Then at some point we realised that due to plugin schema
differences we were laving tables behind and switched to drop_all and
rebuilding the schema using autogeneration at each test.

I think it should be ok to merge this patch. I will hold off the +A to give
other core reviewers a chance to look at it.

Salvatore


On 18 September 2014 11:44, Maru Newby ma...@redhat.com wrote:

 For legacy reasons, the Neutron test suite creates and destroys a db for
 each test.  There is a patch proposed to create the tables once and then
 ensure the tables are wiped at the end of each test [1], providing a
 performance improvement of ~10%.  I was wondering if this is the best way
 to provide isolation, since I’ve heard that isolation via per-test
 transactions should also work.  The patch author reported problems with
 this approach - apparently nested commits were not being rolled back.  Is
 there some trick to isolating with transactions that wouldn’t be
 immediately obvious?

 Thanks,


 Maru

 1: https://review.openstack.org/#/c/122028/
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Alan Pevec
2014-09-17 17:15 GMT+02:00 Thomas Goirand z...@debian.org:
   File bla/tests/test_ssl.py, line 19, in module
 from requests.packages.urllib3 import poolmanager
 ImportError: No module named packages.urllib3

This is in tests only, in runtime code there is conditional import of
vendorized urllib3 falling back to system library.
So what about https://review.openstack.org/122379

Cheers,
Alan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] Weekly IRC: let's cancel the meeting tomorrow

2014-09-18 Thread Vladimir Kozhukalov
+1 for canceling on 9/18/2014.

As for 9/25/2014, let's make a decision later.

Vladimir Kozhukalov

On Thu, Sep 18, 2014 at 9:33 AM, Mike Scherbakov mscherba...@mirantis.com
wrote:

 Hi Fuelers,
 as we have mini-summit in CA and majority of leads is busy with it, let's
 cancel the meeting tomorrow.

 We might want to cancel the meeting on 25th as well.
 Thanks,
 --
 Mike Scherbakov
 #mihgen


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] are we going to remove the novaclient v3 shell or what?

2014-09-18 Thread Day, Phil
 -Original Message-
 From: Kenichi Oomichi [mailto:oomi...@mxs.nes.nec.co.jp]
 Sent: 18 September 2014 02:44
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [nova] are we going to remove the novaclient
 v3 shell or what?
 
 
  -Original Message-
  From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
  Sent: Wednesday, September 17, 2014 11:59 PM
  To: OpenStack Development Mailing List (not for usage questions)
  Subject: [openstack-dev] [nova] are we going to remove the novaclient v3
 shell or what?
 
  This has come up a couple of times in IRC now but the people that
  probably know the answer aren't available.
 
  There are python-novaclient patches that are adding new CLIs to the v2
  (v1_1) and v3 shells, but now that we have the v2.1 API (v2 on v3) why
  do we still have a v3 shell in the client?  Are there plans to remove that?
 
  I don't really care either way, but need to know for code reviews.
 
  One example: [1]
 
  [1] https://review.openstack.org/#/c/108942/
 
 Sorry for a little late response,
 I think we don't need new features of v3 into novaclient anymore.
 For example, the v3 part of the above[1] was not necessary because a new
 feature server-group quota is provided as v2 and v2.1, not v3.

That would be true if there was a version of the client that supported v2.1 
today, but while the V2.1 API is still presented as V3 and doesn't include the 
tenant_id - making the V3 client the only simple way to test new V2.1 features 
in devstack as far as I can see.


How about this as a plan:

1) We add support to the client for --os-compute-api-version=v2.1   which 
maps into the client with the URL set to include v2.1(this won't be usable 
until we do step 2)

2) We change the Nova  to present the v2.1 API  as 
'http://X.X.X.X:8774/v2.1/tenant_id/
 - At this point we will have a working client for all of the stuff that's been 
moved back from V3 to V2.1, but will lose access to any V3 stuff not yet moved 
(which is the opposite of the current state where the v3 client can only be 
used for things that haven't been refactored to V2.1)

3) We remove V3 from the client.


Until we get 1  2 done, to me it still makes sense to allow small changes into 
the v3 client, so that we keep it usable with the V2.1 API



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Christopher Yeoh
On Sat, 13 Sep 2014 06:48:19 -0400
Sean Dague s...@dague.net wrote:

 On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:
  
  Hi Chris,
  
  Thanks for bring it up here,
  
  -Original Message-
  From: Chris St. Pierre [mailto:stpie...@metacloud.com]
  Sent: Saturday, September 13, 2014 2:53 AM
  To: openstack-dev@lists.openstack.org
  Subject: [openstack-dev] [nova] Expand resource name allowed
  characters
 
  We have proposed that the allowed characters for all resource
  names in Nova (flavors, aggregates, etc.) be expanded to all
  printable unicode characters and horizontal spaces:
  https://review.openstack.org/#/c/119741
 
  Currently, the only allowed characters in most resource names are
  alphanumeric, space, and [.-_].
 
 
  We have proposed this change for two principal reasons:
 
  1. We have customers who have migrated data forward since Essex,
  when no restrictions were in place, and thus have characters in
  resource names that are disallowed in the current version of
  OpenStack. This is only likely to be useful to people migrating
  from Essex or earlier, since the current restrictions were added
  in Folsom.
 
  2. It's pretty much always a bad idea to add unnecessary
  restrictions without a good reason. While we don't have an
  immediate need to use, for example, the ever-useful
  http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
  up with a reason people *shouldn't* be allowed to use it.
 
  That said, apparently people have had a need to not be allowed to
  use some characters, but it's not clear why:
  https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
  knows any reason why these printable characters should not be
  joined in holy resource naming, speak now or forever hold your
  peace.
  
  I also could not find the reason of current restriction on the bug
  report, and I'd like to know it as the history.
  On v2 API(not v2.1), each resource name contains the following
  restriction for its name:
  
Resource  | Length  | Pattern
   ---+-+--
aggregate | 1-255   | nothing
backup| nothing | nothing
flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
  | |   [a-zA-Z0-9. _-]*$'
keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
server| 1-255   | nothing
cell  | 1-255   | don't contain . and !
  
  On v2.1 API, we have applied the same restriction rule[1] for whole
  resource names for API consistency, so maybe we need to consider
  this topic for whole names.
  
  [1]:
  https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44
 
 Honestly, I bet this had to do with how the database used to be set
 up.
 

So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
multibyte characters (only 1-3 bytes). For example if you do a create
image call with an image name to glance with a 4 byte multibyte
character in the name it will 500. I'd guess we do something
similar in places with the Nova API where we have inadequate input
validation. If you want 4 byte support you need to use utf8mb4 instead.

I don't know if postgresql has any restrictions (I don't think it
does) or if db2 does too. But I don't think we can/should make it a
complete free for all. It should at most be what most databases support.

I think its a big enough change that this late in the cycle we should
push it off to Kilo. It's always much easier to loosen input validation
than tighten it (or have to have an oops revert on an officially
released Nova). Perhaps some tests to verify that everything we allow
past the input validation checks we can actually store.


 That being said, i'm pro letting names be 'utf8'. The id fields that
 are strings (like flavor_id) I think we should keep constrained, as
 we do actually do joins on them. (And as jay said, the current utf8
 schema is actually highly inefficient and bloaty.)

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] PostgreSQL jobs slow in the gate

2014-09-18 Thread Sean Dague
On 09/17/2014 11:50 PM, Clark Boylan wrote:
 On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:
 On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:


 On 9/17/2014 7:59 PM, Ian Wienand wrote:
 On 09/18/2014 09:49 AM, Clark Boylan wrote:
 Recent sampling of test run times shows that our tempest jobs run
 against clouds using PostgreSQL are significantly slower than jobs run
 against clouds using MySQL.

 FYI There is a possibly relevant review out for max_connections limits
 [1], although it seems to have some issues with shmem usage

 -i

 [1] https://review.openstack.org/#/c/121952/

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 That's a backport of a fix from master where we were hitting fatal 
 errors due to too many DB connections which was brought on by the 
 changes to cinder and glance to run as many workers as there were CPUs 
 available.  So I don't think it probably plays here...

 The errors pointed out in another part of the thread have been around 
 for awhile, I think they are due to negative tests where we're hitting 
 unique constraints because of the negative tests, so they are expected.

 We should also note that the postgresql jobs run with the nova metadata 
 API service, I'm not sure how much of a factor that would have here.

 Is there anything else unique about those jobs from the MySQL ones?

 Good question. There are apparently other differences. The postgres job
 runs Keystone under eventlet instead of via apache mod_wsgi. It also
 sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
 I can find is the one you point out, nova api metadata service is run as
 an independent thing.

 Could these things be related? It would be relatively simple to push a
 change or two to devstack-gate to test this but there are enough options
 here that I probably won't do that until we think at least one of these
 options is at fault.
 I am starting to feel bad that I picked on PostgreSQL and completely
 forgot that there were other items in play here. I went ahead and
 uploaded [0] to run all devstack jobs without keystone wsgi services
 (eventlet) and [1] to run all devstack job with keystone wsgi services
 and the initial results are pretty telling.
 
 It appears that keystone eventlet is the source of the slowness in this
 job. With keystone eventlet all of the devstack jobs are slower and with
 keystone wsgi all of the jobs are quicker. Probably need to collect a
 bit more data but this doesn't look good for keystone eventlet.
 
 Thank you Matt for pointing me in this direction.
 
 [0] https://review.openstack.org/#/c/122299/
 [1] https://review.openstack.org/#/c/122300/

Don't feel bad. :)

The point that Clark highlights here is a good one. There is an
assumption that once someone creates a job in infra, the magic elves are
responsible for it.

But there are no magic elves. So jobs like this need sponsors.

Maybe the right thing to do is not conflate this configuration and put
an eventlet version of the keystone job only on keystone (because the
keystone team was the one that proposed having a config like that, but
it's so far away from their project they aren't ever noticing when it's
regressing).

Same issue with the metadata server split. That's really only a thing
Nova cares about. It shouldn't impact anyone else.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Infra][Cinder] Coraid CI system

2014-09-18 Thread Mykola Grygoriev
Dear community members,

Please have a look to Coraid CI system -
http://38.111.159.9:8080/job/CoraidCI/

We have done all requirements for Third Party CI system, provided here -
http://ci.openstack.org/third_party.html#requirements

Please look at Coraid third-party system one more time and, please, show us
what we have to add or improve in order to get voting rights for gerrit
user coraid-ci.

On Fri, Sep 12, 2014 at 3:15 PM, Roman Bogorodskiy 
rbogorods...@mirantis.com wrote:

 Hi,

 Mykola has some problems sending emails to the list, so he asked me to
 post a response, here it goes:

 ---
 Remy, I have improved Coraid CI system and added logs of all components of
 devstack. Please have a look:

 http://38.111.159.9:8080/job/Coraid_CI/164/

 According to the requirements from
 http://ci.openstack.org/third_party.html#requesting-a-service-account ,
 Gerrit plugin from Jenkins should be given the following options:

 Successful: gerrit approve CHANGE,PATCHSET --message 'Build
 Successful BUILDS_STATS' --verified VERIFIED --code-review
 CODE_REVIEW
 Failed: gerrit approve CHANGE,PATCHSET --message 'Build Failed
 BUILDS_STATS' --verified VERIFIED --code-review CODE_REVIEW
 Unstable: gerrit approve CHANGE,PATCHSET --message 'Build Unstable
 BUILDS_STATS' --verified VERIFIED --code-review CODE_REVIEW

 I configured gerrit plugin this way, so it sends the following comment
 after checking patchset or comment with recheck. For example,
 https://review.openstack.org/#/c/120907/

 Patch Set 1:

 Build Successful

 http://38.111.159.9:8080/job/Coraid_CI/164/ : SUCCESS


 All logs are on this page. They are there as artifacts.

  I took a quick look and I don’t see which test cases are being run?
 We test Coraid Cinder driver with standard tempest tests using
 ./driver_certs/cinder_driver_cert.sh script. Test cases are in the log of
 job.

 Please look at Coraid third-party system one more time and, please, show us
 what we have to add or improve in order to get voting rights for gerrit
 user coraid-ci.

 Also I have set gerrit plugin on our Jenkins to the silent mode as you
 suggested.

 Thank you in advance.
 

 On Fri, Sep 5, 2014 at 7:34 PM, Asselin, Ramy ramy.asse...@hp.com wrote:

 -1 from me (non-cinder core)

 It very nice to see you're making progress. I, personally, was very
 confused about voting.
 Here's my understanding: Voting: it is the ability to provide an
 official +1 -1 vote in the gerrit system.

 I don't see a stable history [1]. Before requesting voting, you should
 enable your system on the cinder project itself.
 Initially, you should disable ALL gerrit comments, i.e. run in silent
 mode, per request from cinder PTL [2]. Once stable there, you can enable
 gerrit comments. At this point, everyone can see pass/fail comments with a
 vote=0.
 Once stable there on real patches, you can request voting again, where
 the pass/fail would vote +1/-1.

 Ramy
 [1] http://38.111.159.9:8080/job/Coraid_CI/35/console
 [2]
 http://lists.openstack.org/pipermail/openstack-dev/2014-August/043876.html


 -Original Message-
 From: Duncan Thomas [mailto:duncan.tho...@gmail.com]
 Sent: Friday, September 05, 2014 7:55 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Infra][Cinder] Coraid CI system

 +1 from me (Cinder core)


 On 5 September 2014 15:09, Mykola Grygoriev mgrygor...@mirantis.com
 wrote:
  Hi,
 
  My name is Mykola Grygoriev and I'm engineer who currently working on
  deploying 3d party CI for Сoraid Сinder driver.
 
  Following instructions on
 
  http://ci.openstack.org/third_party.html#requesting-a-service-account
 
  asking for adding gerrit CI account (coraid-ci) to the Voting
  Third-Party CI Gerrit group.
 
 
 
  We have already added description of Coraid CI system to wiki page -
  https://wiki.openstack.org/wiki/ThirdPartySystems/Coraid_CI
 
  We used openstack-dev/sandbox project to test current CI
  infrastructure with OpenStack Gerrit system. Please find our history
 there.
 
  Please have a look to results of Coraid CI system. it currently takes
  updates from openstack/cinder project:
  http://38.111.159.9:8080/job/Coraid_CI/32/
  http://38.111.159.9:8080/job/Coraid_CI/33/
 
  Thank you in advance.
 
  --
  Best regards,
  Mykola Grygoriev
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 



 --
 Duncan Thomas

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Sean Dague
On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
 On Sat, 13 Sep 2014 06:48:19 -0400
 Sean Dague s...@dague.net wrote:
 
 On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:

 Hi Chris,

 Thanks for bring it up here,

 -Original Message-
 From: Chris St. Pierre [mailto:stpie...@metacloud.com]
 Sent: Saturday, September 13, 2014 2:53 AM
 To: openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [nova] Expand resource name allowed
 characters

 We have proposed that the allowed characters for all resource
 names in Nova (flavors, aggregates, etc.) be expanded to all
 printable unicode characters and horizontal spaces:
 https://review.openstack.org/#/c/119741

 Currently, the only allowed characters in most resource names are
 alphanumeric, space, and [.-_].


 We have proposed this change for two principal reasons:

 1. We have customers who have migrated data forward since Essex,
 when no restrictions were in place, and thus have characters in
 resource names that are disallowed in the current version of
 OpenStack. This is only likely to be useful to people migrating
 from Essex or earlier, since the current restrictions were added
 in Folsom.

 2. It's pretty much always a bad idea to add unnecessary
 restrictions without a good reason. While we don't have an
 immediate need to use, for example, the ever-useful
 http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
 up with a reason people *shouldn't* be allowed to use it.

 That said, apparently people have had a need to not be allowed to
 use some characters, but it's not clear why:
 https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
 knows any reason why these printable characters should not be
 joined in holy resource naming, speak now or forever hold your
 peace.

 I also could not find the reason of current restriction on the bug
 report, and I'd like to know it as the history.
 On v2 API(not v2.1), each resource name contains the following
 restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain . and !

 On v2.1 API, we have applied the same restriction rule[1] for whole
 resource names for API consistency, so maybe we need to consider
 this topic for whole names.

 [1]:
 https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

 Honestly, I bet this had to do with how the database used to be set
 up.

 
 So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
 multibyte characters (only 1-3 bytes). For example if you do a create
 image call with an image name to glance with a 4 byte multibyte
 character in the name it will 500. I'd guess we do something
 similar in places with the Nova API where we have inadequate input
 validation. If you want 4 byte support you need to use utf8mb4 instead.

Oh... fun. :(

 I don't know if postgresql has any restrictions (I don't think it
 does) or if db2 does too. But I don't think we can/should make it a
 complete free for all. It should at most be what most databases support.
 
 I think its a big enough change that this late in the cycle we should
 push it off to Kilo. It's always much easier to loosen input validation
 than tighten it (or have to have an oops revert on an officially
 released Nova). Perhaps some tests to verify that everything we allow
 past the input validation checks we can actually store.

So, honestly, that seems like a pendulum swing in an odd way.

Havana use anything you want!
Icehouse ?
Juno strict asci!
Kilo utf8

Can't we just catch the db exception correctly in glance and not have it
explode? And then allow it. Exploding with a 500 on a bad name seems the
wrong thing to do anyway.

That would also mean that if the user changed their database to support
utf8mb4 (which they might want to do if it was important to them) it
would all work.

I think some release notes would be fine to explain the current
situation and limitations.

Basically, lets skate towards the puck here, realizing some corner cases
exist, but that we're moving in the direction we want to be, not back
tracking.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [zaqar] Signing Off from Core

2014-09-18 Thread Malini Kamalambal
Thank You Alej – not just for the tons of code you wrote for Zaqar, but also in 
making Zaqar a fun, open and welcoming part of the community!
Good Luck for everything ahead!

From: Victoria Martínez de la Cruz 
victo...@vmartinezdelacruz.commailto:victo...@vmartinezdelacruz.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Wednesday, September 17, 2014 6:29 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [zaqar] Signing Off from Core

Thanks for everything Alej!

Besides your contributions to the Zaqar team you also shown to be a great 
person.

I'm truly grateful that I could have you as a mentor during GSoC.

All the best :)

2014-09-17 19:14 GMT-03:00 Flavio Percoco 
fla...@redhat.commailto:fla...@redhat.com:
On 09/17/2014 11:50 PM, Alejandro Cabrera wrote:
 Hey all,

 I am officially removing myself as a core member of the Zaqar project.

 Thanks for all the good times, friends, and I wish you the best for the 
 future!

Alejandro,

I think I speak for everyone when I say the project is where it is also
because of your contributions and support. You played a key role in the
project's growth and we all thank you a lot for that.

I'm sad to see you go and I wish you luck on your new adventures :)
Cheers,
Flavio


--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Ken'ichi Ohmichi
2014-09-18 19:57 GMT+09:00 Sean Dague s...@dague.net:
 On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
 On Sat, 13 Sep 2014 06:48:19 -0400
 Sean Dague s...@dague.net wrote:

 We have proposed that the allowed characters for all resource
 names in Nova (flavors, aggregates, etc.) be expanded to all
 printable unicode characters and horizontal spaces:
 https://review.openstack.org/#/c/119741

 Currently, the only allowed characters in most resource names are
 alphanumeric, space, and [.-_].


 We have proposed this change for two principal reasons:

 1. We have customers who have migrated data forward since Essex,
 when no restrictions were in place, and thus have characters in
 resource names that are disallowed in the current version of
 OpenStack. This is only likely to be useful to people migrating
 from Essex or earlier, since the current restrictions were added
 in Folsom.

 2. It's pretty much always a bad idea to add unnecessary
 restrictions without a good reason. While we don't have an
 immediate need to use, for example, the ever-useful
 http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
 up with a reason people *shouldn't* be allowed to use it.

 That said, apparently people have had a need to not be allowed to
 use some characters, but it's not clear why:
 https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
 knows any reason why these printable characters should not be
 joined in holy resource naming, speak now or forever hold your
 peace.

 I also could not find the reason of current restriction on the bug
 report, and I'd like to know it as the history.
 On v2 API(not v2.1), each resource name contains the following
 restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain . and !

 On v2.1 API, we have applied the same restriction rule[1] for whole
 resource names for API consistency, so maybe we need to consider
 this topic for whole names.

 [1]:
 https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

 Honestly, I bet this had to do with how the database used to be set
 up.


 So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
 multibyte characters (only 1-3 bytes). For example if you do a create
 image call with an image name to glance with a 4 byte multibyte
 character in the name it will 500. I'd guess we do something
 similar in places with the Nova API where we have inadequate input
 validation. If you want 4 byte support you need to use utf8mb4 instead.

 Oh... fun. :(

 I don't know if postgresql has any restrictions (I don't think it
 does) or if db2 does too. But I don't think we can/should make it a
 complete free for all. It should at most be what most databases support.

 I think its a big enough change that this late in the cycle we should
 push it off to Kilo. It's always much easier to loosen input validation
 than tighten it (or have to have an oops revert on an officially
 released Nova). Perhaps some tests to verify that everything we allow
 past the input validation checks we can actually store.

 So, honestly, that seems like a pendulum swing in an odd way.

 Havana use anything you want!
 Icehouse ?
 Juno strict asci!
 Kilo utf8

Correct validation history is

Essex: use anything you want!
Folsom: strict asci!
[..]
Juno: strict asci!

So I don't think we should make the input validation loose right now
to avoid a pendulum swing.


 Can't we just catch the db exception correctly in glance and not have it
 explode? And then allow it. Exploding with a 500 on a bad name seems the
 wrong thing to do anyway.

 That would also mean that if the user changed their database to support
 utf8mb4 (which they might want to do if it was important to them) it
 would all work.

 I think some release notes would be fine to explain the current
 situation and limitations.

 Basically, lets skate towards the puck here, realizing some corner cases
 exist, but that we're moving in the direction we want to be, not back
 tracking.

One idea is that: How about using base64 encoding/decoding if non-ascii
chars come? REST API layer would encode resource names if necessary
before passing it to DB layer, and we don't need to consider backend DB
features. The disadvantage is that available name length becomes short if
non-ascii chars. But maybe that would be acceptable..

Thanks
Ken'ichi Ohmichi

---

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Sean Dague
On 09/18/2014 07:19 AM, Ken'ichi Ohmichi wrote:
 2014-09-18 19:57 GMT+09:00 Sean Dague s...@dague.net:
 On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
 On Sat, 13 Sep 2014 06:48:19 -0400
 Sean Dague s...@dague.net wrote:

 We have proposed that the allowed characters for all resource
 names in Nova (flavors, aggregates, etc.) be expanded to all
 printable unicode characters and horizontal spaces:
 https://review.openstack.org/#/c/119741

 Currently, the only allowed characters in most resource names are
 alphanumeric, space, and [.-_].


 We have proposed this change for two principal reasons:

 1. We have customers who have migrated data forward since Essex,
 when no restrictions were in place, and thus have characters in
 resource names that are disallowed in the current version of
 OpenStack. This is only likely to be useful to people migrating
 from Essex or earlier, since the current restrictions were added
 in Folsom.

 2. It's pretty much always a bad idea to add unnecessary
 restrictions without a good reason. While we don't have an
 immediate need to use, for example, the ever-useful
 http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
 up with a reason people *shouldn't* be allowed to use it.

 That said, apparently people have had a need to not be allowed to
 use some characters, but it's not clear why:
 https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
 knows any reason why these printable characters should not be
 joined in holy resource naming, speak now or forever hold your
 peace.

 I also could not find the reason of current restriction on the bug
 report, and I'd like to know it as the history.
 On v2 API(not v2.1), each resource name contains the following
 restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain . and !

 On v2.1 API, we have applied the same restriction rule[1] for whole
 resource names for API consistency, so maybe we need to consider
 this topic for whole names.

 [1]:
 https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

 Honestly, I bet this had to do with how the database used to be set
 up.


 So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
 multibyte characters (only 1-3 bytes). For example if you do a create
 image call with an image name to glance with a 4 byte multibyte
 character in the name it will 500. I'd guess we do something
 similar in places with the Nova API where we have inadequate input
 validation. If you want 4 byte support you need to use utf8mb4 instead.

 Oh... fun. :(

 I don't know if postgresql has any restrictions (I don't think it
 does) or if db2 does too. But I don't think we can/should make it a
 complete free for all. It should at most be what most databases support.

 I think its a big enough change that this late in the cycle we should
 push it off to Kilo. It's always much easier to loosen input validation
 than tighten it (or have to have an oops revert on an officially
 released Nova). Perhaps some tests to verify that everything we allow
 past the input validation checks we can actually store.

 So, honestly, that seems like a pendulum swing in an odd way.

 Havana use anything you want!
 Icehouse ?
 Juno strict asci!
 Kilo utf8
 
 Correct validation history is
 
 Essex: use anything you want!
 Folsom: strict asci!
 [..]
 Juno: strict asci!

 So I don't think we should make the input validation loose right now
 to avoid a pendulum swing.

Ok, great. That history makes me ok with status quo. I didn't realize it
went back so far.

 Can't we just catch the db exception correctly in glance and not have it
 explode? And then allow it. Exploding with a 500 on a bad name seems the
 wrong thing to do anyway.

 That would also mean that if the user changed their database to support
 utf8mb4 (which they might want to do if it was important to them) it
 would all work.

 I think some release notes would be fine to explain the current
 situation and limitations.

 Basically, lets skate towards the puck here, realizing some corner cases
 exist, but that we're moving in the direction we want to be, not back
 tracking.
 
 One idea is that: How about using base64 encoding/decoding if non-ascii
 chars come? REST API layer would encode resource names if necessary
 before passing it to DB layer, and we don't need to consider backend DB
 features. The disadvantage is that available name length becomes short if
 non-ascii chars. But maybe that would be acceptable..

Honestly, we should utf8 through to the db. If the user's db doesn't
support it... that's their implementation issue.

Also... glance should not 

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/17/2014 10:36 PM, Joe Gordon wrote:
 Hi All,
 
 My understanding of Zaqar is that it's like SQS. SQS uses distributed
 queues, which have a few unusual properties [0]:
 
 
 Message Order
 
 Amazon SQS makes a best effort to preserve order in messages, but due to
 the distributed nature of the queue, we cannot guarantee you will
 receive messages in the exact order you sent them. If your system
 requires that order be preserved, we recommend you place sequencing
 information in each message so you can reorder the messages upon receipt.
 

Zaqar guarantees FIFO. To be more precise, it does that relying on the
storage backend ability to do so as well. Depending on the storage used,
guaranteeing FIFO may have some performance penalties.

We have discussed on adding a way to enable/disable some features - like
FIFO - in a per-queue bases and now that we have flavors, we may work on
allowing things like enabling/disabling FIFO on a per-flavor basis.


 At-Least-Once Delivery
 
 Amazon SQS stores copies of your messages on multiple servers for
 redundancy and high availability. On rare occasions, one of the servers
 storing a copy of a message might be unavailable when you receive or
 delete the message. If that occurs, the copy of the message will not be
 deleted on that unavailable server, and you might get that message copy
 again when you receive messages. Because of this, you must design your
 application to be idempotent (i.e., it must not be adversely affected if
 it processes the same message more than once).

As of now, Zaqar fully relies on the storage replication/clustering
capabilities to provide data consistency, availability and fault
tolerance. However, as far as consuming messages is concerned, it can
guarantee once-and-only-once and/or at-least-once delivery depending on
the message pattern used to consume messages. Using pop or claims
guarantees the former whereas streaming messages out of Zaqar guarantees
the later.


 Message Sample
 
 The behavior of retrieving messages from the queue depends whether you
 are using short (standard) polling, the default behavior, or long
 polling. For more information about long polling, see Amazon SQS Long
 Polling
 http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html.
 
 With short polling, when you retrieve messages from the queue, Amazon
 SQS samples a subset of the servers (based on a weighted random
 distribution) and returns messages from just those servers. This means
 that a particular receive request might not return all your messages.
 Or, if you have a small number of messages in your queue (less than
 1000), it means a particular request might not return any of your
 messages, whereas a subsequent request will. If you keep retrieving from
 your queues, Amazon SQS will sample all of the servers, and you will
 receive all of your messages.
 
 The following figure shows short polling behavior of messages being
 returned after one of your system components makes a receive request.
 Amazon SQS samples several of the servers (in gray) and returns the
 messages from those servers (Message A, C, D, and B). Message E is not
 returned to this particular request, but it would be returned to a
 subsequent request.

Image here:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/images/ArchOverview_Receive.png

Currently the best and only way to pull messages from Zaqar is by using
short polling. We'll work on long-polling and websocket support.
Requests are guaranteed to return available messages if there are any.

 Presumably SQS has these properties because it makes the
 system scalable, if so does Zaqar have the same properties (not just
 making these same guarantees in the API, but actually having these
 properties in the backends)? And if not, why?

Based on our short conversation on IRC last night, I understand you're
concerned that FIFO may result in performance issues. That's a valid
concern and I think the right answer is that it depends on the storage.
If the storage has a built-in FIFO guarantee then there's nothing Zaqar
needs to do there. In the other hand, if the storage does not have a
built-in support for FIFO, Zaqar will cover it in the driver
implementation. In the mongodb driver, each message has a marker that is
used to guarantee FIFO.


 I looked on the wiki [1]
 for information on this, but couldn't find anything.
 


We have this[0] section in the FAQ but it definitely doesn't answer your
questions. I'm sure we had this documented way better but I can't find
the link. :(


[0]
https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#How_does_Zaqar_compare_to_AWS_.28SQS.2FSNS.29.3F

Hope the above helps,
Flavio

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [MagnetoDB] Core developer nomination

2014-09-18 Thread Ajaya Agrawal
+1

Cheers,
Ajaya

On Thu, Sep 18, 2014 at 2:38 PM, Dmitriy Ukhlov dukh...@mirantis.com
wrote:

 +1 from me, Charles is very active contributor and I guess if we have such
 developer in core team
 MagnetoDB project will become much better that it is now.

 On Thu, Sep 18, 2014 at 10:09 AM, Illia Khudoshyn ikhudos...@mirantis.com
  wrote:

 Congrats, Charles! Great job!

 On Thu, Sep 18, 2014 at 12:05 AM, Ilya Sviridov isviri...@mirantis.com
 wrote:

 Hello magnetodb contributors,

 I'm glad to nominate Charles Wang to core developers of MagnetoDB.

 He is top non-core reviewer [1], implemented notifications [2] in mdb
 and made a great progress with performance, stability and scalability
 testing  of MagnetoDB

 [1] http://stackalytics.com/report/contribution/magnetodb/90
 [2]
 https://blueprints.launchpad.net/magnetodb/+spec/magnetodb-notifications

 Welcome to team, Charles!
 Looking forward for your contribution

 --
 Ilya Sviridov
 isviridov @ FreeNode




 --

 Best regards,

 Illia Khudoshyn,
 Software Engineer, Mirantis, Inc.



 38, Lenina ave. Kharkov, Ukraine

 www.mirantis.com http://www.mirantis.ru/

 www.mirantis.ru



 Skype: gluke_work

 ikhudos...@mirantis.com

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Best regards,
 Dmitriy Ukhlov
 Mirantis Inc.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Thomas Goirand
On 09/18/2014 04:01 PM, Flavio Percoco wrote:
 After having gone through the whole thread and read all the concerns,
 problems and reasonings, I think we should stick to requests as-is for
 now and deal with this particular issue.
 
 Regardless of the vendorized urllib3 package, I believe requests is a
 good library with an easy-to-consume API and it has solved several
 problems throughout OpenStack. Not to mention it's also helpped with
 making OpenStack more consistent. We've put a lot of effort to get to
 this point and I don't think we should revert all that because of the
 vendorized `urllib3` package.
 
 Cheers,
 Flavio

I, at least, haven't suggested that we stop using requests. Just that we
don't internally use any of the requests.packages.* stuff.

The rest of the debate about the good or bad things with vendorizing,
even if it is my view that it's a really bad thing, is IMO not
interesting for the OpenStack project.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Flavio Percoco
On 09/18/2014 01:28 PM, Sean Dague wrote:
 On 09/18/2014 07:19 AM, Ken'ichi Ohmichi wrote:
 2014-09-18 19:57 GMT+09:00 Sean Dague s...@dague.net:
 On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
 On Sat, 13 Sep 2014 06:48:19 -0400
 Sean Dague s...@dague.net wrote:

 We have proposed that the allowed characters for all resource
 names in Nova (flavors, aggregates, etc.) be expanded to all
 printable unicode characters and horizontal spaces:
 https://review.openstack.org/#/c/119741

 Currently, the only allowed characters in most resource names are
 alphanumeric, space, and [.-_].


 We have proposed this change for two principal reasons:

 1. We have customers who have migrated data forward since Essex,
 when no restrictions were in place, and thus have characters in
 resource names that are disallowed in the current version of
 OpenStack. This is only likely to be useful to people migrating
 from Essex or earlier, since the current restrictions were added
 in Folsom.

 2. It's pretty much always a bad idea to add unnecessary
 restrictions without a good reason. While we don't have an
 immediate need to use, for example, the ever-useful
 http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
 up with a reason people *shouldn't* be allowed to use it.

 That said, apparently people have had a need to not be allowed to
 use some characters, but it's not clear why:
 https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
 knows any reason why these printable characters should not be
 joined in holy resource naming, speak now or forever hold your
 peace.

 I also could not find the reason of current restriction on the bug
 report, and I'd like to know it as the history.
 On v2 API(not v2.1), each resource name contains the following
 restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain . and !

 On v2.1 API, we have applied the same restriction rule[1] for whole
 resource names for API consistency, so maybe we need to consider
 this topic for whole names.

 [1]:
 https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

 Honestly, I bet this had to do with how the database used to be set
 up.


 So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
 multibyte characters (only 1-3 bytes). For example if you do a create
 image call with an image name to glance with a 4 byte multibyte
 character in the name it will 500. I'd guess we do something
 similar in places with the Nova API where we have inadequate input
 validation. If you want 4 byte support you need to use utf8mb4 instead.

 Oh... fun. :(

 I don't know if postgresql has any restrictions (I don't think it
 does) or if db2 does too. But I don't think we can/should make it a
 complete free for all. It should at most be what most databases support.

 I think its a big enough change that this late in the cycle we should
 push it off to Kilo. It's always much easier to loosen input validation
 than tighten it (or have to have an oops revert on an officially
 released Nova). Perhaps some tests to verify that everything we allow
 past the input validation checks we can actually store.

 So, honestly, that seems like a pendulum swing in an odd way.

 Havana use anything you want!
 Icehouse ?
 Juno strict asci!
 Kilo utf8

 Correct validation history is

 Essex: use anything you want!
 Folsom: strict asci!
 [..]
 Juno: strict asci!

 So I don't think we should make the input validation loose right now
 to avoid a pendulum swing.
 
 Ok, great. That history makes me ok with status quo. I didn't realize it
 went back so far.
 
 Can't we just catch the db exception correctly in glance and not have it
 explode? And then allow it. Exploding with a 500 on a bad name seems the
 wrong thing to do anyway.

 That would also mean that if the user changed their database to support
 utf8mb4 (which they might want to do if it was important to them) it
 would all work.

 I think some release notes would be fine to explain the current
 situation and limitations.

 Basically, lets skate towards the puck here, realizing some corner cases
 exist, but that we're moving in the direction we want to be, not back
 tracking.

 One idea is that: How about using base64 encoding/decoding if non-ascii
 chars come? REST API layer would encode resource names if necessary
 before passing it to DB layer, and we don't need to consider backend DB
 features. The disadvantage is that available name length becomes short if
 non-ascii chars. But maybe that would be acceptable..
 
 Honestly, we should utf8 through to the db. If the user's db doesn't
 support it... that's their 

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Donald Stufft

 On Sep 18, 2014, at 7:43 AM, Thomas Goirand z...@debian.org wrote:
 
 On 09/18/2014 04:01 PM, Flavio Percoco wrote:
 After having gone through the whole thread and read all the concerns,
 problems and reasonings, I think we should stick to requests as-is for
 now and deal with this particular issue.
 
 Regardless of the vendorized urllib3 package, I believe requests is a
 good library with an easy-to-consume API and it has solved several
 problems throughout OpenStack. Not to mention it's also helpped with
 making OpenStack more consistent. We've put a lot of effort to get to
 this point and I don't think we should revert all that because of the
 vendorized `urllib3` package.
 
 Cheers,
 Flavio
 
 I, at least, haven't suggested that we stop using requests. Just that we
 don't internally use any of the requests.packages.* stuff.
 
 The rest of the debate about the good or bad things with vendorizing,
 even if it is my view that it's a really bad thing, is IMO not
 interesting for the OpenStack project.

I don’t believe that’s a good idea. If you’re wanting to use urllib3 in order
to interact with requests than you *should* be using requests.packages.urllib3,
to do anything else risks having two different versions of urllib3 primitives at
play in one subsystem.

It’s not even completely possible in the case that prompted this thread 
originally
since the reason requests.packages.urllib3 was being imported from was so that
there could be an is instance() check against one of the classes. If that wasn’t
imported from requests.packages.urllib3 but instead from just urllib3 than the
isinstance check would always fail.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Thomas Goirand
On 09/18/2014 10:43 AM, Donald Stufft wrote:
 Obviously we can work with the requests team to figure out the best
 approach.

 There's only a single approach that works: have the requests upstream
 authors to stop embedding foreign code, and use the dependency instead.
 
 There are legitimate reasons for a project to vendor things.

Yes, there's lot of reasons. But so fare, I haven't read about any valid
one.

 Linux distributions are not the end be all of distribution models and
 they don’t get to dictate to upstream.

Well, distributions is where the final user is, and where software gets
consumed. Our priority should be the end users.

 Generally I agree that requests should not vendor urllib3

Unfortunately, it doesn't seem requests upstream agree, so we can only
deal with the issue. This means not using requests.packages.*.

   You’re going to get very strange incompatibility problems
 if you try to mis requests.packages.urllib3 and urllib3 in one codebase
 and if you’re using requests at all it’s going to be expecting to use
 the embedded copy of urllib3.

I'm well aware of this. As I wrote, I already had to deal with issues
like that, and I'm expecting even more in the future.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Donald Stufft

 On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
 
 
 Linux distributions are not the end be all of distribution models and
 they don’t get to dictate to upstream.
 
 Well, distributions is where the final user is, and where software gets
 consumed. Our priority should be the end users.


Distributions are not the only place that people get their software from,
unless you think that the ~3 million downloads requests has received
on PyPI in the last 30 days are distributions downloading requests to
package in their OSs.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread Jeremy Stanley
On 2014-09-18 08:06:11 + (+), Sullivan, Jon Paul wrote:
 In my experience if the check results are not fresh enough the
 recheck is automatically run.  I am not on the infra team, so
 without looking up code I am just guessing, but my guess is that
 the workflow score change triggers the check on the presumption
 that it has been approved, not accounting for the recent(ish)
 update that move wip to the workflow score.

We turned off that behavior a couple months ago when the merge-check
pseudo-job was implemented to automatically -1 any open changes with
merge conflicts each time a new change merges to their target
branch. This covered the majority of the problems identified by the
freshness check, but without using any of our worker pool.

 This is not solely about finding reviews.  It is about pruning
 stale reviews.  I think the auto-abandon code was excellent at
 doing this, but alas, it is no more. 

I think it was excellent at arbitrarily abandoning open changes
which happened to meet a poorly-thought-out set of criteria. I'm
personally quite glad it broke and we didn't waste time
reimplementing something similar for new Gerrit versions.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] are we going to remove the novaclient v3 shell or what?

2014-09-18 Thread Alex Xu

On 2014年09月18日 18:14, Day, Phil wrote:

-Original Message-
From: Kenichi Oomichi [mailto:oomi...@mxs.nes.nec.co.jp]
Sent: 18 September 2014 02:44
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] are we going to remove the novaclient
v3 shell or what?



-Original Message-
From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
Sent: Wednesday, September 17, 2014 11:59 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [nova] are we going to remove the novaclient v3

shell or what?

This has come up a couple of times in IRC now but the people that
probably know the answer aren't available.

There are python-novaclient patches that are adding new CLIs to the v2
(v1_1) and v3 shells, but now that we have the v2.1 API (v2 on v3) why
do we still have a v3 shell in the client?  Are there plans to remove that?

I don't really care either way, but need to know for code reviews.

One example: [1]

[1] https://review.openstack.org/#/c/108942/

Sorry for a little late response,
I think we don't need new features of v3 into novaclient anymore.
For example, the v3 part of the above[1] was not necessary because a new
feature server-group quota is provided as v2 and v2.1, not v3.

That would be true if there was a version of the client that supported v2.1 
today, but while the V2.1 API is still presented as V3 and doesn't include the 
tenant_id - making the V3 client the only simple way to test new V2.1 features 
in devstack as far as I can see.


How about this as a plan:

1) We add support to the client for --os-compute-api-version=v2.1   which 
maps into the client with the URL set to include v2.1(this won't be usable until we 
do step 2)

+1


2) We change the Nova  to present the v2.1 API  as 
'http://X.X.X.X:8774/v2.1/tenant_id/
  - At this point we will have a working client for all of the stuff that's 
been moved back from V3 to V2.1, but will lose access to any V3 stuff not yet 
moved (which is the opposite of the current state where the v3 client can only 
be used for things that haven't been refactored to V2.1)


Actually we already can access v2.1 API as 
''http://X.X.X.X:8774/v2.1/tenant_id/..'

https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L64



3) We remove V3 from the client.


Until we get 1  2 done, to me it still makes sense to allow small changes into 
the v3 client, so that we keep it usable with the V2.1 API



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Rally] Load testing of dependent resources

2014-09-18 Thread Ilya Shakhat
Ralliers,

I'd like to share with you an idea on how to test dependent resources.

Let's consider the following example: there is a network and ports
belonging to it and we would like to test port creation process. Currently
I see 2 options of writing scenario:
 a) The scenario creates network and then creates a specified number of
ports. Runner executes the scenario specified number of times. -- In this
case ports belonging to different networks are created in parallel, but all
ports belonging to the same network are created serially.
 b) The network already exists and the scenario just creates ports, -- This
case allows to test concurrent creation of ports belonging to the same
network but not between multiple networks.
It would be really cool to get benefits from both cases, i.e. to test
parallel port creation but without limiting to parent network resource.

One of possible approaches may be to construct chain of scenarios where
preceding scenario prepares context for the further one. From the above
example the execution would be:
 1) The task contains sequence of 2 scenarios:
* network creation - it creates network and returns it
* port creation - it picks the network from incoming params and creates
port on it
 2) The runner has execution pipeline (which currently is represented as
generator producing identical scenario runners). The pipeline starts with
sequence of network creation scenario runners. Once one of runner finishes,
it then puts the further scenario and the result into pipeline. The
pipeline ends once no further scenarios left.

From the above example execution flow would be like:
 * pipeline is filled with N network scenarios (not actually filled, but
contains a generator that is able to do this)
 * R runners pick scenarios from pipeline
 * pipeline contains N-R network scenarios
 * one of runners finishes the scenario and updates pipeline with P port
scenarios
 * pipeline contains N-R-1 network scenarios followed by P port scenarios
 * work-work-work until pipeline is empty

This execution covers case from a) and b) but there still will be sequences
of child resources belonging to the same parent. The improvement may be to
pick scenario from pipeline randomly.

How does it sound?

Thanks,
Ilya
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Chmouel Boudjnah
On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft don...@stufft.io wrote:

 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.



I think Thomas was speaking in the context of how OpenStack is used by the
end user and that probably the point of debate here, requests ships
libraries inside to make it easy for users that doen't use Linux distro
packages, when in OpenStack (or at least in prod) packagers are something
we generally very much care about.

Chmouel
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Donald Stufft

 On Sep 18, 2014, at 9:00 AM, Chmouel Boudjnah chmo...@enovance.com wrote:
 
 
 On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft don...@stufft.io 
 mailto:don...@stufft.io wrote:
 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 
 
 I think Thomas was speaking in the context of how OpenStack is used by the 
 end user and that probably the point of debate here, requests ships libraries 
 inside to make it easy for users that doen't use Linux distro packages, when 
 in OpenStack (or at least in prod) packagers are something we generally very 
 much care about.

Even then, my statement holds true, just with different numbers.

Every distribution modifies upstream in different ways, I think it's insane to
do contortions which will break things for people *not* getting things through
those channels. If distributions are going to modify one upstream project they
should expect to need to modify things that depend on that project in ways that
are sensitive to what they've modified.

The only real sane thing IMO is for openstack to consider requests as it is on
PyPI. If openstack wants to make it easier for downstream to de-vendor urllib3
from requests then when openstack wants to import from requests.packages.* it
can instead do:

try:
from requests.packages import urllib3
except ImportError:
import urllib3

This will cause it to work correctly when requests is installed in a pristine
state, and will fallback to handling the modifications that some downstream
redistributors make.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] are we going to remove the novaclient v3 shell or what?

2014-09-18 Thread Ken'ichi Ohmichi
2014-09-18 21:31 GMT+09:00 Alex Xu x...@linux.vnet.ibm.com:
 On 2014年09月18日 18:14, Day, Phil wrote:

 -Original Message-
 From: Kenichi Oomichi [mailto:oomi...@mxs.nes.nec.co.jp]
 Sent: 18 September 2014 02:44
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [nova] are we going to remove the novaclient
 v3 shell or what?


 -Original Message-
 From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
 Sent: Wednesday, September 17, 2014 11:59 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: [openstack-dev] [nova] are we going to remove the novaclient v3

 shell or what?

 This has come up a couple of times in IRC now but the people that
 probably know the answer aren't available.

 There are python-novaclient patches that are adding new CLIs to the v2
 (v1_1) and v3 shells, but now that we have the v2.1 API (v2 on v3) why
 do we still have a v3 shell in the client?  Are there plans to remove
 that?

 I don't really care either way, but need to know for code reviews.

 One example: [1]

 [1] https://review.openstack.org/#/c/108942/

 Sorry for a little late response,
 I think we don't need new features of v3 into novaclient anymore.
 For example, the v3 part of the above[1] was not necessary because a new
 feature server-group quota is provided as v2 and v2.1, not v3.

 That would be true if there was a version of the client that supported
 v2.1 today, but while the V2.1 API is still presented as V3 and doesn't
 include the tenant_id - making the V3 client the only simple way to test new
 V2.1 features in devstack as far as I can see.

v2.1 URL contains tenant_id already, and we can use /v2.1 endpoint like:

$ curl -i 
'http://192.168.11.62:8774/v2.1/05d0f72534e449a0a3e70720c5d7c206/servers/detail'
-X GET -H Accept: applica
tion/json -H X-Auth-Project-Id: demo -H X-Auth-Token:
f41776736ef34e56a7773e2b2d18d2e3
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1749
X-Openstack-Request-Id: req-96768644-be2a-43c4-8757-ad1802736970
Date: Thu, 18 Sep 2014 13:03:54 GMT

{servers: [{status: ACTIVE, updated: 2014-09-18T11:57:12Z,
hostId: 8c63113abf9b1e13f1b11dd6a2dbdf511f7aa2199cff31e847582414,
OS-EXT-SRV-ATTR:host: oomichi-trusty, addresses: {private:
[{version: 4, type: fixed, addr: 10.0.0.14, mac_addr:
fa:16:3e:65:7e:ab}]}, links: [{href:
http://192.168.11.62:8774/v2.1/05d0f72534e449a0a3e70720c5d7c206/servers/bca4e77b-9763-4cf4-8a84-193df620eeb7;,
rel: self}, {href:
http://192.168.11.62:8774/05d0f72534e449a0a3e70720c5d7c206/servers/bca4e77b-9763-4cf4-8a84-193df620eeb7;,
rel: bookmark}], key_name: key-temporary, image: {id:
2ac73d2e-f0d8-4576-a62c-3b53f70d071b, links: [{href:
http://192.168.11.62:8774/05d0f72534e449a0a3e70720c5d7c206/images/2ac73d2e-f0d8-4576-a62c-3b53f70d071b;,
rel: bookmark}]}, os-security-groups:security_groups: [{name:
sg-temporary}], os-pci:pci_devices: [], OS-EXT-STS:task_state:
null, os-extended-availability-zone:availability_zone: nova,
OS-EXT-STS:vm_state: active, OS-EXT-SRV-ATTR:instance_name:
instance-000e, OS-SRV-USG:launched_at:
2014-09-18T11:57:12.00, OS-EXT-SRV-ATTR:hypervisor_hostname:
oomichi-trusty, flavor: {id: 84, links: [{href:
http://192.168.11.62:8774/05d0f72534e449a0a3e70720c5d7c206/flavors/84;,
rel: bookmark}]}, id: bca4e77b-9763-4cf4-8a84-193df620eeb7,
OS-SRV-USG:terminated_at: null, user_id:
30bd25b4555d4664b1069d8d7e682eef, name: vm01, created:
2014-09-18T11:57:03Z, tenant_id:
05d0f72534e449a0a3e70720c5d7c206,
os-extended-volumes:volumes_attached: [], accessIPv4: ,
accessIPv6: , OS-EXT-STS:locked_by: null, progress: 0,
OS-EXT-STS:power_state: 1, config_drive: , metadata: {}}]}
$

DevStack doesn't register v2.1 endpoint to keytone now, but we can use
it with calling it directly.
It is true that it is difficult to use v2.1 API now and we can check
its behavior via v3 API instead.

 How about this as a plan:

 1) We add support to the client for --os-compute-api-version=v2.1
 which maps into the client with the URL set to include v2.1(this won't
 be usable until we do step 2)

v2.1 behavior(API URLs except the endpoint, request/response bodies,
status codes) should
be the same as v2. So if devstack registers v2.1 endpoint as the type
computev21, we can
use it by specifying computev21 as --service-type of current nova command.


 2) We change the Nova  to present the v2.1 API  as
 'http://X.X.X.X:8774/v2.1/tenant_id/
   - At this point we will have a working client for all of the stuff
 that's been moved back from V3 to V2.1, but will lose access to any V3 stuff
 not yet moved (which is the opposite of the current state where the v3
 client can only be used for things that haven't been refactored to V2.1)


 Actually we already can access v2.1 API as
 ''http://X.X.X.X:8774/v2.1/tenant_id/..'
 https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L64

Yes, right as I mentioned at the above.

Thanks
Ken'ichi Ohmichi


[openstack-dev] [blazar]: proposal for a new lease type

2014-09-18 Thread Lisa

Hi all,

my name is Lisa Zangrando and I work at the Italian National Institute 
for Nuclear Physics (INFN). In particular I am leading a team which is 
addressing the issue concerning the efficiency in the resource usage in 
OpenStack.
Currently OpenStack allows just a static partitioning model where the 
resource allocation to the user teams (i.e. the projects) can be done 
only by considering fixed quotas which cannot be exceeded even if there 
are unused resources (but) assigned to different projects.
We studied the available BLAZAR's documentation and, in agreement with 
Tim Bell (who is responsible the OpenStack cloud project at CERN), we 
think this issue could be addressed within your framework.
Please find attached a document that describes our use cases (actually 
we think that many other environments have to deal with the same 
problems) and how they could be managed in BLAZAR, by defining a new 
lease type (i.e. fairShare lease) to be considered as extension of the 
list of the already supported lease types.

I would then be happy to discuss these ideas with you.

Thanks in advance,
Lisa



Use cases for a new lease type in BLAZAR.pdf
Description: Adobe PDF document
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [requirements][horizon] Dependency freeze exceptions

2014-09-18 Thread Radomir Dopieralski
Hello,

I would like to request dependency freeze exceptions for the following
patches for Horizon:

https://review.openstack.org/#/c/121509/
https://review.openstack.org/#/c/122410/

and

https://review.openstack.org/#/c/113184/

They are all fixing high priority bugs. The first two are related to
bugs with parsing Bootstrap 3.2.0 scss files that have been fixed
upstream. The third one makes the life of packagers a little eaiser,
by using versions that both Debian and Fedora, and possibly many
other distros, can ship.

I am not sure what the formal process for such an exception should be,
so I'm writing here. Please let me know if I should have done it
differently.
-- 
Radomir Dopieralski

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Neutron][L3] Sub team meeting cancelled

2014-09-18 Thread Carl Baldwin
I have a conflict today.  Keep working on RC1.

Carl
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [blazar]: proposal for a new lease type

2014-09-18 Thread Sylvain Bauza


Le 18/09/2014 15:27, Lisa a écrit :

Hi all,

my name is Lisa Zangrando and I work at the Italian National Institute 
for Nuclear Physics (INFN). In particular I am leading a team which is 
addressing the issue concerning the efficiency in the resource usage 
in OpenStack.
Currently OpenStack allows just a static partitioning model where the 
resource allocation to the user teams (i.e. the projects) can be done 
only by considering fixed quotas which cannot be exceeded even if 
there are unused resources (but) assigned to different projects.
We studied the available BLAZAR's documentation and, in agreement with 
Tim Bell (who is responsible the OpenStack cloud project at CERN), we 
think this issue could be addressed within your framework.
Please find attached a document that describes our use cases (actually 
we think that many other environments have to deal with the same 
problems) and how they could be managed in BLAZAR, by defining a new 
lease type (i.e. fairShare lease) to be considered as extension of the 
list of the already supported lease types.

I would then be happy to discuss these ideas with you.

Thanks in advance,
Lisa



Hi Lisa,

Glad to see you're interested in Blazar.

I tried to go thru your proposal, but could you please post the main 
concepts of what you plan to add into an etherpad and create a blueprint 
[1] mapped to it so we could discuss on the implementation ?
Of course, don't hesitate to ping me or the blazar community in 
#openstack-blazar if you need help with the process or the current 
Blazar design.


Thanks,
-Sylvain

[1] https://blueprints.launchpad.net/blazar/




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Oslo final releases ready

2014-09-18 Thread Doug Hellmann
All of the final releases for the Oslo libraries for the Juno cycle are 
available on PyPI. I’m working on a couple of patches to the global 
requirements list to update the baseline in the applications. In all cases, the 
final release is a second tag on a previously released version.

- oslo.config - 1.4.0 (same as 1.4.0.0a5)
- oslo.db - 1.0.0 (same as 0.5.0)
- oslo.i18n - 1.0.0 (same as 0.4.0)
- oslo.messaging - 1.4.0 (same as 1.4.0.0a5)
- oslo.rootwrap - 1.3.0 (same as 1.3.0.0a3)
- oslo.serialization - 1.0.0 (same as 0.3.0)
- oslosphinx - 2.2.0 (same as 2.2.0.0a3)
- oslotest - 1.1.0 (same as 1.1.0.0a2)
- oslo.utils - 1.0.0 (same as 0.3.0)
- cliff - 1.7.0 (previously tagged, so not a new release)
- stevedore - 1.0.0 (same as 1.0.0.0a2)

Congratulations and *Thank You* to the Oslo team for doing an amazing job with 
graduations this cycle!

Doug


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Gordon Sim

On 09/18/2014 12:31 PM, Flavio Percoco wrote:

On 09/17/2014 10:36 PM, Joe Gordon wrote:

My understanding of Zaqar is that it's like SQS. SQS uses distributed
queues, which have a few unusual properties [0]:


 Message Order

Amazon SQS makes a best effort to preserve order in messages, but due to
the distributed nature of the queue, we cannot guarantee you will
receive messages in the exact order you sent them. If your system
requires that order be preserved, we recommend you place sequencing
information in each message so you can reorder the messages upon receipt.



Zaqar guarantees FIFO. To be more precise, it does that relying on the
storage backend ability to do so as well. Depending on the storage used,
guaranteeing FIFO may have some performance penalties.


Would it be accurate to say that at present Zaqar does not use 
distributed queues, but holds all queue data in a storage mechanism of 
some form which may internally distribute that data among servers but 
provides Zaqar with a consistent data model of some form?


[...]

As of now, Zaqar fully relies on the storage replication/clustering
capabilities to provide data consistency, availability and fault
tolerance.


Is the replication synchronous or asynchronous with respect to client 
calls? E.g. will the response to a post of messages be returned only 
once the replication of those messages is confirmed? Likewise when 
deleting a message, is the response only returned when replicas of the 
message are deleted?



However, as far as consuming messages is concerned, it can
guarantee once-and-only-once and/or at-least-once delivery depending on
the message pattern used to consume messages. Using pop or claims
guarantees the former whereas streaming messages out of Zaqar guarantees
the later.


From what I can see, pop provides unreliable delivery (i.e. its similar 
to no-ack). If the delete call using pop fails while sending back the 
response, the messages are removed but didn't get to the client.


What do you mean by 'streaming messages'?

[...]

Based on our short conversation on IRC last night, I understand you're
concerned that FIFO may result in performance issues. That's a valid
concern and I think the right answer is that it depends on the storage.
If the storage has a built-in FIFO guarantee then there's nothing Zaqar
needs to do there. In the other hand, if the storage does not have a
built-in support for FIFO, Zaqar will cover it in the driver
implementation. In the mongodb driver, each message has a marker that is
used to guarantee FIFO.


That marker is a sequence number of some kind that is used to provide 
ordering to queries? Is it generated by the database itself?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] PostgreSQL jobs slow in the gate

2014-09-18 Thread Matt Riedemann



On 9/18/2014 5:49 AM, Sean Dague wrote:

On 09/17/2014 11:50 PM, Clark Boylan wrote:

On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:

On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:



On 9/17/2014 7:59 PM, Ian Wienand wrote:

On 09/18/2014 09:49 AM, Clark Boylan wrote:

Recent sampling of test run times shows that our tempest jobs run
against clouds using PostgreSQL are significantly slower than jobs run
against clouds using MySQL.


FYI There is a possibly relevant review out for max_connections limits
[1], although it seems to have some issues with shmem usage

-i

[1] https://review.openstack.org/#/c/121952/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



That's a backport of a fix from master where we were hitting fatal
errors due to too many DB connections which was brought on by the
changes to cinder and glance to run as many workers as there were CPUs
available.  So I don't think it probably plays here...

The errors pointed out in another part of the thread have been around
for awhile, I think they are due to negative tests where we're hitting
unique constraints because of the negative tests, so they are expected.

We should also note that the postgresql jobs run with the nova metadata
API service, I'm not sure how much of a factor that would have here.

Is there anything else unique about those jobs from the MySQL ones?


Good question. There are apparently other differences. The postgres job
runs Keystone under eventlet instead of via apache mod_wsgi. It also
sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
I can find is the one you point out, nova api metadata service is run as
an independent thing.

Could these things be related? It would be relatively simple to push a
change or two to devstack-gate to test this but there are enough options
here that I probably won't do that until we think at least one of these
options is at fault.

I am starting to feel bad that I picked on PostgreSQL and completely
forgot that there were other items in play here. I went ahead and
uploaded [0] to run all devstack jobs without keystone wsgi services
(eventlet) and [1] to run all devstack job with keystone wsgi services
and the initial results are pretty telling.

It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.

Thank you Matt for pointing me in this direction.

[0] https://review.openstack.org/#/c/122299/
[1] https://review.openstack.org/#/c/122300/


Don't feel bad. :)

The point that Clark highlights here is a good one. There is an
assumption that once someone creates a job in infra, the magic elves are
responsible for it.

But there are no magic elves. So jobs like this need sponsors.

Maybe the right thing to do is not conflate this configuration and put
an eventlet version of the keystone job only on keystone (because the
keystone team was the one that proposed having a config like that, but
it's so far away from their project they aren't ever noticing when it's
regressing).

Same issue with the metadata server split. That's really only a thing
Nova cares about. It shouldn't impact anyone else.

-Sean



Neutron cares about the nova metadata API service right?

--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] PostgreSQL jobs slow in the gate

2014-09-18 Thread Matt Riedemann



On 9/18/2014 12:35 AM, Morgan Fainberg wrote:

-Original Message-
From: Dean Troyer dtro...@gmail.com
Reply: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Date: September 17, 2014 at 21:21:47
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Subject:  Re: [openstack-dev] PostgreSQL jobs slow in the gate



Clark Boylan Wrotr :


It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.




On Wed, Sep 17, 2014 at 11:02 PM, Morgan Fainberg 
morgan.fainb...@gmail.com wrote:


I've kicked off a test[1] as well to check into some tunable options
(eventlet workers) for improving keystone eventlet performance. I'll circle
back with the infra team once we have a little more data on both fronts.
The Keystone team will use this data to figure out the best way to approach
this issue.



Brant submitted https://review.openstack.org/#/c/121384/ to up the Keystone
workers when API_WORKERS is set.

I submitted https://review.openstack.org/#/c/122013/ to set a scaling
default for API_WORKERS based on the available CPUs ((nproc+1)/2). There
is a summary in that commit message of the current reviews addressing the
workers in various projects.

I think it has become clear that DevStack needs to set a default for most
services that are currently either too big (nproc) or too small (Keystone
at 1). Of course, moving things to mod_wsgi moots all of that, but it'll
be a while before everything moves.

dt

--

Dean Troyer
dtro...@gmail.com


Dean,

We should probably look at increasing the default number of workers as well in 
the Keystone configuration (rather than just devstack). It looks like, with 
limited datasets, we are seeing a real improvement with Keystone and 4 workers 
(from my previously mentioned quick test). Thanks for the extra data points. 
This helps to confirm the issue is Keystone under eventlet.

Cheers,
Morgan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I pointed this out in Brant's devstack patch but we had a product team 
internally bring up this same point in Icehouse, they were really 
limited due to the eventlet workers issue in Keystone and once we 
provided the option (backported) it increased their throughput by 20%. 
We've been running with that in our internal Tempest runs (setting 
workers equal to number of CPUs / 2) and so far so good.


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Chris St. Pierre
On Thu, Sep 18, 2014 at 4:19 AM, Ken'ichi Ohmichi ken1ohmi...@gmail.com
wrote:

 Correct validation history is

 Essex: use anything you want!
 Folsom: strict asci!
 [..]
 Juno: strict asci!


I'm not sure that's quite right. My patch doesn't actually add Unicode
support; that was already added in
825499fffc7a320466e976d2842e175c2d158c0e, which appears to have gone in for
Icehouse.  So:

Essex: Use anything you want
Folsom: Strict ASCII, inconsistent restrictions
Grizzly: Strict ASCII, inconsistent restrictions
Icehouse: Unicode, inconsistent restrictions
Juno: Unicode, consistent restrictions
Kilo (?): Use anything you want

At any rate, if accepting Unicode is an issue, then it's been an issue for
a while.

-- 
Chris St. Pierre
Senior Software Engineer
metacloud.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] zeromq work for kilo

2014-09-18 Thread Kapil Thangavelu
On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco fla...@redhat.com wrote:

 On 09/17/2014 04:34 PM, Doug Hellmann wrote:
  This thread [1] has turned more “future focused, so I’m moving the
 conversation to the -dev list where we usually have those sorts of
 discussions.
 
  [1]
 http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
 


 I saw this mentioned in the `openstack` thread but I'd like us to
 reconsider it.

 Since we've gone through the hey please don't deprecate it, I'll help
 process a couple of times with the zmq driver, I'm thinking we should
 pull it out of the code base anyway.


I think the primary issue has been two fold, one is the lack of
reviews/merges on extant patches that fix critical items that have been
outstanding for months. I think that is compounded by the other issue which
was the lack of tests. We've sprinted last week on adding in both unit
tests, the extant patches and functionally verifying them by automating
cloud deployments with zmq for the messaging driver. We're also committing
as a company to supporting it on an ongoing basis. If we get the gates
going for kilo, i don't see any reason for the churn below, if the gates
don't get going we can yank to external in kilo anyway.

cheers,

Kapil


 Here's a rough plan of what I think we should do until the zmq is
 updated and has a proper gate working:

 1. Pull it out the code base into its own repo (stackforge?)
 2. Setup the basic `oslo.messaging` gates for it
 3. Release the driver on pypi as: `oslo.messaging.zmq`
 4. Make lots of noise and make sure people using zmq knows that they
 have to install a separate package now. (I know this means creating a
 new package for many distros).

 If there are folks interested in maintaining this driver, they can still
 do it with the driver outside the code base. If it ever gets enough
 attention and maturity, it could be re-proposed for inclusion into the
 code base.

 I know there are folks interested in a broker-less solution and we now
 have the not-tested-in-production-yet amqp1.0 driver which I think is a
 very good solution and also shares most of the semantics we rely on
 protocol-wise.

 I didn't go through the whole thread so, I'm sorry if I'm repeating
 something that has already been said/proposed/discussed.

 Flavio

  On Sep 17, 2014, at 7:54 AM, James Page james.p...@ubuntu.com wrote:
 
  Signed PGP part
  Hi Li
 
  On 17/09/14 11:58, Li Ma wrote:
  The scale potential is very appealing and is something I want to
  test - - hopefully in the next month or so.
 
  Canonical are interested in helping to maintain this driver and
  hopefully we help any critical issues prior to Juno release.
 
 
  That sounds good. I just went through all the bugs reported in the
  community.
 
  The only critical bug which makes ZeroMQ malfunction is
  https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
  corresponding review is under way:
  https://review.openstack.org/#/c/84938/
 
  Agreed
 
  Others are tagged to 'zmq' in
  https://bugs.launchpad.net/oslo.messaging
 
  Looking through Doug's suggested list of information and collating
  what I know of from our work last week:
 
  1) documentation for how to configure and use zeromq with
  oslo.messaging (note, not the version in oslo-incubator, the version
  in the messaging library repository)
 
  As part of our sprint, I worked on automating deployment of OpenStack
  + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
  that work into some general documentation on how best to configure
  ZeroMQ with OpenStack - the current documentation is a bit raw and
  does not talk about how to configure the oslo-messaging-zmq-receiver
  at all.
 
  I also plan some packaging updates for Debian/Ubuntu in our next dev
  cycle to make this a little easier to configure and digest - for
  example, right now no systemd unit/upstart configuration/sysv init
  script is provided to manage the zmq-receiver.
 
  I'd also like to document the current design of the ZMQ driver - Doug
  - where is the best place todo this? I thought in the source tree
  somewhere.
 
  The documentation in the oslo.messaging repository [2] would be a good
 place to start for that. If we decide deployers/operators need the
 information we can either refer to it from the guides managed by the
 documentation team or we can move/copy the information. How about if you
 start a new drivers subdirectory there, and add information about zmq. We
 can have other driver authors provide similar detail about their drivers in
 the same directory.
 
  [2]
 http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
 
 
  2) a list of the critical bugs that need to be fixed + any existing
  patches associated with those bugs, so they can be reviewed early in
 kilo
 
  This blocks operation of nova+neutron environements:
 
  https://bugs.launchpad.net/oslo.messaging/+bug/1301723
   Summary: Message was sent to wrong node with 

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Clint Byrum
Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
 
  On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
  
  
  Linux distributions are not the end be all of distribution models and
  they don’t get to dictate to upstream.
  
  Well, distributions is where the final user is, and where software gets
  consumed. Our priority should be the end users.
 
 
 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 

Do pypi users not also need to be able to detect and fix any versions
of libraries they might have? If one has some virtualenvs with various
libraries and apps installed and no --system-site-packages, one would
probably still want to run 'pip freeze' in all of them and find out what
libraries are there and need to be fixed.

Anyway, generally security updates require a comprehensive strategy.
One common comprehensive strategy is version assertion.

Vendoring complicates that immensely.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Ian Cordasco
On 9/17/14, 9:24 PM, Thomas Goirand z...@debian.org wrote:

On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
 I think that all of the conversation to this point has been valuable,
 the general consensus is vendoring a library is not as desirable as
 using it strictly as a dependency. It would be nice in a perfect
 world if vendoring wasn’t and issue, but in this case I think the
 root of the matter is that Debian un-vendors urllib3 and we have
 referenced the vendored urllib3 instead of installing and utilizing
 urllib3 directly.
 
 This poses at least one problem for us: we are not able to guarantee
 we’re using the same urllib3 library as requests is. I am unsure how
 big of a deal this ends up being, but it is a concern and has brought
 up a question of how to handle this in the most appropriate and
 consistent way across all of the distributions we as OpenStack support.
 
 Does this make requests a bad library we should toss aside for
 something else? Instead of being concerned with the reasons for
 vendoring urllib3 (or un-vendoring it) we should shift the conversation
 towards two questions:
 
 1. Is it a real issue if the version of urllib3 is mismatched between
 our client libraries and requests?
 2. If it is a real issue how are we solving it?

The main issue is that urllib3 in requests, as other pointed out, is not
up-to-date, and will not be updated. In fact, that's the main reason why
the upstream authors of requests are vendorizing: it's because they
don't want to carry the burden of staying up-to-date.

How involved are you with requests’ development process? You must not be
very involved because this is the exact opposite reason of why we vendor.
More often that not we pull in urllib3 to get unreleased features that we
build upon for our newest release. If what you said was true, 2.4.{0,1}
would not have the ability to pass socket level options that nova client
decides it wants to set. If we were pinning to an old version of urllib3,
this feature would never be possible. We vendor, because in order to
provide these features we don’t want to have to support the ancient
versions of urllib3 that used to cause us headaches on platforms like
Debian.

And then, there's incompatibilities and divergences that appear, leading
to all sorts of unexpected issues, like one thing working with pip, but
not with the packages. This kind of issues are very hard to understand
and debug. Distributions may report the issue upstream, then upstream
will say but it's working for me, and then we may loose a lot of time.
This happened already, and may happen again if we don't care enough.

 Obviously we can work with the requests team to figure out the best
 approach.

There's only a single approach that works: have the requests upstream
authors to stop embedding foreign code, and use the dependency instead.

 We should focus on the solution here rather than continuing
 down the path of whether requests should/shouldn’t be vendoring it’s
 dependencies since it is clear that the team has their reasons and
 does not want to switch to the dependency model again.

I'm sure they have tons of wrong reasons. If they don't want to change
anything, then we can only try to work around the issue, and never use
the embedded version.

The fact is, anyone who doesn’t run their tests in Devstack runs them in a
virtual environment. The line you’re complaining about is actually correct
in that context (the context where pip installs requests). That said, the
test should instead use a conditional import and fallback to the vendored
copy.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Ian Cordasco


On 9/18/14, 2:27 AM, Chmouel Boudjnah chmo...@enovance.com wrote:

Ian Cordasco ian.corda...@rackspace.com writes:

 urllib3 do that automatically. I haven’t started to investigate exactly
 why they do this. Likewise, glance client has custom certificate
 verification in glanceclient.common.https. Why? I’m not exactly certain

this probably come from pre-requests port uses when it was using httplib
which didn't have SSL verification. There is a old wiki page about it
here https://wiki.openstack.org/wiki/SecureClientConnections

Slightly off-topic, speaking about requests and OpenStack client, it
would be nice to have Expect/100-Continue feature tackled for
glanceclient and swiftclient :

https://github.com/kennethreitz/requests/issues/713

I’m glad you linked to that issue Chmouel because it has all of the
information you need to realize that it’s an entirely impossible feature
while still relying on httplib. You can still send that header, but
requests has no meaningful way of special-casing it. The standard library
does not return the header we need to know that we can continue with the
upload and the RFC is wonderfully obscure enough to make requests’ current
behavior correct. It says a client needs to wait for the server to respond
or some amount of time before starting the upload. Granted we don’t call
“sleep” and start the upload immediately, but we have no way of
determining if the server has responded thanks to httplib.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Clint Byrum
Great job highlighting what our friends over at Amazon are doing.

It's clear from these snippets, and a few other pieces of documentation
for SQS I've read, that the Amazon team approached SQS from a _massive_
scaling perspective. I think what may be forcing a lot of this frustration
with Zaqar is that it was designed with a much smaller scale in mind.

I think as long as that is the case, the design will remain in question.
I'd be comfortable saying that the use cases I've been thinking about
are entirely fine with the limitations SQS has.

Excerpts from Joe Gordon's message of 2014-09-17 13:36:18 -0700:
 Hi All,
 
 My understanding of Zaqar is that it's like SQS. SQS uses distributed
 queues, which have a few unusual properties [0]:
 Message Order
 
 Amazon SQS makes a best effort to preserve order in messages, but due to
 the distributed nature of the queue, we cannot guarantee you will receive
 messages in the exact order you sent them. If your system requires that
 order be preserved, we recommend you place sequencing information in each
 message so you can reorder the messages upon receipt.
 At-Least-Once Delivery
 
 Amazon SQS stores copies of your messages on multiple servers for
 redundancy and high availability. On rare occasions, one of the servers
 storing a copy of a message might be unavailable when you receive or delete
 the message. If that occurs, the copy of the message will not be deleted on
 that unavailable server, and you might get that message copy again when you
 receive messages. Because of this, you must design your application to be
 idempotent (i.e., it must not be adversely affected if it processes the
 same message more than once).
 Message Sample
 
 The behavior of retrieving messages from the queue depends whether you are
 using short (standard) polling, the default behavior, or long polling. For
 more information about long polling, see Amazon SQS Long Polling
 http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
 .
 
 With short polling, when you retrieve messages from the queue, Amazon SQS
 samples a subset of the servers (based on a weighted random distribution)
 and returns messages from just those servers. This means that a particular
 receive request might not return all your messages. Or, if you have a small
 number of messages in your queue (less than 1000), it means a particular
 request might not return any of your messages, whereas a subsequent request
 will. If you keep retrieving from your queues, Amazon SQS will sample all
 of the servers, and you will receive all of your messages.
 
 The following figure shows short polling behavior of messages being
 returned after one of your system components makes a receive request.
 Amazon SQS samples several of the servers (in gray) and returns the
 messages from those servers (Message A, C, D, and B). Message E is not
 returned to this particular request, but it would be returned to a
 subsequent request.
 
 
 
 Presumably SQS has these properties because it makes the system scalable,
 if so does Zaqar have the same properties (not just making these same
 guarantees in the API, but actually having these properties in the
 backends)? And if not, why? I looked on the wiki [1] for information on
 this, but couldn't find anything.
 
 
 
 
 
 [0]
 http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html
 [1] https://wiki.openstack.org/wiki/Zaqar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Donald Stufft

 On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote:
 
 Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
 
 On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
 
 
 Linux distributions are not the end be all of distribution models and
 they don’t get to dictate to upstream.
 
 Well, distributions is where the final user is, and where software gets
 consumed. Our priority should be the end users.
 
 
 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 
 
 Do pypi users not also need to be able to detect and fix any versions
 of libraries they might have? If one has some virtualenvs with various
 libraries and apps installed and no --system-site-packages, one would
 probably still want to run 'pip freeze' in all of them and find out what
 libraries are there and need to be fixed.
 
 Anyway, generally security updates require a comprehensive strategy.
 One common comprehensive strategy is version assertion.
 
 Vendoring complicates that immensely.

It doesn’t really matter. PyPI doesn’t dictate to projects who host there what
that project is allowed to do except in some very broad circumstances. Whether
or not requests *should* do this doesn't really have any bearing on what
Openstack should do to cope with it. The facts are that requests does it, and
that people pulling things from PyPI is an actual platform that needs thought
about.

This leaves Openstack with a few reasonable/sane options:

1) Decide that vendoring in requests is unacceptable to what Openstack as a
   project is willing to support, and cease the use of requests.
2) Decide that what requests offers is good enough that it outweighs the fact
   that it vendors urllib3 and continue using it.

If the 2nd option is chosen, then doing anything but supporting the fact that
requests vendors urllib3 within the code that openstack writes is hurting the
users who fetch these projects from PyPI because you don't agree with one of
the choices that requests makes. By all means do conditional imports to lessen
the impact that the choice requests has made (and the one that Openstack has
made to use requests) on downstream distributors, but unconditionally importing
from the top level urllib3 for use within requests is flat out wrong.

Obviously neither of these options excludes the choice to lean on requests to
reverse this decision as well. However that is best done elsewhere as the
person making that decision isn't a member of these mailing lists as far as
I am aware.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron][db] Ensuring db isolation between tests

2014-09-18 Thread Mike Bayer
I’ve done a lot of work on this issue and from my perspective, the code is 
mostly ready to go, however we’re in an extended phase of getting folks to sign 
off as well as that I’m waiting for some last-minute fixup from Robert Collins. 
  Patch: [1]  Blueprint, which is to be moved to Kilo: [2]

A “nested” transaction can actually mean one of two things, either a real 
SAVEPOINT, or a logical transaction block.  However, either kind of block will 
be removed if a ROLLBACK is emitted, which also rolls back the actual 
transactional state all the way to the end.The patch above makes this work 
as the result of two fixes.  One is that I replaced the system by which we do 
transactions with the pysqlite driver so that SAVEPOINT actually works [3].  
The other is that I created a comprehensive fixture in [1] that will maintain a 
transaction + savepoint block at all times, working smoothly with tests that 
call commit or rollback any number of times.

From an isolation perspective, we create on-demand databases per process, so 
that each serial test process uses a distinct database per backend.   The 
entire process is managed by testresources which will organize the order of 
tests to most effectively leave a single schema in place with minimal 
teardown/setup.

I’m hoping that my patches can go right in at the top of Kilo and we can begin 
rolling it out in projects that are participating in oslo.db, with the hopes 
that consuming projects will be able to remove a lot of boilerplate 
setup/teardown code. 


1: https://review.openstack.org/#/c/120870/  
2: https://review.openstack.org/#/c/117335/ 
3: https://review.openstack.org/#/c/113152/


On Sep 18, 2014, at 5:59 AM, Salvatore Orlando sorla...@nicira.com wrote:

 Nested commits in sqlalchemy should be seen as a single transaction on the 
 backend, shouldn't they?
 I don't know anything about this specific problem, but the fact that unit 
 tests use sqlite might be a reason, since it's not really a full DBMS...
 
 I think that wrapping tests in transaction also will require some changes in 
 the architecture of the tests themselves, as many tests call the API router 
 or the plugin which then gets a db session and open a new transaction. 
 Furthermore, it will change the test behaviour possibly hiding errors; some 
 operations indeed perform several distinct transactions, which in this case 
 will be seen a single transaction.
 
 What Kevin is doing here I think was the original way we used to do that in 
 Neutron (Folsom). Then at some point we realised that due to plugin schema 
 differences we were laving tables behind and switched to drop_all and 
 rebuilding the schema using autogeneration at each test.
 
 I think it should be ok to merge this patch. I will hold off the +A to give 
 other core reviewers a chance to look at it.
 
 Salvatore
 
 
 On 18 September 2014 11:44, Maru Newby ma...@redhat.com wrote:
 For legacy reasons, the Neutron test suite creates and destroys a db for each 
 test.  There is a patch proposed to create the tables once and then ensure 
 the tables are wiped at the end of each test [1], providing a performance 
 improvement of ~10%.  I was wondering if this is the best way to provide 
 isolation, since I’ve heard that isolation via per-test transactions should 
 also work.  The patch author reported problems with this approach - 
 apparently nested commits were not being rolled back.  Is there some trick to 
 isolating with transactions that wouldn’t be immediately obvious?
 
 Thanks,
 
 
 Maru
 
 1: https://review.openstack.org/#/c/122028/
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Ian Cordasco
On 9/18/14, 9:18 AM, Clint Byrum cl...@fewbar.com wrote:

Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
 
  On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
  
  
  Linux distributions are not the end be all of distribution models and
  they don’t get to dictate to upstream.
  
  Well, distributions is where the final user is, and where software
gets
  consumed. Our priority should be the end users.
 
 
 Distributions are not the only place that people get their software
from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 

Do pypi users not also need to be able to detect and fix any versions
of libraries they might have? If one has some virtualenvs with various
libraries and apps installed and no --system-site-packages, one would
probably still want to run 'pip freeze' in all of them and find out what
libraries are there and need to be fixed.

Anyway, generally security updates require a comprehensive strategy.
One common comprehensive strategy is version assertion.

Vendoring complicates that immensely.

Except that even OpenStack doesn’t pin requests because of how
extraordinarily stable our API is. While you can argue that Kenneth has
non-standard opinions about his library, Cory and I take backwards
compatibility and stability very seriously. This means anyone can upgrade
to a newer version of requests without worrying that it will be backwards
incompatible. 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread James Polley




On 18 Sep 2014, at 6:06 pm, Sullivan, Jon Paul jonpaul.sulli...@hp.com wrote:

 -Original Message-
 From: James E. Blair [mailto:cor...@inaugust.com]
 Sent: 17 September 2014 23:24
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [TripleO] Set WIP for stale patches?
 
 Sullivan, Jon Paul jonpaul.sulli...@hp.com writes:
 
 I think this highlights exactly why this should be an automated
 process.  No errors in application, and no errors in interpretation of
 what has happened.
 
 So the -1 from Jenkins was a reaction to the comment created by adding
 the workflow -1.  This is going to happen on all of the patches that
 have their workflow value altered (tests will run, result would be
 whatever the result of the test was, of course).
 
 Jenkins only runs tests in reaction to comments if they say recheck.
 
 This is not my experience.
 
 In my experience if the check results are not fresh enough the recheck is 
 automatically run.  I am not on the infra team, so without looking up code I 
 am just guessing, but my guess is that the workflow score change triggers the 
 check on the presumption that it has been approved, not accounting for the 
 recent(ish) update that move wip to the workflow score.

 
 But I also agree that the Jenkins vote should not be included in the
 determination of marking a patch WIP, but a human review should (So
 Code-Review and not Verified column).
 
 And in fact, for the specific example to hand, the last Jenkins vote
 was actually a +1, so as I understand it should not have been marked
 WIP.
 
 I'd like to help you see the reviews you want to see without
 externalizing your individual workflow onto contributors.  What tool do
 you use to find reviews?
 
 If it's gerrit's webui, have you tried using the Review Inbox dashboard?
 Here it is for the tripleo-image-elements project:
 
  https://review.openstack.org/#/projects/openstack/tripleo-image-
 elements,dashboards/important-changes:review-inbox-dashboard
 
 If you would prefer something else, we can customize those dashboards to
 do whatever you want, including ignoring changes that have not been
 updated in 2 weeks.
 
 This is not solely about finding reviews.  It is about pruning stale reviews. 
  I think the auto-abandon code was excellent at doing this, but alas, it is 
 no more. 

The only reason i'm aware of for pruning apparently stale reviews is to make it 
easier to find and review things that are being actively worked on.

Do you have other reasons for wanting stale reviews pruned?
 
 
 -Jim
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 Thanks, 
 Jon-Paul Sullivan ☺ Cloud Services - @hpcloud
 
 Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, 
 Galway.
 Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's 
 Quay, Dublin 2. 
 Registered Number: 361933
 
 The contents of this message and any attachments to it are confidential and 
 may be legally privileged. If you have received this message in error you 
 should delete it from your system immediately and advise the sender.
 
 To any recipient of this message within HP, unless otherwise stated, you 
 should consider this message and attachments as HP CONFIDENTIAL.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/18/2014 04:09 PM, Gordon Sim wrote:
 On 09/18/2014 12:31 PM, Flavio Percoco wrote:
 On 09/17/2014 10:36 PM, Joe Gordon wrote:
 My understanding of Zaqar is that it's like SQS. SQS uses distributed
 queues, which have a few unusual properties [0]:


  Message Order

 Amazon SQS makes a best effort to preserve order in messages, but due to
 the distributed nature of the queue, we cannot guarantee you will
 receive messages in the exact order you sent them. If your system
 requires that order be preserved, we recommend you place sequencing
 information in each message so you can reorder the messages upon
 receipt.


 Zaqar guarantees FIFO. To be more precise, it does that relying on the
 storage backend ability to do so as well. Depending on the storage used,
 guaranteeing FIFO may have some performance penalties.
 
 Would it be accurate to say that at present Zaqar does not use
 distributed queues, but holds all queue data in a storage mechanism of
 some form which may internally distribute that data among servers but
 provides Zaqar with a consistent data model of some form?

I think this is accurate. The queue's distribution depends on the
storage ability to do so and deployers will be able to choose what
storage works best for them based on this as well. I'm not sure how
useful this separation is from a user perspective but I do see the
relevance when it comes to implementation details and deployments.


 [...]
 As of now, Zaqar fully relies on the storage replication/clustering
 capabilities to provide data consistency, availability and fault
 tolerance.
 
 Is the replication synchronous or asynchronous with respect to client
 calls? E.g. will the response to a post of messages be returned only
 once the replication of those messages is confirmed? Likewise when
 deleting a message, is the response only returned when replicas of the
 message are deleted?

It depends on the driver implementation and/or storage configuration.
For example, in the mongodb driver, we use the default write concern
called acknowledged. This means that as soon as the message gets to
the master node (note it's not written on disk yet nor replicated) zaqar
will receive a confirmation and then send the response back to the
client. This is also configurable by the deployer by changing the
default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0].

[0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w

 
 However, as far as consuming messages is concerned, it can
 guarantee once-and-only-once and/or at-least-once delivery depending on
 the message pattern used to consume messages. Using pop or claims
 guarantees the former whereas streaming messages out of Zaqar guarantees
 the later.
 
 From what I can see, pop provides unreliable delivery (i.e. its similar
 to no-ack). If the delete call using pop fails while sending back the
 response, the messages are removed but didn't get to the client.

Correct, pop works like no-ack. If you want to have pop+ack, it is
possible to claim just 1 message and then delete it.

 
 What do you mean by 'streaming messages'?

I'm sorry, that went out wrong. I had the browsability term in my head
and went with something even worse. By streaming messages I meant
polling messages without claiming them. In other words, at-least-once is
guaranteed by default, whereas once-and-only-once is guaranteed just if
claims are used.

 
 [...]
 Based on our short conversation on IRC last night, I understand you're
 concerned that FIFO may result in performance issues. That's a valid
 concern and I think the right answer is that it depends on the storage.
 If the storage has a built-in FIFO guarantee then there's nothing Zaqar
 needs to do there. In the other hand, if the storage does not have a
 built-in support for FIFO, Zaqar will cover it in the driver
 implementation. In the mongodb driver, each message has a marker that is
 used to guarantee FIFO.
 
 That marker is a sequence number of some kind that is used to provide
 ordering to queries? Is it generated by the database itself?

It's a sequence number to provide ordering to queries, correct.
Depending on the driver, it may be generated by Zaqar or the database.
In mongodb's case it's generated by Zaqar[0].

[0]
https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] Set WIP for stale patches?

2014-09-18 Thread James E. Blair
Sullivan, Jon Paul jonpaul.sulli...@hp.com writes:

 This is not solely about finding reviews.  It is about pruning stale
 reviews.  I think the auto-abandon code was excellent at doing this,
 but alas, it is no more.

What's the purpose of pruning stale reviews?  I've read the IRC log of
the meeting you mentioned.  It's becoming apparent to me that this is
about making some numbers that reviewstats produces look good.

I am rather disappointed in that.

If reviewstats is not measuring what you want it to measure (eg, how
well you keep up with incoming reviews) then you should change how it
measures it.  If you want the number of open reviews to be low, then
change the definition of open reviews to what you think it should be --
don't create automated processes to WIP changes just to get your numbers
down.

The reason that we made it so that any core team member could WIP or
abandon a change was so that you could make a judgement call and say
this change needs more work or this change is defunct.  You might
even use a tool to help you find those reviews and make those judgement
calls.  But no automated tool can make those decisions.  Luckily, it
does not need to.

If you want to organize your review queue, use a tool like
gerrit-dash-creator:

  https://github.com/stackforge/gerrit-dash-creator

If you want to change how stats are generated, patch reviewstats.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [MagnetoDB] Async create/delete table

2014-09-18 Thread Illia Khudoshyn
Hi stackers,

We're about to introduce asynchronous table creation and deletion in
MagnetoDB via oslo.messaging.
The draft specification is availiable by the link below.
https://wiki.openstack.org/wiki/MagnetoDB/specs/async-schema-operations

Please share your thoughts

-- 

Best regards,

Illia Khudoshyn,
Software Engineer, Mirantis, Inc.



38, Lenina ave. Kharkov, Ukraine

www.mirantis.com http://www.mirantis.ru/

www.mirantis.ru



Skype: gluke_work

ikhudos...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/18/2014 04:24 PM, Clint Byrum wrote:
 Great job highlighting what our friends over at Amazon are doing.
 
 It's clear from these snippets, and a few other pieces of documentation
 for SQS I've read, that the Amazon team approached SQS from a _massive_
 scaling perspective. I think what may be forcing a lot of this frustration
 with Zaqar is that it was designed with a much smaller scale in mind.
 
 I think as long as that is the case, the design will remain in question.
 I'd be comfortable saying that the use cases I've been thinking about
 are entirely fine with the limitations SQS has.

I think these are pretty strong comments with not enough arguments to
defend them.

Saying that Zaqar was designed with a smaller scale in mid without
actually saying why you think so is not fair besides not being true. So
please, do share why you think Zaqar was not designed for big scales and
provide comments that will help the project to grow and improve.

- Is it because the storage technologies that have been chosen?
- Is it because of the API?
- Is it because of the programing language/framework ?

So far, we've just discussed the API semantics and not zaqar's
scalability, which makes your comments even more surprising.

Flavio

 
 Excerpts from Joe Gordon's message of 2014-09-17 13:36:18 -0700:
 Hi All,

 My understanding of Zaqar is that it's like SQS. SQS uses distributed
 queues, which have a few unusual properties [0]:
 Message Order

 Amazon SQS makes a best effort to preserve order in messages, but due to
 the distributed nature of the queue, we cannot guarantee you will receive
 messages in the exact order you sent them. If your system requires that
 order be preserved, we recommend you place sequencing information in each
 message so you can reorder the messages upon receipt.
 At-Least-Once Delivery

 Amazon SQS stores copies of your messages on multiple servers for
 redundancy and high availability. On rare occasions, one of the servers
 storing a copy of a message might be unavailable when you receive or delete
 the message. If that occurs, the copy of the message will not be deleted on
 that unavailable server, and you might get that message copy again when you
 receive messages. Because of this, you must design your application to be
 idempotent (i.e., it must not be adversely affected if it processes the
 same message more than once).
 Message Sample

 The behavior of retrieving messages from the queue depends whether you are
 using short (standard) polling, the default behavior, or long polling. For
 more information about long polling, see Amazon SQS Long Polling
 http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
 .

 With short polling, when you retrieve messages from the queue, Amazon SQS
 samples a subset of the servers (based on a weighted random distribution)
 and returns messages from just those servers. This means that a particular
 receive request might not return all your messages. Or, if you have a small
 number of messages in your queue (less than 1000), it means a particular
 request might not return any of your messages, whereas a subsequent request
 will. If you keep retrieving from your queues, Amazon SQS will sample all
 of the servers, and you will receive all of your messages.

 The following figure shows short polling behavior of messages being
 returned after one of your system components makes a receive request.
 Amazon SQS samples several of the servers (in gray) and returns the
 messages from those servers (Message A, C, D, and B). Message E is not
 returned to this particular request, but it would be returned to a
 subsequent request.



 Presumably SQS has these properties because it makes the system scalable,
 if so does Zaqar have the same properties (not just making these same
 guarantees in the API, but actually having these properties in the
 backends)? And if not, why? I looked on the wiki [1] for information on
 this, but couldn't find anything.





 [0]
 http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html
 [1] https://wiki.openstack.org/wiki/Zaqar
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] zeromq work for kilo

2014-09-18 Thread Flavio Percoco
On 09/18/2014 04:16 PM, Kapil Thangavelu wrote:
 
 
 On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco fla...@redhat.com
 mailto:fla...@redhat.com wrote:
 
 On 09/17/2014 04:34 PM, Doug Hellmann wrote:
  This thread [1] has turned more “future focused, so I’m moving the 
 conversation to the -dev list where we usually have those sorts of 
 discussions.
 
  [1] 
 http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
 
 
 
 I saw this mentioned in the `openstack` thread but I'd like us to
 reconsider it.
 
 Since we've gone through the hey please don't deprecate it, I'll help
 process a couple of times with the zmq driver, I'm thinking we should
 pull it out of the code base anyway.
 
 
 I think the primary issue has been two fold, one is the lack of
 reviews/merges on extant patches that fix critical items that have been
 outstanding for months. I think that is compounded by the other issue
 which was the lack of tests. We've sprinted last week on adding in both
 unit tests, the extant patches and functionally verifying them by
 automating cloud deployments with zmq for the messaging driver. We're
 also committing as a company to supporting it on an ongoing basis. If we
 get the gates going for kilo, i don't see any reason for the churn
 below, if the gates don't get going we can yank to external in kilo anyway.

The above sounds good to me. Thanks for sharing.

Can we have a deadline for all this? Last time we discussed zmq driver's
faith, we said that k-1 was a good deadline for it to get back in shape.
If the above plan sounds good to other folks, then I'd prefer us to
stick to k-1 for all the above to happen.

Thoughts?

Thanks again, Kapil
Flavio

  
 cheers,
 
 Kapil
 
 
 Here's a rough plan of what I think we should do until the zmq is
 updated and has a proper gate working:
 
 1. Pull it out the code base into its own repo (stackforge?)
 2. Setup the basic `oslo.messaging` gates for it
 3. Release the driver on pypi as: `oslo.messaging.zmq`
 4. Make lots of noise and make sure people using zmq knows that they
 have to install a separate package now. (I know this means creating a
 new package for many distros).
 
 If there are folks interested in maintaining this driver, they can still
 do it with the driver outside the code base. If it ever gets enough
 attention and maturity, it could be re-proposed for inclusion into the
 code base.
 
 I know there are folks interested in a broker-less solution and we now
 have the not-tested-in-production-yet amqp1.0 driver which I think is a
 very good solution and also shares most of the semantics we rely on
 protocol-wise.
 
 I didn't go through the whole thread so, I'm sorry if I'm repeating
 something that has already been said/proposed/discussed.
 
 Flavio
 
  On Sep 17, 2014, at 7:54 AM, James Page james.p...@ubuntu.com
 mailto:james.p...@ubuntu.com wrote:
 
  Signed PGP part
  Hi Li
 
  On 17/09/14 11:58, Li Ma wrote:
  The scale potential is very appealing and is something I want to
  test - - hopefully in the next month or so.
 
  Canonical are interested in helping to maintain this driver and
  hopefully we help any critical issues prior to Juno release.
 
 
  That sounds good. I just went through all the bugs reported in the
  community.
 
  The only critical bug which makes ZeroMQ malfunction is
  https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
  corresponding review is under way:
  https://review.openstack.org/#/c/84938/
 
  Agreed
 
  Others are tagged to 'zmq' in
  https://bugs.launchpad.net/oslo.messaging
 
  Looking through Doug's suggested list of information and collating
  what I know of from our work last week:
 
  1) documentation for how to configure and use zeromq with
  oslo.messaging (note, not the version in oslo-incubator, the version
  in the messaging library repository)
 
  As part of our sprint, I worked on automating deployment of OpenStack
  + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
  that work into some general documentation on how best to configure
  ZeroMQ with OpenStack - the current documentation is a bit raw and
  does not talk about how to configure the oslo-messaging-zmq-receiver
  at all.
 
  I also plan some packaging updates for Debian/Ubuntu in our next dev
  cycle to make this a little easier to configure and digest - for
  example, right now no systemd unit/upstart configuration/sysv init
  script is provided to manage the zmq-receiver.
 
  I'd also like to document the current design of the ZMQ driver - Doug
  - where is the best place todo this? I thought in the source tree
  somewhere.
 
  The documentation in the oslo.messaging 

Re: [openstack-dev] [blazar]: proposal for a new lease type

2014-09-18 Thread Lisa

Hi Sylvain,

thanks a lot for your answer.

I'd like to extend (if possible) the BLAZAR's list of the supported 
lease types by adding a new one which covers a specific use case which, 
it seems, it is not yet supported by BLAZAR. You know, In OpenStack 
every project has a granted fixed quota which cannot be extended 
dynamically. This implies that a project cannot allocate unused 
resources assigned to different projects. The idea is to lease such 
unused resources just for a limited duration time (e.g. from few hours 
to 1 day). Suppose the project A has consumed all own assigned 
resources while the project B, in the current time, has several 
available resources. B may offer its own resources to A just for a 
limited and well known time duration (i.e. not forever) so that in the 
future it can claim, if needed, the available resources belonging to A 
(or other projects).
This is a fair-share mechanism which guarantees the resources usage is 
equally distributed among users and projects by considering the portion 
of the resources allocated to them (i.e. share) and the resources 
already consumed. Please remark the share is a different concept than 
quota in the cloud terminology. You can consider this proposal as a 
way to update dynamically the quotas by considering the historical usage 
of each project. This approach should make OpenStack very efficient in 
terms of resource utilization.


About the blueprint I will try to create a new one.

Thanks again.
Cheers,
Lisa

On 18/09/2014 16:00, Sylvain Bauza wrote:


Le 18/09/2014 15:27, Lisa a écrit :

Hi all,

my name is Lisa Zangrando and I work at the Italian National 
Institute for Nuclear Physics (INFN). In particular I am leading a 
team which is addressing the issue concerning the efficiency in the 
resource usage in OpenStack.
Currently OpenStack allows just a static partitioning model where the 
resource allocation to the user teams (i.e. the projects) can be done 
only by considering fixed quotas which cannot be exceeded even if 
there are unused resources (but) assigned to different projects.
We studied the available BLAZAR's documentation and, in agreement 
with Tim Bell (who is responsible the OpenStack cloud project at 
CERN), we think this issue could be addressed within your framework.
Please find attached a document that describes our use cases 
(actually we think that many other environments have to deal with the 
same problems) and how they could be managed in BLAZAR, by defining a 
new lease type (i.e. fairShare lease) to be considered as extension 
of the list of the already supported lease types.

I would then be happy to discuss these ideas with you.

Thanks in advance,
Lisa



Hi Lisa,

Glad to see you're interested in Blazar.

I tried to go thru your proposal, but could you please post the main 
concepts of what you plan to add into an etherpad and create a 
blueprint [1] mapped to it so we could discuss on the implementation ?
Of course, don't hesitate to ping me or the blazar community in 
#openstack-blazar if you need help with the process or the current 
Blazar design.


Thanks,
-Sylvain

[1] https://blueprints.launchpad.net/blazar/




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] zeromq work for kilo

2014-09-18 Thread Davanum Srinivas
Kapil,

I see just 2 relevant reviews for zmq in the oslo.messaging queue:
https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z

Are there others i am missing? (fix critical items, tests from your email)

thanks,
dims

On Thu, Sep 18, 2014 at 10:16 AM, Kapil Thangavelu
kapil.thangav...@canonical.com wrote:


 On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco fla...@redhat.com wrote:

 On 09/17/2014 04:34 PM, Doug Hellmann wrote:
  This thread [1] has turned more “future focused, so I’m moving the
  conversation to the -dev list where we usually have those sorts of
  discussions.
 
  [1]
  http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
 


 I saw this mentioned in the `openstack` thread but I'd like us to
 reconsider it.

 Since we've gone through the hey please don't deprecate it, I'll help
 process a couple of times with the zmq driver, I'm thinking we should
 pull it out of the code base anyway.


 I think the primary issue has been two fold, one is the lack of
 reviews/merges on extant patches that fix critical items that have been
 outstanding for months. I think that is compounded by the other issue which
 was the lack of tests. We've sprinted last week on adding in both unit
 tests, the extant patches and functionally verifying them by automating
 cloud deployments with zmq for the messaging driver. We're also committing
 as a company to supporting it on an ongoing basis. If we get the gates going
 for kilo, i don't see any reason for the churn below, if the gates don't get
 going we can yank to external in kilo anyway.

 cheers,

 Kapil


 Here's a rough plan of what I think we should do until the zmq is
 updated and has a proper gate working:

 1. Pull it out the code base into its own repo (stackforge?)
 2. Setup the basic `oslo.messaging` gates for it
 3. Release the driver on pypi as: `oslo.messaging.zmq`
 4. Make lots of noise and make sure people using zmq knows that they
 have to install a separate package now. (I know this means creating a
 new package for many distros).

 If there are folks interested in maintaining this driver, they can still
 do it with the driver outside the code base. If it ever gets enough
 attention and maturity, it could be re-proposed for inclusion into the
 code base.

 I know there are folks interested in a broker-less solution and we now
 have the not-tested-in-production-yet amqp1.0 driver which I think is a
 very good solution and also shares most of the semantics we rely on
 protocol-wise.

 I didn't go through the whole thread so, I'm sorry if I'm repeating
 something that has already been said/proposed/discussed.

 Flavio

  On Sep 17, 2014, at 7:54 AM, James Page james.p...@ubuntu.com wrote:
 
  Signed PGP part
  Hi Li
 
  On 17/09/14 11:58, Li Ma wrote:
  The scale potential is very appealing and is something I want to
  test - - hopefully in the next month or so.
 
  Canonical are interested in helping to maintain this driver and
  hopefully we help any critical issues prior to Juno release.
 
 
  That sounds good. I just went through all the bugs reported in the
  community.
 
  The only critical bug which makes ZeroMQ malfunction is
  https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
  corresponding review is under way:
  https://review.openstack.org/#/c/84938/
 
  Agreed
 
  Others are tagged to 'zmq' in
  https://bugs.launchpad.net/oslo.messaging
 
  Looking through Doug's suggested list of information and collating
  what I know of from our work last week:
 
  1) documentation for how to configure and use zeromq with
  oslo.messaging (note, not the version in oslo-incubator, the version
  in the messaging library repository)
 
  As part of our sprint, I worked on automating deployment of OpenStack
  + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
  that work into some general documentation on how best to configure
  ZeroMQ with OpenStack - the current documentation is a bit raw and
  does not talk about how to configure the oslo-messaging-zmq-receiver
  at all.
 
  I also plan some packaging updates for Debian/Ubuntu in our next dev
  cycle to make this a little easier to configure and digest - for
  example, right now no systemd unit/upstart configuration/sysv init
  script is provided to manage the zmq-receiver.
 
  I'd also like to document the current design of the ZMQ driver - Doug
  - where is the best place todo this? I thought in the source tree
  somewhere.
 
  The documentation in the oslo.messaging repository [2] would be a good
  place to start for that. If we decide deployers/operators need the
  information we can either refer to it from the guides managed by the
  documentation team or we can move/copy the information. How about if you
  start a new drivers subdirectory there, and add information about zmq. We
  can have other driver authors provide similar detail about their drivers in
  the same directory.
 
  [2]
  

Re: [openstack-dev] Oslo final releases ready

2014-09-18 Thread Davanum Srinivas
w00t!

-- dims

On Thu, Sep 18, 2014 at 10:04 AM, Doug Hellmann d...@doughellmann.com wrote:
 All of the final releases for the Oslo libraries for the Juno cycle are 
 available on PyPI. I’m working on a couple of patches to the global 
 requirements list to update the baseline in the applications. In all cases, 
 the final release is a second tag on a previously released version.

 - oslo.config - 1.4.0 (same as 1.4.0.0a5)
 - oslo.db - 1.0.0 (same as 0.5.0)
 - oslo.i18n - 1.0.0 (same as 0.4.0)
 - oslo.messaging - 1.4.0 (same as 1.4.0.0a5)
 - oslo.rootwrap - 1.3.0 (same as 1.3.0.0a3)
 - oslo.serialization - 1.0.0 (same as 0.3.0)
 - oslosphinx - 2.2.0 (same as 2.2.0.0a3)
 - oslotest - 1.1.0 (same as 1.1.0.0a2)
 - oslo.utils - 1.0.0 (same as 0.3.0)
 - cliff - 1.7.0 (previously tagged, so not a new release)
 - stevedore - 1.0.0 (same as 1.0.0.0a2)

 Congratulations and *Thank You* to the Oslo team for doing an amazing job 
 with graduations this cycle!

 Doug


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Steve Lewis
On September 18, 2014 7:45 AM, Flavio Percoco wrote:

 On 09/18/2014 04:09 PM, Gordon Sim wrote:

 However, as far as consuming messages is concerned, it can
 guarantee once-and-only-once and/or at-least-once delivery depending on
 the message pattern used to consume messages. Using pop or claims
 guarantees the former whereas streaming messages out of Zaqar guarantees
 the later.

 From what I can see, pop provides unreliable delivery (i.e. its similar
 to no-ack). If the delete call using pop fails while sending back the
 response, the messages are removed but didn't get to the client.
 
 Correct, pop works like no-ack. If you want to have pop+ack, it is
 possible to claim just 1 message and then delete it.

Having some experience using SQS I would expect that there would be a mode
something like what SQS provides where a message gets picked up by one 
consumer (let's say by short-polling) and is hidden for a timeout duration from 
other consumers, so that the consumer has time to ack. Is that how you are 
using the term 'claim' in this case?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/18/2014 05:42 PM, Steve Lewis wrote:
 On September 18, 2014 7:45 AM, Flavio Percoco wrote:
 
 On 09/18/2014 04:09 PM, Gordon Sim wrote:

 However, as far as consuming messages is concerned, it can
 guarantee once-and-only-once and/or at-least-once delivery depending on
 the message pattern used to consume messages. Using pop or claims
 guarantees the former whereas streaming messages out of Zaqar guarantees
 the later.

 From what I can see, pop provides unreliable delivery (i.e. its similar
 to no-ack). If the delete call using pop fails while sending back the
 response, the messages are removed but didn't get to the client.

 Correct, pop works like no-ack. If you want to have pop+ack, it is
 possible to claim just 1 message and then delete it.
 
 Having some experience using SQS I would expect that there would be a mode
 something like what SQS provides where a message gets picked up by one 
 consumer (let's say by short-polling) and is hidden for a timeout duration 
 from 
 other consumers, so that the consumer has time to ack. Is that how you are 
 using the term 'claim' in this case?

Correct, that's the name of the feature in Zaqar. You can find a bit of
extra information here[0]. Hope this helps.

[0] https://wiki.openstack.org/wiki/Zaqar/specs/api/v1.1#Claims

Cheers,
Flavio

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Devananda van der Veen
On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote:
 On 09/18/2014 04:09 PM, Gordon Sim wrote:
 On 09/18/2014 12:31 PM, Flavio Percoco wrote:
 Zaqar guarantees FIFO. To be more precise, it does that relying on the
 storage backend ability to do so as well. Depending on the storage used,
 guaranteeing FIFO may have some performance penalties.

 Would it be accurate to say that at present Zaqar does not use
 distributed queues, but holds all queue data in a storage mechanism of
 some form which may internally distribute that data among servers but
 provides Zaqar with a consistent data model of some form?

 I think this is accurate. The queue's distribution depends on the
 storage ability to do so and deployers will be able to choose what
 storage works best for them based on this as well. I'm not sure how
 useful this separation is from a user perspective but I do see the
 relevance when it comes to implementation details and deployments.

Guaranteeing FIFO and not using a distributed queue architecture
*above* the storage backend are both scale-limiting design choices.
That Zaqar's scalability depends on the storage back end is not a
desirable thing in a cloud-scale messaging system in my opinion,
because this will prevent use at scales which can not be accommodated
by a single storage back end.

And based on my experience consulting for companies whose needs grew
beyond the capabilities of a single storage backend, moving to
application-aware sharding required a significant amount of
rearchitecture in most cases.

-Devananda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ceilometer] Performance degradation (docs) for sampling rate less than 10 mins

2014-09-18 Thread Srikanta Patanjali
Hi Team,

I was considering to change the default sampling rate of the Ceilometer
from 10 mins to less than that. I foresee an adverse impact on its
performance due to increase in the data inside the MondoDB.

I was wondering if the QA team (or anyone else) has done any of the load
tests with these parameters ? It would be helpful to have access to these
results (if any).

If not, I would like to know the views on increasing the sample rate value
(in the pipleline.yaml file)

Cheers,
Srikanta
InIT ¦ ICCLab
ZHAW
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Devananda van der Veen
On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco fla...@redhat.com wrote:
 On 09/18/2014 04:24 PM, Clint Byrum wrote:
 Great job highlighting what our friends over at Amazon are doing.

 It's clear from these snippets, and a few other pieces of documentation
 for SQS I've read, that the Amazon team approached SQS from a _massive_
 scaling perspective. I think what may be forcing a lot of this frustration
 with Zaqar is that it was designed with a much smaller scale in mind.

 I think as long as that is the case, the design will remain in question.
 I'd be comfortable saying that the use cases I've been thinking about
 are entirely fine with the limitations SQS has.

 I think these are pretty strong comments with not enough arguments to
 defend them.


Please see my prior email. I agree with Clint's assertions here.

 Saying that Zaqar was designed with a smaller scale in mid without
 actually saying why you think so is not fair besides not being true. So
 please, do share why you think Zaqar was not designed for big scales and
 provide comments that will help the project to grow and improve.

 - Is it because the storage technologies that have been chosen?
 - Is it because of the API?
 - Is it because of the programing language/framework ?

It is not because of the storage technology or because of the
programming language.

 So far, we've just discussed the API semantics and not zaqar's
 scalability, which makes your comments even more surprising.

- guaranteed message order
- not distributing work across a configurable number of back ends

These are scale-limiting design choices which are reflected in the
API's characteristics.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Alex Xu

On 2014年09月18日 18:57, Sean Dague wrote:

On 09/18/2014 06:38 AM, Christopher Yeoh wrote:

On Sat, 13 Sep 2014 06:48:19 -0400
Sean Dague s...@dague.net wrote:


On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:

Hi Chris,

Thanks for bring it up here,


-Original Message-
From: Chris St. Pierre [mailto:stpie...@metacloud.com]
Sent: Saturday, September 13, 2014 2:53 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [nova] Expand resource name allowed
characters

We have proposed that the allowed characters for all resource
names in Nova (flavors, aggregates, etc.) be expanded to all
printable unicode characters and horizontal spaces:
https://review.openstack.org/#/c/119741

Currently, the only allowed characters in most resource names are
alphanumeric, space, and [.-_].


We have proposed this change for two principal reasons:

1. We have customers who have migrated data forward since Essex,
when no restrictions were in place, and thus have characters in
resource names that are disallowed in the current version of
OpenStack. This is only likely to be useful to people migrating
from Essex or earlier, since the current restrictions were added
in Folsom.

2. It's pretty much always a bad idea to add unnecessary
restrictions without a good reason. While we don't have an
immediate need to use, for example, the ever-useful
http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
up with a reason people *shouldn't* be allowed to use it.

That said, apparently people have had a need to not be allowed to
use some characters, but it's not clear why:
https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
knows any reason why these printable characters should not be
joined in holy resource naming, speak now or forever hold your
peace.

I also could not find the reason of current restriction on the bug
report, and I'd like to know it as the history.
On v2 API(not v2.1), each resource name contains the following
restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain . and !

On v2.1 API, we have applied the same restriction rule[1] for whole
resource names for API consistency, so maybe we need to consider
this topic for whole names.

[1]:
https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

Honestly, I bet this had to do with how the database used to be set
up.


So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
multibyte characters (only 1-3 bytes). For example if you do a create
image call with an image name to glance with a 4 byte multibyte
character in the name it will 500. I'd guess we do something
similar in places with the Nova API where we have inadequate input
validation. If you want 4 byte support you need to use utf8mb4 instead.

Oh... fun. :(


I don't know if postgresql has any restrictions (I don't think it
does) or if db2 does too. But I don't think we can/should make it a
complete free for all. It should at most be what most databases support.

I think its a big enough change that this late in the cycle we should
push it off to Kilo. It's always much easier to loosen input validation
than tighten it (or have to have an oops revert on an officially
released Nova). Perhaps some tests to verify that everything we allow
past the input validation checks we can actually store.

So, honestly, that seems like a pendulum swing in an odd way.

Havana use anything you want!
Icehouse ?
Juno strict asci!
Kilo utf8

Can't we just catch the db exception correctly in glance and not have it
explode? And then allow it. Exploding with a 500 on a bad name seems the
wrong thing to do anyway.

That would also mean that if the user changed their database to support
utf8mb4 (which they might want to do if it was important to them) it
would all work.

I think some release notes would be fine to explain the current
situation and limitations.

Basically, lets skate towards the puck here, realizing some corner cases
exist, but that we're moving in the direction we want to be, not back
tracking.

When we can return the json-schema to user in the future, can we say 
that means API accepting utf8 or utf8mb4 is discoverable? If it is 
discoverable, then we needn't limit anything in our python code.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Rally] Load testing of dependent resources

2014-09-18 Thread Boris Pavlovic
Ilya,

It sounds good, moreover few other devs asked about similar feature=)
So I am working on proposal how to implement this=)


Best regards,



On Thu, Sep 18, 2014 at 4:35 PM, Ilya Shakhat ishak...@mirantis.com wrote:

 Ralliers,

 I'd like to share with you an idea on how to test dependent resources.

 Let's consider the following example: there is a network and ports
 belonging to it and we would like to test port creation process. Currently
 I see 2 options of writing scenario:
  a) The scenario creates network and then creates a specified number of
 ports. Runner executes the scenario specified number of times. -- In this
 case ports belonging to different networks are created in parallel, but all
 ports belonging to the same network are created serially.
  b) The network already exists and the scenario just creates ports, --
 This case allows to test concurrent creation of ports belonging to the same
 network but not between multiple networks.
 It would be really cool to get benefits from both cases, i.e. to test
 parallel port creation but without limiting to parent network resource.

 One of possible approaches may be to construct chain of scenarios where
 preceding scenario prepares context for the further one. From the above
 example the execution would be:
  1) The task contains sequence of 2 scenarios:
 * network creation - it creates network and returns it
 * port creation - it picks the network from incoming params and
 creates port on it
  2) The runner has execution pipeline (which currently is represented as
 generator producing identical scenario runners). The pipeline starts with
 sequence of network creation scenario runners. Once one of runner finishes,
 it then puts the further scenario and the result into pipeline. The
 pipeline ends once no further scenarios left.

 From the above example execution flow would be like:
  * pipeline is filled with N network scenarios (not actually filled, but
 contains a generator that is able to do this)
  * R runners pick scenarios from pipeline
  * pipeline contains N-R network scenarios
  * one of runners finishes the scenario and updates pipeline with P port
 scenarios
  * pipeline contains N-R-1 network scenarios followed by P port scenarios
  * work-work-work until pipeline is empty

 This execution covers case from a) and b) but there still will be
 sequences of child resources belonging to the same parent. The improvement
 may be to pick scenario from pipeline randomly.

 How does it sound?

 Thanks,
 Ilya

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Expand resource name allowed characters

2014-09-18 Thread Sean Dague
On 09/18/2014 12:08 PM, Alex Xu wrote:
 On 2014年09月18日 18:57, Sean Dague wrote:
 On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
 On Sat, 13 Sep 2014 06:48:19 -0400
 Sean Dague s...@dague.net wrote:

 On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:
 Hi Chris,

 Thanks for bring it up here,

 -Original Message-
 From: Chris St. Pierre [mailto:stpie...@metacloud.com]
 Sent: Saturday, September 13, 2014 2:53 AM
 To: openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [nova] Expand resource name allowed
 characters

 We have proposed that the allowed characters for all resource
 names in Nova (flavors, aggregates, etc.) be expanded to all
 printable unicode characters and horizontal spaces:
 https://review.openstack.org/#/c/119741

 Currently, the only allowed characters in most resource names are
 alphanumeric, space, and [.-_].


 We have proposed this change for two principal reasons:

 1. We have customers who have migrated data forward since Essex,
 when no restrictions were in place, and thus have characters in
 resource names that are disallowed in the current version of
 OpenStack. This is only likely to be useful to people migrating
 from Essex or earlier, since the current restrictions were added
 in Folsom.

 2. It's pretty much always a bad idea to add unnecessary
 restrictions without a good reason. While we don't have an
 immediate need to use, for example, the ever-useful
 http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
 up with a reason people *shouldn't* be allowed to use it.

 That said, apparently people have had a need to not be allowed to
 use some characters, but it's not clear why:
 https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
 knows any reason why these printable characters should not be
 joined in holy resource naming, speak now or forever hold your
 peace.
 I also could not find the reason of current restriction on the bug
 report, and I'd like to know it as the history.
 On v2 API(not v2.1), each resource name contains the following
 restriction for its name:

Resource  | Length  | Pattern
   ---+-+--
aggregate | 1-255   | nothing
backup| nothing | nothing
flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
  | |   [a-zA-Z0-9. _-]*$'
keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
server| 1-255   | nothing
cell  | 1-255   | don't contain . and !

 On v2.1 API, we have applied the same restriction rule[1] for whole
 resource names for API consistency, so maybe we need to consider
 this topic for whole names.

 [1]:
 https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

 Honestly, I bet this had to do with how the database used to be set
 up.

 So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
 multibyte characters (only 1-3 bytes). For example if you do a create
 image call with an image name to glance with a 4 byte multibyte
 character in the name it will 500. I'd guess we do something
 similar in places with the Nova API where we have inadequate input
 validation. If you want 4 byte support you need to use utf8mb4 instead.
 Oh... fun. :(

 I don't know if postgresql has any restrictions (I don't think it
 does) or if db2 does too. But I don't think we can/should make it a
 complete free for all. It should at most be what most databases support.

 I think its a big enough change that this late in the cycle we should
 push it off to Kilo. It's always much easier to loosen input validation
 than tighten it (or have to have an oops revert on an officially
 released Nova). Perhaps some tests to verify that everything we allow
 past the input validation checks we can actually store.
 So, honestly, that seems like a pendulum swing in an odd way.

 Havana use anything you want!
 Icehouse ?
 Juno strict asci!
 Kilo utf8

 Can't we just catch the db exception correctly in glance and not have it
 explode? And then allow it. Exploding with a 500 on a bad name seems the
 wrong thing to do anyway.

 That would also mean that if the user changed their database to support
 utf8mb4 (which they might want to do if it was important to them) it
 would all work.

 I think some release notes would be fine to explain the current
 situation and limitations.

 Basically, lets skate towards the puck here, realizing some corner cases
 exist, but that we're moving in the direction we want to be, not back
 tracking.

 When we can return the json-schema to user in the future, can we say
 that means API accepting utf8 or utf8mb4 is discoverable? If it is
 discoverable, then we needn't limit anything in our python code.

Honestly, we should accept utf8 (no weird mysqlism not quite utf8). We
should make the default scheme for our dbs support that on names (but
only for the name columns). The failure of a backend to do utf8 for real
should return an error to the user. Let's not make this more 

Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

2014-09-18 Thread Adam Young

On 09/17/2014 11:56 AM, Matthieu Huin wrote:

Hi,

- Original Message -

From: Adam Young ayo...@redhat.com
To: openstack-dev@lists.openstack.org
Sent: Wednesday, September 17, 2014 5:00:16 PM
Subject: Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

On 09/17/2014 10:35 AM, David Chadwick wrote:



this would work as well, but wouldn't it require two different API calls?

I think it would be 2 calls no matter what.

OK, lets talk this through:

1. Configure Horizon to return a generic login page, with a button that says
Or do Federated
2. Use clicks that button and gets the Federated UI. No new HTTP request
needed for this, still just static Javascript. Form has an edit field for
the user to enter the SAML IdP, anda button to fetch the list of the public
IdPs from Keystone.
3. Assume user clicks the list of public IdPs, those fill out a drop-down
box. If the user clicks on one, it populates the textbox with the URL for
the IdP.
3a. However, if the users IdP is not in the list, they go back to the email
they got from their IT guy that has Paste this URL into the IdP edit box

4. User clicks OK.

Window pops up with the WebUI for the user to visit the SAML IdP URL. This
will be of the form

GET
htps://keystone/main/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth

Which will perform the necessary redirects to get the user the SAML
assertion, and return it to Keystone.

5. Javascript picks up the Federated unscoped token from the response at the
end of step 4 and use that to get a scoped token.


I think the jump to step 6 isn't unnecessary: logging in Horizon requires only 
a username and a password,
unless I am wrong scoping is done afterwards by selecting a project in a list. 
Besides, should we expect
a federated user to necessarily know beforehand to which domain/project she was 
mapped ?


Think of it this way;  today, Horizon uses the userid and password to 
get a token.  In the case where that password is in LDAP, Horizon should 
never, ever see it.  Even  Keystone should never see it.  So the Token 
abstracts that away, and says  instead I've checked their password.  
its good.





6. Javascript submites the scoped token to Horizon and user is logged in.

Horizon will also need to keep track of the fact that a federated auth was used:
Once Horizon has a token, it can do all this.  Federation is kindof 
irrelevant.


* to list projects and domains available to the user
* to scope the unscoped saml token

As these are done through the federation API rather than the usual one.


Whew.

Whew indeed !






On 17/09/2014 15:17, Adam Young wrote:



On 09/17/2014 10:07 AM, David Chadwick wrote:



On 17/09/2014 14:55, Marek Denis wrote:



On 17.09.2014 15:45, Steve Martinelli wrote:



++ to your suggestion David, I think making the list of trusted IdPs
publicly available makes sense.
I think this might be useful in an academic/science world but on the
other hand most cloud providers from the 'business' world might be very
reluctant to expose list of their clients for free.
It is interesting that this latter comment came from the
academic/science world, whereas the supportive one came from the
business world :-)

So maybe there could be a config parameter in keystone to determine
which option is installed?
My thought was that there would be a public list, which is a subset of
the overall list.

For non-publicized IdPs, the end users would get an URL out of  band and
enter that in when prompted;  if they enter an invalid URL, they would
get an warning message.

It wouldn't hide the fact that a customer was registered with a given
cloud provider, but wouldn't advertise it, either.



regards

David



cheers,

Marek.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Oslo final releases ready

2014-09-18 Thread Thomas Goirand
On 09/18/2014 10:04 PM, Doug Hellmann wrote:
 All of the final releases for the Oslo libraries for the Juno cycle are 
 available on PyPI. I’m working on a couple of patches to the global 
 requirements list to update the baseline in the applications. In all cases, 
 the final release is a second tag on a previously released version.
 
 - oslo.config - 1.4.0 (same as 1.4.0.0a5)
 - oslo.db - 1.0.0 (same as 0.5.0)
 - oslo.i18n - 1.0.0 (same as 0.4.0)
 - oslo.messaging - 1.4.0 (same as 1.4.0.0a5)
 - oslo.rootwrap - 1.3.0 (same as 1.3.0.0a3)
 - oslo.serialization - 1.0.0 (same as 0.3.0)
 - oslosphinx - 2.2.0 (same as 2.2.0.0a3)
 - oslotest - 1.1.0 (same as 1.1.0.0a2)
 - oslo.utils - 1.0.0 (same as 0.3.0)
 - cliff - 1.7.0 (previously tagged, so not a new release)
 - stevedore - 1.0.0 (same as 1.0.0.0a2)
 
 Congratulations and *Thank You* to the Oslo team for doing an amazing job 
 with graduations this cycle!
 
 Doug

Great! Thanks for doing this early in the freeze.

FYI, I have already uploaded all Juno dependencies to Debian, either in
Sid, or in Experimental.

However, I made a small mistake. I used 1.4.0.0~a5 instead of 1.4.0~a5.
As a consequence, I may upload 1.4.0.0 instead of 1.4.0, so that it is
greater than 1.4.0.0~a5 (to avoid adding an EPOC, which is ugly and
annoying to maintain). It's going to be the same version though, I just
need to add a new tag, which isn't much of a problem for me (since I use
a git based workflow).

The 1.4.0.0a5 is confusing... :(

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

2014-09-18 Thread David Chadwick
Adam
I agree with you
David

On 18/09/2014 17:17, Adam Young wrote:
 On 09/17/2014 11:53 AM, Marek Denis wrote:
 Hi,

 First of all, we should clarify whether your JS client wants to
 implement ECP or WebSSO workflow. They are slightly different.
 
 ECP seems to be poorly supported in live deployments, and we cannot
 count on it. WebSSO is the first round.  We should do  ECP as a second
 step.
 

 I feel JS is smart enough to implement the ECP flow and then and it
 could simply implement what we already have in the keystoneclient [0].
 This + some discovery service described by Adam would be required.
 However, some problems may arise when it comes to ADFS  support (we
 needed separate plugin in keystoneclient) and other protocols which
 should work by default from browsers.

 [0]
 https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/auth/v3/saml2.py#L29


 Rest of the comments inlined.

 On 17.09.2014 17:00, Adam Young wrote:
 On 09/17/2014 10:35 AM, David Chadwick wrote:
 this would work as well, but wouldn't it require two different API
 calls?

 I think it would be 2 calls no matter what.

 OK,  lets talk this through:

 1. Configure Horizon to return a generic login page, with a button that
 says Or do Federated
 2.  Use clicks that button and gets the Federated UI.  No new HTTP
 request needed for this, still just static Javascript.  Form has an edit
 field for the user to enter the SAML IdP, anda button to fetch the list
 of the public IdPs from Keystone.
 3.  Assume user clicks the list of public IdPs, those fill out a
 drop-down box.  If the user clicks on one, it populates the textbox with
 the URL for the IdP.
 3a.  However, if the users IdP is not in the list, they go back to the
 email they got from their IT guy that has Paste this URL into the IdP
 edit box

 Well, we don't keep any IdPs' URL in Keystone backend at the moment.
 However, this can be easily fixed.
 Right.  That is a feature request.
 
 

 4. User clicks OK.

 OK

 Window pops up with the WebUI for the user to visit the SAML IdP URL.
 This will be of the form
 |
 GET
 htps://keystone/main/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth|


 Which will perform the necessary redirects to get the user the SAML
 assertion, and return it to Keystone.

 Let's assume this step would work for now. I am interested how would
 the SP-IDP-SP workflow look like from the JS perspective. In classic
 WebSSO, where user uses only his browser:

 1) Go to the protected resource
 (/v3/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth
 in our case)

 2) (skipping the DS step for now) get redirected to the IdP
 3) Authenticate with the IdP
 4) Get redirected to the SP
 5) Read your protected content, because you have been authenticated
 and authorized to do so (get an unscoped, federated token issued by
 Keystone in our case)

 Now, I can imagine, if that's the websso approach can we somehow make
 JS mimic the browser with all its blessing? So the user would actualy
 see the real HTML website? If so, that would be enough to make it work.
 I think that we need to show the user the real website in a Popup window
 until we have a guarantee of ECP support.
 

 5.  Javascript picks up the Federated unscoped token from the response
 at the end of step 4 and use that to get a scoped token.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Clint Byrum
Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
 
  On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote:
  
  Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
  
  On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
  
  
  Linux distributions are not the end be all of distribution models and
  they don’t get to dictate to upstream.
  
  Well, distributions is where the final user is, and where software gets
  consumed. Our priority should be the end users.
  
  
  Distributions are not the only place that people get their software from,
  unless you think that the ~3 million downloads requests has received
  on PyPI in the last 30 days are distributions downloading requests to
  package in their OSs.
  
  
  Do pypi users not also need to be able to detect and fix any versions
  of libraries they might have? If one has some virtualenvs with various
  libraries and apps installed and no --system-site-packages, one would
  probably still want to run 'pip freeze' in all of them and find out what
  libraries are there and need to be fixed.
  
  Anyway, generally security updates require a comprehensive strategy.
  One common comprehensive strategy is version assertion.
  
  Vendoring complicates that immensely.
 
 It doesn’t really matter. PyPI doesn’t dictate to projects who host there what
 that project is allowed to do except in some very broad circumstances. Whether
 or not requests *should* do this doesn't really have any bearing on what
 Openstack should do to cope with it. The facts are that requests does it, and
 that people pulling things from PyPI is an actual platform that needs thought
 about.
 
 This leaves Openstack with a few reasonable/sane options:
 
 1) Decide that vendoring in requests is unacceptable to what Openstack as a
project is willing to support, and cease the use of requests.
 2) Decide that what requests offers is good enough that it outweighs the fact
that it vendors urllib3 and continue using it.
 

There's also 3) fork requests, which is the democratic way to vote out
an upstream that isn't supporting the needs of the masses.

I don't think we're anywhere near there, but I wanted to make it clear
there _is_ a more extreme option.

 If the 2nd option is chosen, then doing anything but supporting the fact that
 requests vendors urllib3 within the code that openstack writes is hurting the
 users who fetch these projects from PyPI because you don't agree with one of
 the choices that requests makes. By all means do conditional imports to lessen
 the impact that the choice requests has made (and the one that Openstack has
 made to use requests) on downstream distributors, but unconditionally 
 importing
 from the top level urllib3 for use within requests is flat out wrong.
 
 Obviously neither of these options excludes the choice to lean on requests to
 reverse this decision as well. However that is best done elsewhere as the
 person making that decision isn't a member of these mailing lists as far as
 I am aware.
 

To be clear, I think we should keep using requests. But we should lend
our influence upstream and explain that our users are required to deal
with this in a way that perhaps hasn't been considered or given the
appropriate priority.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Keystone][Oslo] Federation and Policy

2014-09-18 Thread David Chadwick
Our recent work on federation suggests we need an improvement to the way
the policy engine works. My understanding is that most functions are
protected by the policy engine, but some are not. The latter functions
are publicly accessible. But there is no way in the policy engine to
specify public access to a function and there ought to be. This will
allow an administrator to configure the policy for a function to range
from very lax (publicly accessible) to very strict (admin only). A
policy of  means that any authenticated user can access the function.
But there is no way in the policy to specify that an unauthenticated
user (i.e. public) has access to a function.

We have already identified one function (get trusted IdPs
identity:list_identity_providers) that needs to be publicly accessible
in order for users to choose which IdP to use for federated login.
However some organisations may not wish to make this API call publicly
accessible, whilst others may wish to restrict it to Horizon only etc.
This indicates that that the policy needs to be set by the
administrator, and not by changes to the code (i.e. to either call the
policy engine or not, or to have two different API calls).

If we can invent some policy syntax that indicates public access, e.g.
reserved keyword of public, then Keystone can always call the policy
file for every function and there would be no need to differentiate
between protected APIs and non-protected APIs as all would be protected
to a greater or lesser extent according to the administrator's policy.

Comments please

regards

David






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Donald Stufft

 On Sep 18, 2014, at 12:29 PM, Clint Byrum cl...@fewbar.com wrote:
 
 Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
 
 On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote:
 
 Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
 
 On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
 
 
 Linux distributions are not the end be all of distribution models and
 they don’t get to dictate to upstream.
 
 Well, distributions is where the final user is, and where software gets
 consumed. Our priority should be the end users.
 
 
 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 
 
 Do pypi users not also need to be able to detect and fix any versions
 of libraries they might have? If one has some virtualenvs with various
 libraries and apps installed and no --system-site-packages, one would
 probably still want to run 'pip freeze' in all of them and find out what
 libraries are there and need to be fixed.
 
 Anyway, generally security updates require a comprehensive strategy.
 One common comprehensive strategy is version assertion.
 
 Vendoring complicates that immensely.
 
 It doesn’t really matter. PyPI doesn’t dictate to projects who host there 
 what
 that project is allowed to do except in some very broad circumstances. 
 Whether
 or not requests *should* do this doesn't really have any bearing on what
 Openstack should do to cope with it. The facts are that requests does it, and
 that people pulling things from PyPI is an actual platform that needs thought
 about.
 
 This leaves Openstack with a few reasonable/sane options:
 
 1) Decide that vendoring in requests is unacceptable to what Openstack as a
   project is willing to support, and cease the use of requests.
 2) Decide that what requests offers is good enough that it outweighs the fact
   that it vendors urllib3 and continue using it.
 
 
 There's also 3) fork requests, which is the democratic way to vote out
 an upstream that isn't supporting the needs of the masses.
 
 I don't think we're anywhere near there, but I wanted to make it clear
 there _is_ a more extreme option.

Technically that’s just a specific case of option 1) ;)

But yes that’s a thing Openstack can do.

 
 If the 2nd option is chosen, then doing anything but supporting the fact that
 requests vendors urllib3 within the code that openstack writes is hurting the
 users who fetch these projects from PyPI because you don't agree with one of
 the choices that requests makes. By all means do conditional imports to 
 lessen
 the impact that the choice requests has made (and the one that Openstack has
 made to use requests) on downstream distributors, but unconditionally 
 importing
 from the top level urllib3 for use within requests is flat out wrong.
 
 Obviously neither of these options excludes the choice to lean on requests to
 reverse this decision as well. However that is best done elsewhere as the
 person making that decision isn't a member of these mailing lists as far as
 I am aware.
 
 
 To be clear, I think we should keep using requests. But we should lend
 our influence upstream and explain that our users are required to deal
 with this in a way that perhaps hasn't been considered or given the
 appropriate priority.

I think that’s completely reasonable. I don’t think there’s going to be much 
movement,
I’ve had this argument with Kenneth on more than one occasion and he’s very 
happy
with his decision to vendor urllib3, but hey maybe Openstack would have better 
luck.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Clint Byrum
Excerpts from Ian Cordasco's message of 2014-09-18 07:35:10 -0700:
 On 9/18/14, 9:18 AM, Clint Byrum cl...@fewbar.com wrote:
 
 Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
  
   On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
   
   
   Linux distributions are not the end be all of distribution models and
   they don’t get to dictate to upstream.
   
   Well, distributions is where the final user is, and where software
 gets
   consumed. Our priority should be the end users.
  
  
  Distributions are not the only place that people get their software
 from,
  unless you think that the ~3 million downloads requests has received
  on PyPI in the last 30 days are distributions downloading requests to
  package in their OSs.
  
 
 Do pypi users not also need to be able to detect and fix any versions
 of libraries they might have? If one has some virtualenvs with various
 libraries and apps installed and no --system-site-packages, one would
 probably still want to run 'pip freeze' in all of them and find out what
 libraries are there and need to be fixed.
 
 Anyway, generally security updates require a comprehensive strategy.
 One common comprehensive strategy is version assertion.
 
 Vendoring complicates that immensely.
 
 Except that even OpenStack doesn’t pin requests because of how
 extraordinarily stable our API is. While you can argue that Kenneth has
 non-standard opinions about his library, Cory and I take backwards
 compatibility and stability very seriously. This means anyone can upgrade
 to a newer version of requests without worrying that it will be backwards
 incompatible. 
 

All of your hard work is very much appreciated. I don't understand what
your assertion means though. We don't pin things. However, our users end
up pinning when they install via pip, and our distros end up pinning
when they deliver a version. Without any indication that urllib3 is in
the system, they will fail at any cursory version audit that looks for it.

I'm not saying either way is right or wrong either.. I'm suggesting
that this is a valid, proven method for large scale risk assessment,
and it is complicated quite a bit by vendored libraries.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Joe Gordon
On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen 
devananda@gmail.com wrote:

 On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco fla...@redhat.com wrote:
  On 09/18/2014 04:24 PM, Clint Byrum wrote:
  Great job highlighting what our friends over at Amazon are doing.
 
  It's clear from these snippets, and a few other pieces of documentation
  for SQS I've read, that the Amazon team approached SQS from a _massive_
  scaling perspective. I think what may be forcing a lot of this
 frustration
  with Zaqar is that it was designed with a much smaller scale in mind.
 
  I think as long as that is the case, the design will remain in question.
  I'd be comfortable saying that the use cases I've been thinking about
  are entirely fine with the limitations SQS has.
 
  I think these are pretty strong comments with not enough arguments to
  defend them.
 

 Please see my prior email. I agree with Clint's assertions here.

  Saying that Zaqar was designed with a smaller scale in mid without
  actually saying why you think so is not fair besides not being true. So
  please, do share why you think Zaqar was not designed for big scales and
  provide comments that will help the project to grow and improve.
 
  - Is it because the storage technologies that have been chosen?
  - Is it because of the API?
  - Is it because of the programing language/framework ?

 It is not because of the storage technology or because of the
 programming language.

  So far, we've just discussed the API semantics and not zaqar's
  scalability, which makes your comments even more surprising.

 - guaranteed message order
 - not distributing work across a configurable number of back ends

 These are scale-limiting design choices which are reflected in the
 API's characteristics.


I agree with Clint and Devananda



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Devananda van der Veen
On Thu, Sep 18, 2014 at 8:54 AM, Devananda van der Veen
devananda@gmail.com wrote:
 On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote:
 On 09/18/2014 04:09 PM, Gordon Sim wrote:
 On 09/18/2014 12:31 PM, Flavio Percoco wrote:
 Zaqar guarantees FIFO. To be more precise, it does that relying on the
 storage backend ability to do so as well. Depending on the storage used,
 guaranteeing FIFO may have some performance penalties.

 Would it be accurate to say that at present Zaqar does not use
 distributed queues, but holds all queue data in a storage mechanism of
 some form which may internally distribute that data among servers but
 provides Zaqar with a consistent data model of some form?

 I think this is accurate. The queue's distribution depends on the
 storage ability to do so and deployers will be able to choose what
 storage works best for them based on this as well. I'm not sure how
 useful this separation is from a user perspective but I do see the
 relevance when it comes to implementation details and deployments.

 Guaranteeing FIFO and not using a distributed queue architecture
 *above* the storage backend are both scale-limiting design choices.
 That Zaqar's scalability depends on the storage back end is not a
 desirable thing in a cloud-scale messaging system in my opinion,
 because this will prevent use at scales which can not be accommodated
 by a single storage back end.


It may be worth qualifying this a bit more.

While no single instance of any storage back-end is infinitely
scalable, some of them are really darn fast. That may be enough for
the majority of use cases. It's not outside the realm of possibility
that the inflection point [0] where these design choices result in
performance limitations is at the very high end of scale-out, eg.
public cloud providers who have the resources to invest further in
improving zaqar.

As an example of what I mean, let me refer to the 99th percentile
response time graphs in Kurt's benchmarks [1]... increasing the number
of clients with write-heavy workloads was enough to drive latency from
10ms to 200 ms with a single service. That latency significantly
improved as storage and application instances were added, which is
good, and what I would expect. These benchmarks do not (and were not
intended to) show the maximal performance of a public-cloud-scale
deployment -- but they do show that performance under different
workloads improves as additional services are started.

While I have no basis for comparing the configuration of the
deployment he used in those tests to what a public cloud operator
might choose to deploy, and presumably such an operator would put
significant work into tuning storage and running more instances of
each service and thus shift that inflection point to the right, my
point is that, by depending on a single storage instance, Zaqar has
pushed the *ability* to scale out down into the storage
implementation. Given my experience scaling SQL and NoSQL data stores
(in my past life, before working on OpenStack) I have a knee-jerk
reaction to believing that this approach will result in a
public-cloud-scale messaging system.

-Devananda

[0] http://en.wikipedia.org/wiki/Inflection_point -- in this context,
I mean the point on the graph of throughput vs latency where the
derivative goes from near-zero (linear growth) to non-zero
(exponential growth)

[1] https://wiki.openstack.org/wiki/Zaqar/Performance/PubSub/Redis

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Joe Gordon
On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote:

 On 09/18/2014 04:09 PM, Gordon Sim wrote:
  On 09/18/2014 12:31 PM, Flavio Percoco wrote:
  On 09/17/2014 10:36 PM, Joe Gordon wrote:
  My understanding of Zaqar is that it's like SQS. SQS uses distributed
  queues, which have a few unusual properties [0]:
 
 
   Message Order
 
  Amazon SQS makes a best effort to preserve order in messages, but due
 to
  the distributed nature of the queue, we cannot guarantee you will
  receive messages in the exact order you sent them. If your system
  requires that order be preserved, we recommend you place sequencing
  information in each message so you can reorder the messages upon
  receipt.
 
 
  Zaqar guarantees FIFO. To be more precise, it does that relying on the
  storage backend ability to do so as well. Depending on the storage used,
  guaranteeing FIFO may have some performance penalties.
 
  Would it be accurate to say that at present Zaqar does not use
  distributed queues, but holds all queue data in a storage mechanism of
  some form which may internally distribute that data among servers but
  provides Zaqar with a consistent data model of some form?

 I think this is accurate. The queue's distribution depends on the
 storage ability to do so and deployers will be able to choose what
 storage works best for them based on this as well. I'm not sure how
 useful this separation is from a user perspective but I do see the
 relevance when it comes to implementation details and deployments.


  [...]
  As of now, Zaqar fully relies on the storage replication/clustering
  capabilities to provide data consistency, availability and fault
  tolerance.
 
  Is the replication synchronous or asynchronous with respect to client
  calls? E.g. will the response to a post of messages be returned only
  once the replication of those messages is confirmed? Likewise when
  deleting a message, is the response only returned when replicas of the
  message are deleted?

 It depends on the driver implementation and/or storage configuration.
 For example, in the mongodb driver, we use the default write concern
 called acknowledged. This means that as soon as the message gets to
 the master node (note it's not written on disk yet nor replicated) zaqar
 will receive a confirmation and then send the response back to the
 client. This is also configurable by the deployer by changing the
 default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0].


This means that by default Zaqar cannot guarantee a message will be
delivered at all. A message can be acknowledged and then the 'master node'
crashes and the message is lost. Zaqar's ability to guarantee delivery is
limited by the reliability of a single node, while something like swift can
only loose a piece of data if 3 machines crash at the same time.



 [0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w

 
  However, as far as consuming messages is concerned, it can
  guarantee once-and-only-once and/or at-least-once delivery depending on
  the message pattern used to consume messages. Using pop or claims
  guarantees the former whereas streaming messages out of Zaqar guarantees
  the later.
 
  From what I can see, pop provides unreliable delivery (i.e. its similar
  to no-ack). If the delete call using pop fails while sending back the
  response, the messages are removed but didn't get to the client.

 Correct, pop works like no-ack. If you want to have pop+ack, it is
 possible to claim just 1 message and then delete it.

 
  What do you mean by 'streaming messages'?

 I'm sorry, that went out wrong. I had the browsability term in my head
 and went with something even worse. By streaming messages I meant
 polling messages without claiming them. In other words, at-least-once is
 guaranteed by default, whereas once-and-only-once is guaranteed just if
 claims are used.

 
  [...]
  Based on our short conversation on IRC last night, I understand you're
  concerned that FIFO may result in performance issues. That's a valid
  concern and I think the right answer is that it depends on the storage.
  If the storage has a built-in FIFO guarantee then there's nothing Zaqar
  needs to do there. In the other hand, if the storage does not have a
  built-in support for FIFO, Zaqar will cover it in the driver
  implementation. In the mongodb driver, each message has a marker that is
  used to guarantee FIFO.
 
  That marker is a sequence number of some kind that is used to provide
  ordering to queries? Is it generated by the database itself?

 It's a sequence number to provide ordering to queries, correct.
 Depending on the driver, it may be generated by Zaqar or the database.
 In mongodb's case it's generated by Zaqar[0].

 [0]

 https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185

 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-18 Thread Ian Cordasco


On 9/18/14, 11:29 AM, Clint Byrum cl...@fewbar.com wrote:

Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
 
  On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote:
  
  Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
  
  On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote:
  
  
  Linux distributions are not the end be all of distribution models
and
  they don’t get to dictate to upstream.
  
  Well, distributions is where the final user is, and where software
gets
  consumed. Our priority should be the end users.
  
  
  Distributions are not the only place that people get their software
from,
  unless you think that the ~3 million downloads requests has received
  on PyPI in the last 30 days are distributions downloading requests to
  package in their OSs.
  
  
  Do pypi users not also need to be able to detect and fix any versions
  of libraries they might have? If one has some virtualenvs with various
  libraries and apps installed and no --system-site-packages, one would
  probably still want to run 'pip freeze' in all of them and find out
what
  libraries are there and need to be fixed.
  
  Anyway, generally security updates require a comprehensive strategy.
  One common comprehensive strategy is version assertion.
  
  Vendoring complicates that immensely.
 
 It doesn’t really matter. PyPI doesn’t dictate to projects who host
there what
 that project is allowed to do except in some very broad circumstances.
Whether
 or not requests *should* do this doesn't really have any bearing on what
 Openstack should do to cope with it. The facts are that requests does
it, and
 that people pulling things from PyPI is an actual platform that needs
thought
 about.
 
 This leaves Openstack with a few reasonable/sane options:
 
 1) Decide that vendoring in requests is unacceptable to what Openstack
as a
project is willing to support, and cease the use of requests.
 2) Decide that what requests offers is good enough that it outweighs
the fact
that it vendors urllib3 and continue using it.
 

There's also 3) fork requests, which is the democratic way to vote out
an upstream that isn't supporting the needs of the masses.

Given requests’ download count, I have to doubt that OpenStack users
constitute the masses in this case.

I don't think we're anywhere near there, but I wanted to make it clear
there _is_ a more extreme option.

 If the 2nd option is chosen, then doing anything but supporting the
fact that
 requests vendors urllib3 within the code that openstack writes is
hurting the
 users who fetch these projects from PyPI because you don't agree with
one of
 the choices that requests makes. By all means do conditional imports to
lessen
 the impact that the choice requests has made (and the one that
Openstack has
 made to use requests) on downstream distributors, but unconditionally
importing
 from the top level urllib3 for use within requests is flat out wrong.
 
 Obviously neither of these options excludes the choice to lean on
requests to
 reverse this decision as well. However that is best done elsewhere as
the
 person making that decision isn't a member of these mailing lists as
far as
 I am aware.
 

To be clear, I think we should keep using requests. But we should lend
our influence upstream and explain that our users are required to deal
with this in a way that perhaps hasn't been considered or given the
appropriate priority.

It’s been considered several times. There have been multiple issues.
There’s more than just the one you linked to. The decision is highly
unlikely to change whether it’s coming from a group of people in OpenStack
or another distribution package maintainer.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ceilometer] Work at bug Ceilometer HBase driver cannot handle non-ascii characters

2014-09-18 Thread Ilya Tyaptin
Hi, folks!

I want to engage this bug
https://bugs.launchpad.net/ceilometer/+bug/1350826 [1].

But fixing of this bug has raised question.
Issue is that happybase does not support unicode symbols and approaches to
processing unicode symbols is needed to discuss.

I suggest next variants:
1. Anyway to check containing unicode symbols in string, if contains then
screen theys.
2. Transform unicode symbols like a html encoding (/xa0 - %2Fxa0, for
example)

But in these approaches we can't be sure that we haven't false positives at
getting data from HBase.

Thanks for all, wait for your response!

[1] https://bugs.launchpad.net/ceilometer/+bug/1350826

-- 
Best regards,

Tyaptin Ilia,

Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ceilometer] Performance degradation (docs) for sampling rate less than 10 mins

2014-09-18 Thread Ilya Tyaptin
Hi Patanjali!

We have inspected this question and got a document with results of this
testing.
Tests are running on physical lab and includes 2000 VMs ran by Nova and 60
second polling interval.

Expanded information in doc
https://docs.google.com/a/mirantis.com/document/d/1jvqIy6fWQBvTZEfdnk37FvF5xtA3l7Cx09VNoe6bBRU

If you have any question I'll be happy to answer.

On Thu, Sep 18, 2014 at 7:56 PM, Srikanta Patanjali p.srika...@gmail.com
wrote:

 Hi Team,

 I was considering to change the default sampling rate of the Ceilometer
 from 10 mins to less than that. I foresee an adverse impact on its
 performance due to increase in the data inside the MondoDB.

 I was wondering if the QA team (or anyone else) has done any of the load
 tests with these parameters ? It would be helpful to have access to these
 results (if any).

 If not, I would like to know the views on increasing the sample rate value
 (in the pipleline.yaml file)

 Cheers,
 Srikanta
 InIT ¦ ICCLab
 ZHAW

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 

Best regards,

Tyaptin Ilia,

Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Thoughts on OpenStack Layers and a Big Tent model

2014-09-18 Thread Monty Taylor
Hey all,

I've recently been thinking a lot about Sean's Layers stuff. So I wrote
a blog post which Jim Blair and Devananda were kind enough to help me edit.

http://inaugust.com/post/108

Enjoy.

Monty

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Gordon Sim

On 09/18/2014 03:45 PM, Flavio Percoco wrote:

On 09/18/2014 04:09 PM, Gordon Sim wrote:

Is the replication synchronous or asynchronous with respect to client
calls? E.g. will the response to a post of messages be returned only
once the replication of those messages is confirmed? Likewise when
deleting a message, is the response only returned when replicas of the
message are deleted?


It depends on the driver implementation and/or storage configuration.
For example, in the mongodb driver, we use the default write concern
called acknowledged. This means that as soon as the message gets to
the master node (note it's not written on disk yet nor replicated) zaqar
will receive a confirmation and then send the response back to the
client.


So in that mode it's unreliable. If there is failure right after the 
response is sent the message may be lost, but the client believes it has 
been confirmed so will not resend.



This is also configurable by the deployer by changing the
default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0].

[0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w


So you could change that to majority to get reliable publication 
(at-least-once).


[...]

 From what I can see, pop provides unreliable delivery (i.e. its similar
to no-ack). If the delete call using pop fails while sending back the
response, the messages are removed but didn't get to the client.


Correct, pop works like no-ack. If you want to have pop+ack, it is
possible to claim just 1 message and then delete it.


Right, claim+delete is ack (and if the claim is replicated and 
recoverable etc you can verify whether deletion occurred to ensure 
message is processed only once). Using delete-with-pop is no-ak, i.e. 
at-most-once.



What do you mean by 'streaming messages'?


I'm sorry, that went out wrong. I had the browsability term in my head
and went with something even worse. By streaming messages I meant
polling messages without claiming them. In other words, at-least-once is
guaranteed by default, whereas once-and-only-once is guaranteed just if
claims are used.


I don't see that the claim mechanism brings any stronger guarantee, it 
just offers a competing consumer behaviour where browsing is 
non-competing (non-destructive). In both cases you require the client to 
be able to remember which messages it had processed in order to ensure 
exactly once. The claim reduces the scope of any doubt, but the client 
still needs to be able to determine whether it has already processed any 
message in the claim already.


[...]

That marker is a sequence number of some kind that is used to provide
ordering to queries? Is it generated by the database itself?


It's a sequence number to provide ordering to queries, correct.
Depending on the driver, it may be generated by Zaqar or the database.
In mongodb's case it's generated by Zaqar[0].


Zaqar increments a counter held within the database, am I reading that 
correctly? So mongodb is responsible for the ordering and atomicity of 
multiple concurrent requests for a marker?



[0]
https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Log Rationalization -- Bring it on!

2014-09-18 Thread Doug Hellmann

On Sep 17, 2014, at 7:42 PM, Rochelle.RochelleGrober 
rochelle.gro...@huawei.com wrote:

 TL;DR:  I consider the poor state of log consistency a major impediment for 
 more widespread adoption of OpenStack and would like to volunteer to own this 
 cross-functional process to begin to unify and standardize logging messages 
 and attributes for Kilo while dealing with the most egregious issues as the 
 community identifies them.
  
 Recap from some mail threads:
  
 From Sean Dague on Kilo cycle goals:
 2. Consistency in southbound interfaces (Logging first)
  
 Logging and notifications are south bound interfaces from OpenStack providing 
 information to people, or machines, about what is going on.
 There is also a 3rd proposed south bound with osprofiler.
  
 For Kilo: I think it's reasonable to complete the logging standards and 
 implement them. I expect notifications (which haven't quite kicked off) are 
 going to take 2 cycles.
  
 I'd honestly *really* love to see a unification path for all the the 
 southbound parts, logging, osprofiler, notifications, because there is quite 
 a bit of overlap in the instrumentation/annotation inside the main code for 
 all of these.
  
 And from Doug Hellmann:
 1. Sean has done a lot of analysis and started a spec on standardizing 
 logging guidelines where he is gathering input from developers, deployers, 
 and operators [1]. Because it is far enough for us to see real progress, it’s 
 a good place for us to start experimenting with how to drive cross-project 
 initiatives involving code and policy changes from outside of a single 
 project. We have a couple of potentially related specs in Oslo as part of the 
 oslo.log graduation work [2] [3], but I think most of the work will be within 
 the applications.
  
 [1] https://review.openstack.org/#/c/91446/
 [2] 
 https://blueprints.launchpad.net/oslo.log/+spec/app-agnostic-logging-parameters
 [3] https://blueprints.launchpad.net/oslo.log/+spec/remove-context-adapter
  
 And from James Blair:
 1) Improve log correlation and utility
  
 If we're going to improve the stability of OpenStack, we have to be able to 
 understand what's going on when it breaks.  That's both true as developers 
 when we're trying to diagnose a failure in an integration test, and it's true 
 for operators who are all too often diagnosing the same failure in a real 
 deployment.  Consistency in logging across projects as well as a 
 cross-project request token would go a long way toward this.
  
 While I am not currently managing an OpenStack deployment, writing tests or 
 code, or debugging the stack, I have spent many years doing just that.  
 Through QA, Ops and Customer support, I have come to revel in good logging 
 and log messages and curse the holes and vagaries in many systems.
  
 Defining/refining logs to be useful and usable is a cross-functional effort 
 that needs to include:
 · Operators
 · QA
 · End Users
 · Community managers
 · Tech Pubs
 · Translators
 · Developers
 · TC (which provides the forum and impetus for all the projects to 
 cooperate on this)
  
 At the moment, I think this effort may best work under the auspices of Oslo 
 (oslo.log), I’d love to hear other proposals.

I’m sure there will be changes to make in the log library. However, because of 
the cross-project nature of the policy decisions, I think we should drive this 
from outside of Oslo. We can use the oslo.log developer docs as a place to 
formally document guidelines, and we can change the library to make it easier 
to follow those guidelines, but the specs to define the guidelines and the 
planning for rolling out the changes should happen in a more central place than 
oslo-specs. 

  
 Here is the beginnings of my proposal of how to attack and subdue the painful 
 state of logs:
  
 · Post this email to the MLs (dev, ops, enduser) to get feedback, 
 garner support and participants in the process
 (Done;-)

FWIW, I’m only replying on the -dev list to avoid duplicate message from 
cross-posting. Figuring out how to gather input and collect it is one of the 
procedural issues we need to work out as part of starting an initiative like 
this. I like that you’ve started an etherpad for that.

We really do need to have the meta conversation about running cross-project 
initiatives, and I think this one has enough clear support that we could have 
that discussion without being side-tracked by what the initiative is trying to 
accomplish.

Doug

 · In parallel:
 o   Collect up problems, issues, ideas, solutions on an etherpad 
 https://etherpad.openstack.org/p/Log-Rationalization where anyone in the 
 communities can post.
 o   Categorize  reported Log issues into classes (already identified classes):
 §  Format Consistency across projects
 §  Log level definition and categorization across classes
 §  Time syncing entries across tens of logfiles
 §  Relevancy/usefulness of 

  1   2   >