Re: [openstack-dev] [osc][swift] Setting storage policy for a container possible via the client?
On 20/04/18 04:54, Dean Troyer wrote: On Thu, Apr 19, 2018 at 7:51 AM, Doug Hellmann wrote: Excerpts from Mark Kirkwood's message of 2018-04-19 16:47:58 +1200: Swift has had storage policies for a while now. These are enabled by setting the 'X-Storage-Policy' header on a container. It looks to me like this is not possible using openstack-client (even in master branch) - while there is a 'set' operation for containers this will *only* set 'Meta-*' type headers. It seems to me that adding this would be highly desirable. Is it in the pipeline? If not I might see how much interest there is at my end for adding such - as (famous last words) it looks pretty straightforward to do. I can't imagine why we wouldn't want to implement that and I'm not aware of anyone working on it. If you're interested and have time, please do work on the patch(es). The primary thing that hinders Swift work like this is OSC does not use swiftclient as it wasn't a standalone thing yet when I wrote that bit (lifting much of the actual API code from swiftclient) . We decided a while ago to not add that dependency and drop the OSC-specific object code and use the SDK when we start using SDK for everything else, after there is an SDK 1.0 release. Moving forward on this today using either OSC's api.object code or the SDK would be fine, with the same SDK caveat we have with Neutron, since SDK isn't 1.0 we may have to play catch-up and maintain multiple SDK release compatibilities (which has happened at least twice). Ok, understood. I've uploaded a small patch that adding policy specification to 'container create' and adds some policy details display to 'container show' and 'object store account show' [1]. It uses the existing api design, but tries to get the display to look a little like what the swift cli provides (particularly for the account info). regards Mark [1] Gerrit topic is objectstorepolicies __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [osc][swift] Setting storage policy for a container possible via the client?
Swift has had storage policies for a while now. These are enabled by setting the 'X-Storage-Policy' header on a container. It looks to me like this is not possible using openstack-client (even in master branch) - while there is a 'set' operation for containers this will *only* set 'Meta-*' type headers. It seems to me that adding this would be highly desirable. Is it in the pipeline? If not I might see how much interest there is at my end for adding such - as (famous last words) it looks pretty straightforward to do. regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][all][tc] A proposal to rearchitect Trove
On 21/06/17 02:08, Jay Pipes wrote: On 06/20/2017 09:42 AM, Doug Hellmann wrote: Does "service VM" need to be a first-class thing? Akanda creates them, using a service user. The VMs are tied to a "router" which is the billable resource that the user understands and interacts with through the API. Frankly, I believe all of these types of services should be built as applications that run on OpenStack (or other) infrastructure. In other words, they should not be part of the infrastructure itself. There's really no need for a user of a DBaaS to have access to the host or hosts the DB is running on. If the user really wanted that, they would just spin up a VM/baremetal server and install the thing themselves. Yes, I think this area is where some hard thinking would be rewarded. I recall when I first met Trove, in my mind I expected to be 'carving off a piece of database'...and was a bit surprised to discover that it (essentially) leveraged Nova VM + OS + DB (no criticism intended - just saying I was surprised). Of course after delving into how it worked I realized that it did make sense to make use of the various Nova things (schedulers etc)*but* now we are thinking about re-architecting (plus more options exist now), it would make sense to revisit this area. Best wishes Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] Erasure coding and geo replication
Hi, Has the release of 2.7 significantly changed the assessment here? Thanks Mark On 15/02/16 23:29, Kota TSUYUZAKI wrote: Hello Mark, AFAIK, a few reasons for that we still are in working progress for erasure code + geo replication. and expect to survive a region outage... With that I mind I did some experiments (Liberty swift) and it looks to me like if you have: - num_data_frags < num_nodes in (smallest) region and: - num_parity_frags = num_data_frags then having a region fail does not result in service outage. Good point but note that the PyECLib v1.0.7 (pinned to Kilo/Liberty stable) still have a problem which cannot decode the original data when all feed fragments are parity frags[1]. (i.e. if set num_parity_frags = num_data frags and then, num_parity_frags comes into proxy for GET request, it will fail at the decoding) The problem was already resolved in the PyECLib/liberasurecode at master branch and current swift master has the PyECLib>=1.0.7 dependencies so if you thought to use the newest Swift, it might be not a matter. In the Swift perspective, I think that we need more tests/discussion for geo replication around write/read affinity[2] which is geo replication stuff in Swift itself and performances. For the write/read affinity, actually we didn't consider the affinity control to simplify the implementation until EC landed into Swift master[3] so I think it's time to make sure how we can use the affinity control with EC but it's not done yet. For the performance perspective, in my experiments, more parities causes quite performance degradation[4]. To prevent the degradation, I am working for the spec which makes duplicated copy from data/parity fragments and spread them out into geo regions. To sumurize, we've not done the work yet but we welcome to discuss and contribute for EC + geo replication anytime, IMO. Thanks, Kota 1: https://bitbucket.org/tsg-/liberasurecode/commits/a01b1818c874a65d1d1fb8f11ea441e9d3e18771 2: http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters 3: http://docs.openstack.org/developer/swift/overview_erasure_code.html#region-support 4: https://specs.openstack.org/openstack/swift-specs/specs/in_progress/global_ec_cluster.html (2016/02/15 18:00), Mark Kirkwood wrote: After looking at: https://www.youtube.com/watch?v=9YHvYkcse-k I have a question (that follows on from Bruno's) about using erasure coding with geo replication. Now the example given to show why you could/should not use erasure coding with geo replication is somewhat flawed as it is immediately clear that you cannot set: - num_data_frags > num_devices (or nodes) in a region and expect to survive a region outage... With that I mind I did some experiments (Liberty swift) and it looks to me like if you have: - num_data_frags < num_nodes in (smallest) region and: - num_parity_frags = num_data_frags then having a region fail does not result in service outage. So my real question is - it looks like it *is* possible to use erasure coding in geo replicated situations - however I may well be missing something significant, so I'd love some clarification here [1]! Cheers Mark [1] Reduction is disk usage and net traffic looks attractive __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] Erasure coding and geo replication
On 16/02/16 17:10, Mark Kirkwood wrote: On 15/02/16 23:29, Kota TSUYUZAKI wrote: Hello Mark, AFAIK, a few reasons for that we still are in working progress for erasure code + geo replication. and expect to survive a region outage... With that I mind I did some experiments (Liberty swift) and it looks to me like if you have: - num_data_frags < num_nodes in (smallest) region and: - num_parity_frags = num_data_frags then having a region fail does not result in service outage. Good point but note that the PyECLib v1.0.7 (pinned to Kilo/Liberty stable) still have a problem which cannot decode the original data when all feed fragments are parity frags[1]. (i.e. if set num_parity_frags = num_data frags and then, num_parity_frags comes into proxy for GET request, it will fail at the decoding) The problem was already resolved in the PyECLib/liberasurecode at master branch and current swift master has the PyECLib>=1.0.7 dependencies so if you thought to use the newest Swift, it might be not a matter. Ah right, in my testing I always took down my "1st" region...which will have had data fragments therein. For interest I'll try to provoke a situation where I have all parity ones to assemble (and see what happens). So tried this out - still works fine. Checking version of pyeclib I see Ubuntu 15.10 is giving me: - Swift 2.5.0 - pyeclib 1.0.8 Hmmm - Canonical deliberately upping the version of pyeclib (shock)? Interesting...anyway explains why I cannot get it to fail. However, all your other points are noted, and again thanks! Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] Erasure coding and geo replication
On 15/02/16 23:29, Kota TSUYUZAKI wrote: Hello Mark, AFAIK, a few reasons for that we still are in working progress for erasure code + geo replication. and expect to survive a region outage... With that I mind I did some experiments (Liberty swift) and it looks to me like if you have: - num_data_frags < num_nodes in (smallest) region and: - num_parity_frags = num_data_frags then having a region fail does not result in service outage. Good point but note that the PyECLib v1.0.7 (pinned to Kilo/Liberty stable) still have a problem which cannot decode the original data when all feed fragments are parity frags[1]. (i.e. if set num_parity_frags = num_data frags and then, num_parity_frags comes into proxy for GET request, it will fail at the decoding) The problem was already resolved in the PyECLib/liberasurecode at master branch and current swift master has the PyECLib>=1.0.7 dependencies so if you thought to use the newest Swift, it might be not a matter. Ah right, in my testing I always took down my "1st" region...which will have had data fragments therein. For interest I'll try to provoke a situation where I have all parity ones to assemble (and see what happens). In the Swift perspective, I think that we need more tests/discussion for geo replication around write/read affinity[2] which is geo replication stuff in Swift itself and performances. For the write/read affinity, actually we didn't consider the affinity control to simplify the implementation until EC landed into Swift master[3] so I think it's time to make sure how we can use the affinity control with EC but it's not done yet. For the performance perspective, in my experiments, more parities causes quite performance degradation[4]. To prevent the degradation, I am working for the spec which makes duplicated copy from data/parity fragments and spread them out into geo regions. To sumurize, we've not done the work yet but we welcome to discuss and contribute for EC + geo replication anytime, IMO. Thanks, Kota 1: https://bitbucket.org/tsg-/liberasurecode/commits/a01b1818c874a65d1d1fb8f11ea441e9d3e18771 2: http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters 3: http://docs.openstack.org/developer/swift/overview_erasure_code.html#region-support 4: https://specs.openstack.org/openstack/swift-specs/specs/in_progress/global_ec_cluster.html Excellent - thank you for a very comprehensive answer. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Swift] Erasure coding and geo replication
After looking at: https://www.youtube.com/watch?v=9YHvYkcse-k I have a question (that follows on from Bruno's) about using erasure coding with geo replication. Now the example given to show why you could/should not use erasure coding with geo replication is somewhat flawed as it is immediately clear that you cannot set: - num_data_frags > num_devices (or nodes) in a region and expect to survive a region outage... With that I mind I did some experiments (Liberty swift) and it looks to me like if you have: - num_data_frags < num_nodes in (smallest) region and: - num_parity_frags = num_data_frags then having a region fail does not result in service outage. So my real question is - it looks like it *is* possible to use erasure coding in geo replicated situations - however I may well be missing something significant, so I'd love some clarification here [1]! Cheers Mark [1] Reduction is disk usage and net traffic looks attractive __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [swift] Storage service startup should await ring creation
On 10/07/15 12:43, Mark Kirkwood wrote: Hi, I am using puppet-swift to deploy a swift multi node cluster (Icehouse), following the setup in supplied tests/site.pp. I am running into two issues that seem to be related to the subject above: 1/ Errors when the storage replication services try to start before the ring files exist. e.g: Error: Could not start Service[swift-object-replicator]: Execution of '/sbin/start swift-object-replicator' returned 1: start: Job failed to start Wrapped exception: Execution of '/sbin/start swift-object-replicator' returned 1: start: Job failed to start Error: /Stage[main]/Swift::Storage::Object/Swift::Storage::Generic[object]/Service[swift-object-replicator]/ensure: change from stopped to running failed: Could not start Service[swift-object-replicator]: Execution of '/sbin/start swift-object-replicator' returned 1: start: Job failed to start Now these will be fixed the *next* time I do a puppet run (provided I've performed a run on the appropriate proxy/ringmaster). However the failing services make scripted testing difficult as we have to put in logic to the effect "don't worry about errors the 1st time". 2/ Container and object stats not updated without full restart of services This one is a bit more subtle - everything works but 'swift stat' always shows zero objects and bytes for every container. The only way to fix this is to stop and start all services on each storage node. Again this complicates scripted builds as there is the need to go and stop + start all the swift storage services! Not to mention an extra little quirk for ops to remember at zero dark 30 oclock... I've made a patch that prevents these services starting until the ring files exist (actually for now it just checks the object ring) - see attached. Now while I'm not entirely sure that this is the best way to solve the issue (custom fact that changes the service start flag)...I *do* think that making the storage services await the ring existence *is* a needed change, so any thoughts on better ways to do this are appreciated. Also note that this change *does* require one more puppet run on the storage nodes: - one to create the storage servers config and drives - one to get the ring from the proxy/ringmaster - one to start the services I decided to work around these rather than trying to battle in my patch to the swift module. For 1/ I'm trapping the return code for the 1st puppet run and handling errors there...run not doing anything for any subsequent run as there shouldn't be any errors thereafter. Seems to work ok For 2/ I'm inserting an exec in our driving puppet code to just stop and start (not restart as that does not solve it...growl) *all* the services on a storage node. e.g (see tests/site.pp in the swift module for context): # collect resources for synchronizing the ring databases Swift::Ringsync<<||>> # stop and start all swift services # this is a workaround due to the services starting before the ring # is synced which stops stats (and maybe other things) working. # We can't just restart as this does *not* achieve the same result. exec { 'stop-start-services': provider => shell, command => 'for x in `initctl list|grep swift|awk \'{print $1}\'`;do stop $x;start $x;done;exit 0', path => '/usr/bin:/usr/sbin:/bin:/sbin', logoutput=> true, } # make sure we stop and start all services after syncing ring Swift::Ringsync<||> -> Exec['stop-start-services'] Ok, so it is a pretty big hammer, but leaves all of the services in a known good state, so I'm happy. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] [ceilometer] Ceilometer log dir permissions bust swift proxy
On 14/07/15 11:05, Mark Kirkwood wrote: On 13/07/15 06:44, Emilien Macchi wrote: Yeah, I guess I should raise a bug with Ubuntu so it (maybe) gets sorted out. However in the meantime we may have to work around it by amending the dir permissions in puppet (the alternative - rolling our own packages is undesirable in general). However I note there are also python version mistakes in the ceilometer requires.txt which need patching (sigh). For completeness, with reference to that last point the latest (2015.1.5) releases for ceilometer on Ubuntu 14.04 seem to have corrected the requires.txt (which is great). Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] [ceilometer] Ceilometer log dir permissions bust swift proxy
On 13/07/15 06:44, Emilien Macchi wrote: I think this is something that could be fixed in packaging scripts. I can see you're using Puppet to deploy OpenStack, and fwiw, we are stopping to manage permissions in Puppet because of packaging overlap. From now, we totally rely on packaging permissions. We are in the process to drop all the code that hardcode POSIX User/groups/permissions in our manifests. Yeah, I guess I should raise a bug with Ubuntu so it (maybe) gets sorted out. However in the meantime we may have to work around it by amending the dir permissions in puppet (the alternative - rolling our own packages is undesirable in general). However I note there are also python version mistakes in the ceilometer requires.txt which need patching (sigh). Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [swift] [ceilometer] Ceilometer log dir permissions bust swift proxy
Hi, I'm deploying a swift 1.13 cluster on Ubuntu 14.04 and enabling ceilometer in the proxy pipeline results in it not working. The cause appears to be the log directory perms (note I am running the proxy under Apache): [Fri Jul 10 05:12:15.126214 2015] [:error] [pid 6844:tid 140048779998976] [remote 192.168.5.1:21419] IOError: [Errno 13] Permission denied: '/var/log/ceilometer/proxy-server.wsgi.log' Sure enough: $ ls -la /var/log/ceilometer/ total 16 drwxr-x--- 2 ceilometer adm4096 Jul 10 02:55 . Looking at the swift user: $ id swift uid=50144(swift) gid=50145(swift) groups=50145(swift),4(adm),106(puppet),115(ceilometer) This looks like https://bugs.launchpad.net/swift/+bug/1269473 but the code I have includes the fix for that. In fact it looks like the directory permissions are just being set wrong (indeed chmod'ing them to be 770 fixes this). Am I missing something? I don't see how this can possibly work unless the directory allows group write. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [puppet] [swift] Storage service startup should await ring creation
Hi, I am using puppet-swift to deploy a swift multi node cluster (Icehouse), following the setup in supplied tests/site.pp. I am running into two issues that seem to be related to the subject above: 1/ Errors when the storage replication services try to start before the ring files exist. e.g: Error: Could not start Service[swift-object-replicator]: Execution of '/sbin/start swift-object-replicator' returned 1: start: Job failed to start Wrapped exception: Execution of '/sbin/start swift-object-replicator' returned 1: start: Job failed to start Error: /Stage[main]/Swift::Storage::Object/Swift::Storage::Generic[object]/Service[swift-object-replicator]/ensure: change from stopped to running failed: Could not start Service[swift-object-replicator]: Execution of '/sbin/start swift-object-replicator' returned 1: start: Job failed to start Now these will be fixed the *next* time I do a puppet run (provided I've performed a run on the appropriate proxy/ringmaster). However the failing services make scripted testing difficult as we have to put in logic to the effect "don't worry about errors the 1st time". 2/ Container and object stats not updated without full restart of services This one is a bit more subtle - everything works but 'swift stat' always shows zero objects and bytes for every container. The only way to fix this is to stop and start all services on each storage node. Again this complicates scripted builds as there is the need to go and stop + start all the swift storage services! Not to mention an extra little quirk for ops to remember at zero dark 30 oclock... I've made a patch that prevents these services starting until the ring files exist (actually for now it just checks the object ring) - see attached. Now while I'm not entirely sure that this is the best way to solve the issue (custom fact that changes the service start flag)...I *do* think that making the storage services await the ring existence *is* a needed change, so any thoughts on better ways to do this are appreciated. Also note that this change *does* require one more puppet run on the storage nodes: - one to create the storage servers config and drives - one to get the ring from the proxy/ringmaster - one to start the services Regards Mark >From 1583d68eedbeaecbacb5a29258343b9e980ce4a4 Mon Sep 17 00:00:00 2001 From: Mark Kirkwood Date: Fri, 10 Jul 2015 11:18:16 +1200 Subject: [PATCH] Make storage service startup wait for the ring This solves two problems: 1/ Errors when the replication services fail to start when the ring files are absent. 2/ Container and object stats not updated without full restart of services While neither of these issues prevent a working deployment, they play havoc with fully automated deployment for ci etc (and make arcane knowledge necessary for ops). This change *does* require one more puppet run on the storage nodes: - one to create the storage servers config and drives - one to get the ring from the proxy/ringmaster - one to start the services (amend: make sure the custom fact goes in the correct dir) Change-Id: I168c26aed5f6dfc337ea8bc5f863e15f6e86d4a2 --- modules/swift/lib/facter/ringfact.rb | 5 + modules/swift/manifests/storage/account.pp | 14 ++ modules/swift/manifests/storage/container.pp | 17 - modules/swift/manifests/storage/object.pp| 15 +++ 4 files changed, 38 insertions(+), 13 deletions(-) create mode 100644 modules/swift/lib/facter/ringfact.rb diff --git a/modules/swift/lib/facter/ringfact.rb b/modules/swift/lib/facter/ringfact.rb new file mode 100644 index 000..4816cb8 --- /dev/null +++ b/modules/swift/lib/facter/ringfact.rb @@ -0,0 +1,5 @@ +Facter.add("ringexists") do + setcode do +File.exists?("/etc/swift/object.ring.gz") ? "true" : "false" + end +end diff --git a/modules/swift/manifests/storage/account.pp b/modules/swift/manifests/storage/account.pp index a4398c3..1719776 100644 --- a/modules/swift/manifests/storage/account.pp +++ b/modules/swift/manifests/storage/account.pp @@ -18,16 +18,22 @@ class swift::storage::account( $enabled= true, $package_ensure = 'present' ) { + if $::ringexists == "false" { +$service_enabled = false + } else { +$service_enabled = $enabled + } + swift::storage::generic { 'account': manage_service => $manage_service, -enabled=> $enabled, +enabled=> $service_enabled, package_ensure => $package_ensure, } include swift::params if $manage_service { -if $enabled { +if $service_enabled { $service_ensure = 'running' } else { $service_ensure = 'stopped' @@ -37,7 +43,7 @@ class swift::storage::account( service { 'swift-account-reaper': ensure
Re: [openstack-dev] Debian already using Python 3.5: please gate on that
On 21/06/15 01:32, Doug Hellmann wrote: Excerpts from Dave Walker's message of 2015-06-20 14:05:48 +0100: On 20 Jun 2015 1:05 pm, "Doug Hellmann" wrote: Whether we want to support 3.4 and 3.5, or just 3.4 and then just 3.5 is an ecosystem question IMO, not an upstream one. 3.4 and 3.5 are very similar when you consider the feature set crossover with 2.7. Right, and IIRC that's why we said at the summit that we would rather not take the resources (in terms of people and CI servers) to run tests against both, yet. Let's get one project fully working on one version of python 3, and then we can start thinking about whether we need to test multiple versions. Having information available, even if there is not immediate intent to support seems like something we should constantly be driving for. It should certainly be non-voting, but if resources are limited - perhaps a periodic job, rather that gate? OTOH, if Canonical doesn't release a version of 3.4 that removes the core dump bug soon, I will support moving fully to 3.5 or another test platform, because that bug is causing us trouble in Oslo still. s/Canonical/ubuntu Can you link to the bug? I did a quick search, but couldn't find it quickly. Hmm, https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907 is marked as released but somehow it's still being triggered for oslo.messaging. FWIW http://packages.ubuntu.com/search?keywords=python3.4 suggests that Trusty is still using python 3.4.0, and you don't get 3.4.3 until Vivid, which is a bit sad - definitely worth quizzing the Ubuntu folks about what the plan is there. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
With respect to using a seed - the facility to supply one to the rebalance operation has recently been added to puppet-swift master branch (commit b8b4434), however the seed parameter is not available to any of the usual calling methods (this looks to be deliberate from the commit message), so is not immediately useful without surgery :-) Regards Mark On 13/06/15 18:05, Mark Kirkwood wrote: From what I can see, the ring gets created and rebalanced in puppet-swift/manifest/ringbuilder.pp i.e calling: class { '::swift::ringbuilder': # the part power should be determined by assuming 100 partitions per drive part_power => '18', replicas => '3', min_part_hours => 1, require=> Class['swift'], } *not* when each device is added. Yeah, using a seed is probably a good solution too. For the moment I'm using the idea of one proxy being a 'ring server/master' which achieves the same thing (identical rings everywhere). However I'll have a look at using a seed, as this may simplify the code and also the operational procedure needed to replace said 'master' if it fails (i.e to avoid accidentally creating a new ring when you really don't need to...) Regards, Mark On 12/06/15 23:10, McCabe, Donagh wrote: I skimmed the code, but since I'm not familiar with the environment, I could not find where "swift-ring-builder rebalance" is invoked. I'm guessing that each time you add a device to a ring, a rebalance is also done. Leaving aside how inefficient that is, the key thing is that the rebalance command has an optional "seed" parameter. Unless you explicitly set the seed (to same value on all node obviously), you won't get the same ring on all nodes. You also need to make sure you add the same set of drives and in same order. Regards, Donagh -Original Message- From: Mark Kirkwood [mailto:mark.kirkw...@catalyst.net.nz] Sent: 12 June 2015 06:28 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings I've looking at using puppet-swift to deploy a swift cluster. Firstly - without http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp I would have struggled a great deal more to get up and running, so a big thank you for a nice worked example of how to do multiple nodes! However I have stumbled upon a problem - with respect to creating multiple proxy nodes. There are some recipes around that follow on from the site.pp above and explicitly build >1 proxy (e.g https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp) Now the problem is - each proxy node does a ring builder create, so ends up with *different* builder (and therefore) ring files. This is not good, as the end result is a cluster with all storage nodes and *one* proxy with the same set of ring files, and *all* other proxies with *different* ring (and builder) files. I have used logic similar to the attached to work around this, i.e only create rings if we are the 'ring server', otherwise get 'em via rsync. Thoughts? Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
From what I can see, the ring gets created and rebalanced in puppet-swift/manifest/ringbuilder.pp i.e calling: class { '::swift::ringbuilder': # the part power should be determined by assuming 100 partitions per drive part_power => '18', replicas => '3', min_part_hours => 1, require=> Class['swift'], } *not* when each device is added. Yeah, using a seed is probably a good solution too. For the moment I'm using the idea of one proxy being a 'ring server/master' which achieves the same thing (identical rings everywhere). However I'll have a look at using a seed, as this may simplify the code and also the operational procedure needed to replace said 'master' if it fails (i.e to avoid accidentally creating a new ring when you really don't need to...) Regards, Mark On 12/06/15 23:10, McCabe, Donagh wrote: I skimmed the code, but since I'm not familiar with the environment, I could not find where "swift-ring-builder rebalance" is invoked. I'm guessing that each time you add a device to a ring, a rebalance is also done. Leaving aside how inefficient that is, the key thing is that the rebalance command has an optional "seed" parameter. Unless you explicitly set the seed (to same value on all node obviously), you won't get the same ring on all nodes. You also need to make sure you add the same set of drives and in same order. Regards, Donagh -Original Message- From: Mark Kirkwood [mailto:mark.kirkw...@catalyst.net.nz] Sent: 12 June 2015 06:28 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings I've looking at using puppet-swift to deploy a swift cluster. Firstly - without http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp I would have struggled a great deal more to get up and running, so a big thank you for a nice worked example of how to do multiple nodes! However I have stumbled upon a problem - with respect to creating multiple proxy nodes. There are some recipes around that follow on from the site.pp above and explicitly build >1 proxy (e.g https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp) Now the problem is - each proxy node does a ring builder create, so ends up with *different* builder (and therefore) ring files. This is not good, as the end result is a cluster with all storage nodes and *one* proxy with the same set of ring files, and *all* other proxies with *different* ring (and builder) files. I have used logic similar to the attached to work around this, i.e only create rings if we are the 'ring server', otherwise get 'em via rsync. Thoughts? Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
On 12/06/15 17:27, Mark Kirkwood wrote: I've looking at using puppet-swift to deploy a swift cluster. Firstly - without http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp I would have struggled a great deal more to get up and running, so a big thank you for a nice worked example of how to do multiple nodes! However I have stumbled upon a problem - with respect to creating multiple proxy nodes. There are some recipes around that follow on from the site.pp above and explicitly build >1 proxy (e.g https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp) Now the problem is - each proxy node does a ring builder create, so ends up with *different* builder (and therefore) ring files. This is not good, as the end result is a cluster with all storage nodes and *one* proxy with the same set of ring files, and *all* other proxies with *different* ring (and builder) files. I have used logic similar to the attached to work around this, i.e only create rings if we are the 'ring server', otherwise get 'em via rsync. Thoughts? I should have noted that the previously mentioned site.pp introduced the idea of a ringmaster or ringserver host, and I made use of that i.e only one proxy actually builds the rings. Also I see in my effort to provide a simple fragment I left off a $ in front of some variables, and I should probably have defined the ringserver ip to make it clear what was happening, e.g: $ringserver_local_net_ip = '192.168.5.200' if $ipaddress_eth0 == $ringserver_local_net_ip { Cheers Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
I've looking at using puppet-swift to deploy a swift cluster. Firstly - without http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp I would have struggled a great deal more to get up and running, so a big thank you for a nice worked example of how to do multiple nodes! However I have stumbled upon a problem - with respect to creating multiple proxy nodes. There are some recipes around that follow on from the site.pp above and explicitly build >1 proxy (e.g https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp) Now the problem is - each proxy node does a ring builder create, so ends up with *different* builder (and therefore) ring files. This is not good, as the end result is a cluster with all storage nodes and *one* proxy with the same set of ring files, and *all* other proxies with *different* ring (and builder) files. I have used logic similar to the attached to work around this, i.e only create rings if we are the 'ring server', otherwise get 'em via rsync. Thoughts? Regards Mark # create the ring if we are the ring server if $ipaddress_eth0 == ringserver_local_net_ip { class { 'swift::ringbuilder': # the part power should be determined by assuming 100 partitions per drive part_power => '18' replicas => '2' min_part_hours => 1 require=> Class['swift'], } # sets up an rsync db that can be used to sync the ring DB class { 'swift::ringserver': local_net_ip => $ipaddress_eth0, } # exports rsync gets that can be used to sync the ring files @@swift::ringsync { ['account', 'object', 'container']: ring_server => $ipaddress_eth0, } } else { # collect resources for synchronizing the ring databases Swift::Ringsync<<||>> } __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] who is the ptl of trove?
On 09/05/15 02:28, Monty Taylor wrote: On 05/08/2015 03:45 AM, Nikhil Manchanda wrote: Comments and answers inline. Li Tianqing writes: [...] 1) why we put the trove vm into user's tenant, not the trove's tenant? User can login on that vm, and that vm must connect to rabbitmq. It is quite insecure. what's about put the tenant into trove tenant? While the default configuration of Trove in devstack puts Trove guest VMs into the users' respective tenants, it's possible to configure Trove to create VMs in a single "Trove" tenant. You would do this by overriding the default novaclient class in Trove's remote.py with one that creates all Trove VMs in a particular tenant whose user credentials you will need to supply. In fact, most production instances of Trove do something like this. Might I suggest that if this is how people regularly deploy, that such a class be included in trove proper, and that a config option be provided like "use_tenant='name_of_tenant_to_use'" that would trigger the use of the overridden novaclient class? I think asking an operator as a standard practice to override code in remote.py is a bad pattern. 2) Why there is no trove mgmt cli, but mgmt api is in the code? Does it disappear forever ? The reason for this is because the old legacy Trove client was rewritten to be in line with the rest of the openstack clients. The new client has bindings for the management API, but we didn't complete the work on writing the shell pieces for it. There is currently an effort to support Trove calls in the openstackclient, and we're looking to support the management client calls as part of this as well. If this is something that you're passionate about, we sure could use help landing this in Liberty. 3) The trove-guest-agent is in vm. it is connected by taskmanager by rabbitmq. We designed it. But is there some prectise to do this? how to make the vm be connected in vm-network and management network? Most deployments of Trove that I am familiar with set up a separate RabbitMQ server in cloud that is used by Trove. It is not recommended to use the same infrastructure RabbitMQ server for Trove for security reasons. Also most deployments of Trove set up a private (neutron) network that the RabbitMQ server and guests are connected to, and all RPC messages are sent over this network. This sounds like a great chunk of information to potentially go into deployer docs. I'd like to +1 this. It is misleading that the standard documentation (and the devstack setup) describes a configuration that is unsafe/unwise to use in production. This is surely unusual to say the least! Normally when test or dev setups use unsafe configurations the relevant docs clearly state this - and describe how it should actually be done. In addition the fact that several extended question threads were required to extract this vital information is ...disappointing, and does not display the right spirit for an open source project in my opinion! Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Mysql db connection leaking?
On 17/04/15 09:20, Qiming Teng wrote: Wondering if there is something misconfigured in my devstack environment, which was reinstalled on RHEL7 about 10 days ago. I'm often running into mysql connections problem as shown below: $ mysql ERROR 1040 (HY000): Too many connections When I try dump the mysql connection list, I'm getting the followng result after a 'systemctl restart mariadb.service': $ mysqladmin processlist | grep nova | wc -l 125 Most of the connections are at Sleep status: $ mysqladmin processlist | grep nova | grep Sleep | wc -l 123 As for the workload, I'm currently only running two VMs in a multi-host devstack environment. So, my questions: - Why do we have so many mysql connections from nova? - Is it possible this is caused some misconfigurations? - 125 connections in such a toy setup is insane, any hints on nailing down the connections to the responsible nova components? Can you show us the full listing for mysqladmin processlist as 125 seems high for a toy setup? How many nova instances are running? I have a (single node) devstack setup with 2 nova instances running and I have 9 mysql connections to the nova db. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] Confused about nova_proxy_admin_* settings
Thanks Nikhil, I figured I must have missed something that actually used the proxy - I didn't have metering enabled in devstack. I'll enable it and (I guess) it should fail until I give it correct proxy admin credentials... Cheers Mark On 22/01/15 05:55, Nikhil Manchanda wrote: Hi Mark: It's been a little while since I looked at this last, but as I recall these values seem to be used and needed only by the trove taskmanager. If you have support for metering messages turned on, this account gets used to look up instance details when sending periodic metering messages to an AMQP exchange. These aren't needed on the guest-agent, and that's probably the reason why a non-existing value of "radmin" seems to work. The fact that this exists in the guest configuration as well is probably a bug and should be cleaned up. Cheers, Nikhil On Tue, Jan 20, 2015 at 4:05 PM, Mark Kirkwood < mark.kirkw...@catalyst.net.nz> wrote: I've been looking at how the 3 nova_proxy_admin_* settings are used. I'm coming to the conclusion that I'm confused: E.g I note that a standard devstack (stable/juno branch) with trove enabled sets these as follows: nova_proxy_admin_pass = nova_proxy_admin_tenant_name = trove nova_proxy_admin_user = radmin However there is no 'radmin' user (or role) created in keystone, so the settings above cannot possibly work (if they were needed/used). Some experimentation involving removing these three settings from all of the trove config files seems to support the idea that they are in fact not needed, which has me somewhat puzzled. If someone could shed some light here that would be awesome! Thanks Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [trove] Confused about nova_proxy_admin_* settings
I've been looking at how the 3 nova_proxy_admin_* settings are used. I'm coming to the conclusion that I'm confused: E.g I note that a standard devstack (stable/juno branch) with trove enabled sets these as follows: nova_proxy_admin_pass = nova_proxy_admin_tenant_name = trove nova_proxy_admin_user = radmin However there is no 'radmin' user (or role) created in keystone, so the settings above cannot possibly work (if they were needed/used). Some experimentation involving removing these three settings from all of the trove config files seems to support the idea that they are in fact not needed, which has me somewhat puzzled. If someone could shed some light here that would be awesome! Thanks Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] confused about trove-guestagent need nova's auth info
On 11/01/15 22:25, Denis Makogon wrote: Guest agent doesn't need configuration options described above. IIRC, only taskmanager needs them. Right - so we need to update the default config files and doco - as they have them in there. About passing auth data. What are those benefits of changing the way in which auth data is shipped? If you still think of security risks - you may use SSL protocol that is available in most of messaging services. I guessing the original poster was thinking along these lines: breaking into the image gives the tenant access to privileged passwords. Whether the communication is SSL or not is another (interesting) factor. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] confused about trove-guestagent need nova's auth info
On 18/12/14 14:30, 乔建 wrote: When using trove, we need to configure nova’s user information in the configuration file of trove-guestagent, such as lnova_proxy_admin_user lnova_proxy_admin_pass lnova_proxy_admin_tenant_name Is it necessary? In a public cloud environment, It will lead to serious security risks. I traced the code, and noticed that the auth data mentioned above is packaged in a context object, then passed to the trove-conductor via message queue. Is it more suitable for trove-conductor to get the corresponding information from its own conf file? Yes - all good points. Experimenting with devstack Juno branch, it seems you can happily remove these three settings. However the guest agent does seem to need the rabbit host and password, which is probably undesirable for the same reasons that you mentioned above. Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] MySQL performance and Mongodb backend maturity question
On 25/09/14 15:37, Qiming Teng wrote: Hi, Some weeks ago, I checked my then latest devstack install and I learned this: event support in Ceilometer is only available for sqlalchemy backend; mongodb backend was still under development. I have been using MySQL during the past weeks and now I think I'm trapped by a performance problem of MySQL. One or two Nova servers were launched and remain idle for about 10 days. Now I'm seeing a lot of data accumulated in db and I wanted to cleanse it manually. Here is what I got: mysql> select count(*) from metadata_text; +--+ | count(*) | +--+ | 25249913 | +--+ 1 row in set (3.83 sec) mysql> delete from metadata_text limit 1000; Query OK, 1000 rows affected (0.02 sec) mysql> delete from metadata_text limit 1; Query OK, 1 rows affected (0.39 sec) mysql> delete from metadata_text limit 10; Query OK, 10 rows affected (2.31 sec) mysql> delete from metadata_text limit 100; Query OK, 100 rows affected (25.32 sec) mysql> delete from metadata_text limit 200; Query OK, 200 rows affected (1 min 16.17 sec) mysql> delete from metadata_text limit 400; Query OK, 400 rows affected (7 min 40.40 sec) There were 25M records in one table. The deletion time is reaching an unacceptable level (7 minutes for 4M records) and it was not increasing in a linear way. Maybe DB experts can show me how to optimize this? Writes of bigger datasets will take non linear time when (possibly default?) configs are outgrown. For instance (assumimg metadata_text is an innodb table, take a look at: - innodb_log_buffer_size - innodb_log_file_size (warning: read the manual carefully before changing this) - innodb_buffer_pool_size Also index maintenance can get to be a limiting factor, I'm not sure if mysql will use the sort buffer to help with this, but maybe try increase - sort_buffer_size (just for the session doing the delete) and see if it helps. There are many (way too many) other parameters to tweak, but the above ones are probably the best to start with. Cheers Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Trove] Backup/restore namespace config move has leftovers
On 02/08/14 18:24, Denis Makogon wrote: Mark, we don't have to add backup/restore namespace options to datastores that does't support backup/restore feature. You should take a look how backup procedure is being executed at Trove-API service site, see https://github.com/openstack/trove/blob/master/trove/backup/models.py (Method called _validate_can_perform_action) If you'll have another questions, feel free to catch me up at IRC (denis_makogon). Thanks Denis - I did wonder if it was an optional specification! Doh! However, while I'm a bit ignorant wrt redis and cassandra, I do have a bit todo with mongo and that certainly *does* support backup/restore... Cheers Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Trove] Backup/restore namespace config move has leftovers
On 01/08/14 21:35, Denis Makogon wrote: I'd suggest to file a bug report and fix given issue. Done. https://bugs.launchpad.net/trove/+bug/1351545 I also took the opportunity to check if all the currently defined datastores had backup/restore_namespace set - they didn't, so I noted that too (I'm guessing they now actually *need* to have something set to avoid breakage)... regards Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Trove] Backup/restore namespace config move has leftovers
In my latest devstack pull I notice that backup_namespace restore_namespace have moved from the default conf group to per datastore (commit 61935d3). However they still appear in the common_opts section of trove/common/cfg.py This seems like an oversight - or is there something I'm missing? Cheers Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Trove] Postgresql Anyone working on DBaaS?
Thanks Denis, will do! On 10/07/14 19:23, Denis Makogon wrote: Hello Mark. There are several patches for Postgresql are hanging at the review queue. Here's useful link: https://blueprints.launchpad.net/trove/+spec/postgresql-support Contact point: Kevin Conway (you can ping him in IRC and ask if he needs any help) Best regards, Denis Makogon On Thu, Jul 10, 2014 at 9:24 AM, Mark Kirkwood mailto:mark.kirkw...@catalyst.net.nz>> wrote: Where I work we make use of Postgresql for most of our database needs. It would be nice to be able to offer a Postgresql flavor within the Trove framework. Is anyone working on adding it in? If noone else is, then I might look at doing it, if there are folks working on it - let me know if I can help with any part thereof. Regards Mark _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org <mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Trove] Postgresql Anyone working on DBaaS?
Where I work we make use of Postgresql for most of our database needs. It would be nice to be able to offer a Postgresql flavor within the Trove framework. Is anyone working on adding it in? If noone else is, then I might look at doing it, if there are folks working on it - let me know if I can help with any part thereof. Regards Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] [Trove] Trove instance got stuck in BUILD state
On 08/07/14 17:08, Denis Makogon wrote: Mark, there are also no documentation about service tuning(no description of service related options, sample configs in Trove repo is not enough). So, I think we should extend your list of significant things to document. ...and in case it might be helpful: here's my notes for installing openstack/trove on Ubuntu 14.04 using the Ubuntu packages, and debugging it (ahem). Some of the issues (virtio on bridges) are caused by it being all on one node, but hopefully it is generally useful (I cover how to build the guest image and get backups etc going). I make no claim for it being the best/only way to force the beast into being :-) but I think it works (for the logic devotees - sufficient not maybe not necessary)! Regards Mark README-OPENSTACK.gz Description: application/gzip ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] [Trove] Trove instance got stuck in BUILD state
On 08/07/14 17:08, Denis Makogon wrote: Mark, there are also no documentation about service tuning(no description of service related options, sample configs in Trove repo is not enough). So, I think we should extend your list of significant things to document. Right - I guess most of the tuning/config parameters could be better documented too (I do recall seeing this mentioned for one of the Trove meetings). One other thing I recall is: - mysql security install/setup in guest (mysql root password). I had to struggle through all of these - and it took a lot of time, because essentially the only viable way to debug each issue was: - check in an equivalent devstack build - read devstack setup code or (if the issue was present in devstack as well) - read the trove code and insert debug logging as appropriate ...and while this was a very interesting exercise, it was not a fast one! Cheers Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] [Trove] Trove instance got stuck in BUILD state
On 08/07/14 00:40, Amrith Kumar wrote: I think it is totally ludicrous (and to all the technical writers who work on OpenStack, downright offensive) to say the “docs are useless”. Not only have I been able to install and successfully operate a OpenStack installation by (largely) following the documentation, but “trove-integration” and “redstack” are useful for developers but I would highly doubt that a production deployment of Trove would use ‘redstack’. Syed, maybe you need to download a guest image for Trove, or maybe there is something else amiss with your setup. Happy to catch up with you on IRC and help you with that. Optionally, email me and I’ll give you a hand. It is a bit harsh, to be sure. However critical areas are light/thin or not covered at all - and this is bound to generate a bit of frustration for folk wanting to use this feature. In particular: - guest image preparation - guest file injection (/etc/guest_info) nova interaction - dns requirements for guest image (self hostname resolv) - swift backup config authorization - api_extensions_path setting and how critical that is There are probably more that I have forgotten (repressed perhaps...)! Regards Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] neutron config not working
On 25/06/14 10:59, Rob Crittenden wrote: Before I get punted onto the operators list, I post this here because this is the default config and I'd expect the defaults to just work. Running devstack inside a VM with a single NIC configured and this in localrc: disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron Q_USE_DEBUG_COMMAND=True Results in a successful install but no DHCP address assigned to hosts I launch and other oddities like no CIDR in nova net-list output. Is this still the default way to set things up for single node? It is according to https://wiki.openstack.org/wiki/NeutronDevstack That does look ok: I have an essentially equivalent local.conf: ... ENABLED_SERVICES+=,-n-net ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,tempest I don't have 'neutron' specifically enabled... not sure if/why that might make any difference tho. However instance launching and ip address assignment seem to work ok. However I *have* seen the issue of instances not getting ip addresses in single host setups, and it is often due to use of virt io with bridges (with is the default I think). Try: nova.conf: ... libvirt_use_virtio_for_bridges=False Regards Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Working in devstack?
Thanks...yes, I'd not realized that running ./stack.sh again would unpatch my murano-api/setup.sh! Rerunning the db setup as you suggested gives me the tables. I didn't do it in as tidy a manner as yours however :-) $ export OS_USERNAME=admin $ export OS_PASSWORD=swordfish $ export OS_TENANT_NAME=demo $ export OS_AUTH_URL=http://host:5000/v2.0/ $ murano-manage --config-file /etc/murano/murano-api.conf db-sync Cheers Mark On 16/04/14 11:23, Georgy Okrokvertskhov wrote: Hi Mark, Thank you for a detailed report. As I know Murano team is working on fixing devstack scripts. As for DB setup it should be done by a command: tox -evenv -- murano-manage --config-file etc/murano/murano-api.conf db-sync It works in my testing environment. Thanks Georgy On Tue, Apr 15, 2014 at 4:11 PM, Mark Kirkwood < mark.kirkw...@catalyst.net.nz> wrote: Hi all, There is some interest here in making use of Murano for Samba ADDC a service...so we've been (attempting) to get it up and running in devstack. In the process I've managed to get myself confused about the correct instructions for doing this: - the docs suggest http://murano-docs.github.io/latest/getting-started/ content/ch04s03.html - the project provides murano-deployment/devstack-scripts/README.rst) ...which are markedly different approaches (it *looks* like the project README.rst is out of date, as it the scripts it runs try to get heat-horizon from repos that do not exist anymore). If this approach is not longer workable, we should really remove (or correct these instructions). So following http://murano-docs.github.io/latest/getting-started/ content/ch04s03.html inside a Ubuntu 12.04 VM I stumbled into a few bugs: - several missing deps in various murano-*/requirements.txt (see attached) - typo in /murano-api/setup.sh (db_sync vs db-sync) Fixing these seems to get most things going (some tabs crash with missing tables, so I need to dig a bit more to find where they get created...). Any pointers/etc would be much appreciated! Cheers Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Murano] Working in devstack?
Hi all, There is some interest here in making use of Murano for Samba ADDC a service...so we've been (attempting) to get it up and running in devstack. In the process I've managed to get myself confused about the correct instructions for doing this: - the docs suggest http://murano-docs.github.io/latest/getting-started/content/ch04s03.html - the project provides murano-deployment/devstack-scripts/README.rst) ...which are markedly different approaches (it *looks* like the project README.rst is out of date, as it the scripts it runs try to get heat-horizon from repos that do not exist anymore). If this approach is not longer workable, we should really remove (or correct these instructions). So following http://murano-docs.github.io/latest/getting-started/content/ch04s03.html inside a Ubuntu 12.04 VM I stumbled into a few bugs: - several missing deps in various murano-*/requirements.txt (see attached) - typo in /murano-api/setup.sh (db_sync vs db-sync) Fixing these seems to get most things going (some tabs crash with missing tables, so I need to dig a bit more to find where they get created...). Any pointers/etc would be much appreciated! Cheers Mark Getting Murano to run in devstack = $ git clone https://github.com/openstack-dev/devstack.git $ cd devstack $ cat local.conf [[local|localrc]] ADMIN_PASSWORD=swordfish MYSQL_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD SERVICE_TOKEN=servicetoken $ ./stack.sh $ ./unstack.sh $ sudo apt-get purge apache2# make sure default site does not linger $ mv files/images ~ # save images! $ cd $ git clone https://github.com/stackforge/murano-deployment.git $ cp -r murano-deployment/devstack-integration/* devstack/ $ cd devstack $ cp single-node.local.conf local.conf $ vi local.conf # set HOST_IP $ ./stack.sh Bugs 1/ Several missing pip requirements: This necessitates re-running ./stack.sh several times and correcting the errors below one by one (sigh, there is hopefully a better way) $ tail /opt/stack/murano-api/requirements.txt yaql $ tail /opt/stack/murano-conductor/requirements.txt jsonpath celery $ tail /opt/stack/murano-repository/test-requirements.txt flask-testing $ tail /opt/stack/murano-dashboard/requirements.txt bunch 2/ Missing directory $ mkdir /etc/tgt 3/ Missing tables No tables in murano db's (error is install script) $ vi /opt/stack/murano-api/setup.sh # db_sync -> db-sync ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] can someone help me : cannot create VM via libvirt+xen
On 01/04/14 19:49, Tian, Shuangtai wrote: Hi, Though the libvirt+ xen is in the group C to support, I am still interested in a try to have look at the case. When I used the last xen and libvirt code from the community. I cannot boot a VM, the error log is below. Is anyone once encountered such situation? Can someone give me some suggestions, will be greatly appreciated. BTW, Will the Icehouse realy deprecate the group C? Thanks! Config change : libvirt_type=xen libvirt_disk_prefix=xvd use_cow_images=False Openstack log: 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] Traceback (most recent call last): 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/compute/manager.py", line 1043, in _build_instance 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] set_access_ip=set_access_ip) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/compute/manager.py", line 1426, in _spawn 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] LOG.exception(_('Instance failed to spawn'), instance=instance) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/compute/manager.py", line 1423, in _spawn 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] block_device_info) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2091, in spawn 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] block_device_info, context=context) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3249, in _create_domain_and_network 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] domain = self._create_domain(xml, instance=instance, power_on=power_on) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3192, in _create_domain 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] domain.XMLDesc(0)) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3187, in _create_domain 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] domain.createWithFlags(launch_flags) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] result = proxy_call(self._autowrap, f, *args, **kwargs) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] rv = execute(f,*args,**kwargs) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] rv = meth(*args,**kwargs) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 896, in createWithFlags 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2014-04-01 13:58:38.899 TRACE nova.compute.manager [instance: fa7f57c2-fb18-4dd3-90b0-521e6b4a25bb] libvirtError: internal error: libxenlight failed to create new domain 'instance-000a' Libvirt log: libxl: debug: libxl_create.c:1356:do_domain_create: ao 0x7f4a380020c0: create: how=(nil) callback=(nil) poller=0x7f4a38000ac0 libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=sda spec.backend=tap libxl: debug: libxl_device.c:210:disk_try_backend: Disk vdev=sda, backend tap unsuitable because blktap not available libxl: error: libxl_device.c:289:libxl__device_disk_set_backend: no suitable backend for disk sda libxl: debug: libxl_e
Re: [openstack-dev] [Openstack] [TROVE] Manual Installation Again
I've outlined two methods here, both assume you are using devstack, so would need some editorializing for an actual deployment (userids, paths etc): 1/ diskimage builder + triplo This essentially copies how trove-integration does it. 2/ Do it yourself install of required software in base image This was my hope for a simpler method that needed fewer extra tools - it ended up being about a similar level of complexity I fear. One difference between the two methods: 1/ installs mysql but rsyncs the trove software on 1st boot. 2/ installs mysql and trove in the image (so it starts up quicker). I guess an addition 3/ is possible where we install mysql on 1st boot and also the python deps + rsync trove software then too. This might make more sense when we have official pkgs for trove that we can just apt-get (and will pull in the right deps at that point). I'm happy to add this to the review, but the text is quite bulky and maybe we need to decide which method is best (at the moment anyway)! Cheers Mark On 17/03/14 22:37, tattabbum wrote: Hi Mark, have you some news about the right steps (Prepare Image) that are needed perform in order to launch a trove instance with a correctly configured trove-guest agent? As you, I have read all the trove-integration/script/redstack script to find the right commands to perform a manual trove installatio, and I have realized the gerrit review https://review.openstack.org/#/c/78608/. I would be happy if you could contribute to the review, particularly for the "Prepare Image" steps. Method 1 - Building our own vm via diskimage builder and tripleo Generate keys: $ ssh-keygen [no passphrase] $ cat .ssh/id_rsa.pub >> .ssh/authorized_keys Get custom tripleo elements for various db builds; $ git clone https://github.com/openstack/trove-integration.git $ cp -rp trove-integration/scripts/files ~/scripts $ rm -rf trove-integration $ vi scripts/files/elements/fedora-guest/extra-data.d/20-guest-upstart $ vi scripts/files/elements/ubuntu-guest/extra-data.d/20-guest-upstart [ s/REDSTACK_SCRIPTS/CUSTOM_SCRIPTS ] Check out diskimage-builder and tripleo-image-elements, so after ./stack.sh has run do: $ cd /opt/stack $ git clone git://git.openstack.org/openstack/diskimage-builder.git $ git clone git://git.openstack.org/openstack/tripleo-image-elements.git $ export CUSTOM_SCRIPTS=/home/stack/scripts $ export ELEMENTS_PATH=$CUSTOM_SCRIPTS/files/elements:/opt/stack/tripleo-image-elements/elements $ export HOMEDIR=/home/stack $ export SSH_DIR=${HOMEDIR}/.ssh $ export GUEST_USERNAME=stack $ export HOST_USERNAME=stack $ export NETWORK_GATEWAY=10.0.0.1 $ export SERVICE_TYPE=mysql $ export GUEST_LOGDIR=/tmp $ export ESCAPED_GUEST_LOGDIR=`echo $GUEST_LOGDIR | sed 's/\//\//g'` $ export RELEASE=precise $ export DIB_RELEASE=$RELEASE $ export DIB_CLOUD_IMAGES=cloud-images.ubuntu.com $ export PATH_TROVE=/opt/stack/trove $ export ESCAPED_PATH_TROVE=`echo $PATH_TROVE | sed 's/\//\//g'` $ /opt/stack/diskimage-builder/bin/disk-image-create \ -a amd64 -o ubuntu_mysql \ ubuntu vm heat-cfntools ubuntu-guest ubuntu-mysql Method 2 - building own vm with minimal other tools --- Generate keys: $ ssh-keygen [no passphrase] $ cat .ssh/id_rsa.pub >> .ssh/authorized_keys Get the ubuntu cloud image and set it up for easy access: $ sudo apt-get install cloud-utils genisoimage $ cd images $ wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img $ cat > my-user-data <:/home/stack/.ssh\* home/stack/.ssh (guest) $ sudo -u stack rsync -a :/opt/stack/trove /home/stack (guest) $ sudo apt-get install libmysqlclient18 mysql-server-5.5 mysql-client-5.5 (guest) $ sudo apt-get install python-dev libxml2-dev libxslt1-dev python-setuptools \ python-pip python-sqlalchemy python-lxml python-routes python-eventlet \ python-webob python-pastedeploy python-paste python-netaddr \ python-httplib2 python-iso8601 python-pexpect python-mysqldb python-migrate (guest) $ sudo pip install pip\>=1.4 (guest) $ sudo pip install http://tarballs.openstack.org/oslo.config/oslo.config-1.2.0a3.tar.gz\#egg=oslo.config-1.2.0a3 (guest) $ sudo pip install extras python-novaclient python-swiftclient python-cinderclient \ 'kombu>2.4.7' six babel python-heatclient passlib jinja2 (guest) $ sudo vi /etc/sudoers.d/60_trove_guest (guest) $ cat /etc/sudoers.d/60_trove_guest stack ALL=(ALL) NOPASSWD:ALL (guest) $ sudo chmod 440 /etc/sudoers.d/60_trove_guest (guest) $ sudo vi /etc/init/trove-guest.conf (guest) $ cat /etc/init/trove-guest.conf description "Trove Guest" author "Auto-Gen" start on (filesystem and net-device-up IFACE!=lo) stop on runlevel [016] chdir /var/run pre-start script mkdir -p /var/run/trove chown stack:root /var/run/trove/ mkdir -p /var/lock/trove chown sta
Re: [openstack-dev] [Openstack] [TROVE] Manual Installation Again
On 07/03/14 18:03, Mark Kirkwood wrote: The wrong concerns the action given to trove-manage in the "Prepare Database" section: $ trove-manage --config-file= image_update mysql `nova --os-username trove --os-password trove --os-tenant-name trove --os-auth-url http://:5000/v2.0 image-list | awk '/trove-image/ {print $2}'` This should probably be: $ trove-manage --config-file= datastore_version_update mysql mysql-5.5 mysql `nova --os-username trove --os-password trove --os-tenant-name trove --os-auth-url http://:5000/v2.0 image-list | awk '/trove-image/ {print $2}'` 1 ...which is a bit of a mouthful - might be better to break it into 2 steps. ...and I got it wrong too - forgot the package arg, sorry: $ trove-manage --config-file= datastore_version_update mysql mysql-5.5 mysql `nova --os-username trove --os-password trove --os-tenant-name trove --os-auth-url http://:5000/v2.0 image-list | awk '/trove-image/ {print $2}'` mysql-server-5.5 1 Especially in the light of the above I think a less confusing presentation would be: $ nova --os-username trove --os-password trove --os-tenant-name trove --os-auth-url http://:5000/v2.0 image-list | awk '/trove-image/ {print $2}' $ trove-manage --config-file= datastore_version_update mysql mysql-5.5 mysql mysql-server-5.5 1 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Openstack] [TROVE] Manual Installation Again
I've been looking at setting up Trove manually, and of course the first document I stumbled on was: http://docs.openstack.org/developer/trove/dev/manual_install.html Now, while this proved to be very handy, there are some points where it is wrong, and others where it is errm...lean. So in the spirit of trying to improve things here I go. The wrong concerns the action given to trove-manage in the "Prepare Database" section: $ trove-manage --config-file= image_update mysql `nova --os-username trove --os-password trove --os-tenant-name trove --os-auth-url http://:5000/v2.0 image-list | awk '/trove-image/ {print $2}'` This should probably be: $ trove-manage --config-file= datastore_version_update mysql mysql-5.5 mysql `nova --os-username trove --os-password trove --os-tenant-name trove --os-auth-url http://:5000/v2.0 image-list | awk '/trove-image/ {print $2}'` 1 ...which is a bit of a mouthful - might be better to break it into 2 steps. The lean area concerns the stuff in "Prepare Image". It seems to me that more needs to be done than simply converting to qcow2. After spending a while reading stuff in trove-integration/scripts repo I suspect that something like following is needed: 1/ setup relevant os user (e.g trove or stack) for what follows 2/ install mysql 5.5 in the image (or arrange it to be installed on 1st boot) 3/ setup keys so guest can rsync the trove client software (or install it in the image to avoid the need) 4/ configure the trove guest agent service to start (otherwise db instance stays stuck in 'BUILD' state forever) I note that the trove-integration repo uses diskimage-builder and triplo to do all these mods to the initial base image. Now I understand that some of this area is gonna be in flux (e.g use of first-boot.d in the tripleo elements), but some mention of what customizations to the base image are needed would be most excellent. regards Mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev