Re: [openstack-dev] [tripleo][manila] Ganesha deployment
On Tue, Apr 11, 2017 at 9:45 PM, Ben Nemec wrote: > > > On 04/11/2017 02:00 PM, Giulio Fidente wrote: >> >> On Tue, 2017-04-11 at 16:50 +0200, Jan Provaznik wrote: >>> >>> On Mon, Apr 10, 2017 at 6:55 PM, Ben Nemec >>> wrote: On 04/10/2017 03:22 AM, Jan Provaznik wrote: Well, on second thought it might be possible to make the Storage network only routable within overcloud Neutron by adding a bridge mapping for the Storage network and having the admin configure a shared Neutron network for it. That would be somewhat more secure since it wouldn't require the Storage network to be routable by the world. I also think this would work today in TripleO with no changes. >>> >>> This sounds interesting, I was searching for more info how bridge >>> mapping should be done in this case and how specific setup steps >>> should look like, but the process is still not clear to me, I would >>> be >>> grateful for more details/guidance with this. >> >> >> I think this will be represented in neutron as a provider network, >> which has to be created by the overcloud admin, after the overcloud >> deployment is finished >> >> While based on Kilo, this was one of the best docs I could find and it >> includes config examples [1] >> >> It assumes that the operator created a bridge mapping for it when >> deploying the overcloud >> I think the answer here will be the same as for vanilla Ceph. You need to make the network routable to instances, and you'd have the same options as I discussed above. >>> >>> Yes, it seems that using the mapping to provider network would solve >>> the existing problem when using ceph directly and when using ganesha >>> servers in future (it would be just matter of to which network is >>> exposed). >> >> >> +1 >> >> regarding the composability questions, I think this represents a >> "composable HA" scenario where we want to manage a remote service with >> pacemaker using pacemaker-remote >> >> yet at this stage I think we want to add support for new services by >> running them in containers first (only?) and pacemaker+containers is >> still a work in progress so there aren't easy answers >> >> containers will have access to the host networks though, so the case >> for a provider network in the overcloud remains valid >> >> 1. https://docs.openstack.org/kilo/networking-guide/scenario_provider_o >> vs.html >> > > I think there are three major pieces that would need to be in place to have > a storage provider network: > > 1) The storage network must be bridged in the net-iso templates. I don't > think our default net-iso templates do that, but there are examples of > bridged networks in them: > https://github.com/openstack/tripleo-heat-templates/blob/master/network/config/multiple-nics/compute.yaml#L121 > For the rest of the steps I will assume the bridge was named br-storage. > > 2) Specify a bridge mapping when deploying the overcloud. The environment > file would look something like this (datacentre is the default value, so I'm > including it too): > > parameter_defaults: > NeutronBridgeMappings: 'datacentre:br-ex,storage:br-storage' > > 3) Create a provider network after deployment as described in the link > Giulio provided. The specific command will depend on the network > architecture, but it would need to include "--provider:physical_network > storage". > > We might need to add the ability to do 3 as part of the deployment, > depending what is needed for the Ganesha deployment itself. We've typically > avoided creating network resources like this in the deployment because of > the huge variations in what people want, but this might be an exceptional > case since the network will be a required part of the overcloud. > > Thank you both for your help, based on steps suggested above I was able to mount ceph volume in user instance when Overcloud was deployed with net-iso with net-single-nic-with-vlans (which is the easiest one I can deploy in my virtualenv). For net-single-nic-with-vlans I skipped creation of additional bridge (since single bridge is used for all networks in this case) and deployed OC as usually, then I configured networking: neutron net-create storage --shared --provider:physical_network datacentre --provider:network_type vlan --provider:segmentation_id 30 neutron subnet-create --name storage-subnet --allocation-pool start=172.16.1.100,end=172.16.1.120--enable-dhcp storage 172.16.1.0/24 and created user instance with tenant and storage network: | f7d4e619-c8f5-4de3-a4c3-4120eea818d1 | Server1 | ACTIVE | - | Running | default-net=192.168.2.107, 192.168.24.100; storage=172.16.1.110 | The obstacle I'm hitting though is that the second interface with storage network doesn't come up automatically on instance boot: 3: eth1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether fa:16:3e:71:90:df brd ff:ff:ff:ff:ff:ff from cloud-init log: ci-info: ++
Re: [openstack-dev] [tripleo][manila] Ganesha deployment
On 04/11/2017 02:00 PM, Giulio Fidente wrote: On Tue, 2017-04-11 at 16:50 +0200, Jan Provaznik wrote: On Mon, Apr 10, 2017 at 6:55 PM, Ben Nemec wrote: On 04/10/2017 03:22 AM, Jan Provaznik wrote: Well, on second thought it might be possible to make the Storage network only routable within overcloud Neutron by adding a bridge mapping for the Storage network and having the admin configure a shared Neutron network for it. That would be somewhat more secure since it wouldn't require the Storage network to be routable by the world. I also think this would work today in TripleO with no changes. This sounds interesting, I was searching for more info how bridge mapping should be done in this case and how specific setup steps should look like, but the process is still not clear to me, I would be grateful for more details/guidance with this. I think this will be represented in neutron as a provider network, which has to be created by the overcloud admin, after the overcloud deployment is finished While based on Kilo, this was one of the best docs I could find and it includes config examples [1] It assumes that the operator created a bridge mapping for it when deploying the overcloud I think the answer here will be the same as for vanilla Ceph. You need to make the network routable to instances, and you'd have the same options as I discussed above. Yes, it seems that using the mapping to provider network would solve the existing problem when using ceph directly and when using ganesha servers in future (it would be just matter of to which network is exposed). +1 regarding the composability questions, I think this represents a "composable HA" scenario where we want to manage a remote service with pacemaker using pacemaker-remote yet at this stage I think we want to add support for new services by running them in containers first (only?) and pacemaker+containers is still a work in progress so there aren't easy answers containers will have access to the host networks though, so the case for a provider network in the overcloud remains valid 1. https://docs.openstack.org/kilo/networking-guide/scenario_provider_o vs.html I think there are three major pieces that would need to be in place to have a storage provider network: 1) The storage network must be bridged in the net-iso templates. I don't think our default net-iso templates do that, but there are examples of bridged networks in them: https://github.com/openstack/tripleo-heat-templates/blob/master/network/config/multiple-nics/compute.yaml#L121 For the rest of the steps I will assume the bridge was named br-storage. 2) Specify a bridge mapping when deploying the overcloud. The environment file would look something like this (datacentre is the default value, so I'm including it too): parameter_defaults: NeutronBridgeMappings: 'datacentre:br-ex,storage:br-storage' 3) Create a provider network after deployment as described in the link Giulio provided. The specific command will depend on the network architecture, but it would need to include "--provider:physical_network storage". We might need to add the ability to do 3 as part of the deployment, depending what is needed for the Ganesha deployment itself. We've typically avoided creating network resources like this in the deployment because of the huge variations in what people want, but this might be an exceptional case since the network will be a required part of the overcloud. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo][manila] Ganesha deployment
On Tue, 2017-04-11 at 16:50 +0200, Jan Provaznik wrote: > On Mon, Apr 10, 2017 at 6:55 PM, Ben Nemec > wrote: > > On 04/10/2017 03:22 AM, Jan Provaznik wrote: > > Well, on second thought it might be possible to make the Storage > > network > > only routable within overcloud Neutron by adding a bridge mapping > > for the > > Storage network and having the admin configure a shared Neutron > > network for > > it. That would be somewhat more secure since it wouldn't require > > the > > Storage network to be routable by the world. I also think this > > would work > > today in TripleO with no changes. > > > > This sounds interesting, I was searching for more info how bridge > mapping should be done in this case and how specific setup steps > should look like, but the process is still not clear to me, I would > be > grateful for more details/guidance with this. I think this will be represented in neutron as a provider network, which has to be created by the overcloud admin, after the overcloud deployment is finished While based on Kilo, this was one of the best docs I could find and it includes config examples [1] It assumes that the operator created a bridge mapping for it when deploying the overcloud > > I think the answer here will be the same as for vanilla Ceph. You > > need to > > make the network routable to instances, and you'd have the same > > options as I > > discussed above. > > > > Yes, it seems that using the mapping to provider network would solve > the existing problem when using ceph directly and when using ganesha > servers in future (it would be just matter of to which network is > exposed). +1 regarding the composability questions, I think this represents a "composable HA" scenario where we want to manage a remote service with pacemaker using pacemaker-remote yet at this stage I think we want to add support for new services by running them in containers first (only?) and pacemaker+containers is still a work in progress so there aren't easy answers containers will have access to the host networks though, so the case for a provider network in the overcloud remains valid 1. https://docs.openstack.org/kilo/networking-guide/scenario_provider_o vs.html -- Giulio Fidente GPG KEY: 08D733BA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo][manila] Ganesha deployment
On Mon, Apr 10, 2017 at 6:55 PM, Ben Nemec wrote: > I'm not really an expert on composable roles so I'll leave that to someone > else, but see my thoughts inline on the networking aspect. > > On 04/10/2017 03:22 AM, Jan Provaznik wrote: >> >> 2) define a new VIP (for IP failover) and 2 networks for NfsStorage role: >> a) a frontend network between users and ganesha servers (e.g. >> NfsNetwork name), used by tenants to mount nfs shares - this network >> should be accessible from user instances. > > > Adding a new network is non-trivial today, so I think we want to avoid that > if possible. Is there a reason the Storage network couldn't be used for > this? That is already present on compute nodes by default so it would be > available to user instances, and it seems like the intended use of the > Storage network matches this use case. In a Ceph deployment today that's > the network which exposes data to user instances. > Access to the ceph public network (StorageNetwork) is a big privilege (from discussing this with ceph team), bigger than accessing only ganesha nfs servers, so StorageNetwork should be exposed only when really necessary. >> b) a backend network between ganesha servers ans ceph cluster - >> this could just map to the existing StorageNetwork I think. > > > This actually sounds like a better fit for StorageMgmt to me. It's > non-user-facing storage communication, which is what StorageMgmt is used for > in the vanilla Ceph case. > If StorageMgmt is used for replication and internal ceph nodes communication, I wonder if it's not too permissive access? Ganesha servers should need access ti ceph public network only. >> What i'm not sure at all is how network definition should look like. >> There are following Overcloud deployment options: >> 1) no network isolation is used - then both direct ceph mount and >> mount through ganesha should work because StorageNetwork and >> NfsNetwork are accessible from user instances (there is no restriction >> in accessing other networks it seems). > > > There are no other networks without network-isolation. Everything runs over > the provisioning network. The network-isolation templates should mostly > handle this for you though. > >> 2) network isolation is used: >> a) ceph is used directly - user instances need access to the ceph >> public network (which is StorageNetwork in Overcloud) - how should I >> enable access to this network? I filled a bug for this deployment >> variant here [3] > > > So does this mean that the current manila implementation is completely > broken in network-isolation? If so, that's rather concerning. > This affects deployments of manila with internal (=deployed by TripleO) ceph backend. > If I'm understanding correctly, it sounds like what needs to happen is to > make the Storage network routable so it's available from user instances. > That's not actually something TripleO can do, it's an underlying > infrastructure thing. I'm not sure what the security implications of it are > either. > > Well, on second thought it might be possible to make the Storage network > only routable within overcloud Neutron by adding a bridge mapping for the > Storage network and having the admin configure a shared Neutron network for > it. That would be somewhat more secure since it wouldn't require the > Storage network to be routable by the world. I also think this would work > today in TripleO with no changes. > This sounds interesting, I was searching for more info how bridge mapping should be done in this case and how specific setup steps should look like, but the process is still not clear to me, I would be grateful for more details/guidance with this. > Alternatively I guess you could use ServiceNetMap to move the public Ceph > traffic to the public network, which has to be routable. That seems like it > might have a detrimental effect on the public network's capacity, but it > might be okay in some instances. > I would rather avoid this option (both because of network traffic and because of exposing ceph public network to everybody). >> b) ceph is used through ganesha - user instances need access to >> ganesha servers (NfsNetwork in previous paragraph) - how should I >> enable access to this network? > > > I think the answer here will be the same as for vanilla Ceph. You need to > make the network routable to instances, and you'd have the same options as I > discussed above. > Yes, it seems that using the mapping to provider network would solve the existing problem when using ceph directly and when using ganesha servers in future (it would be just matter of to which network is exposed). >> >> The ultimate (and future) plan is to deploy ganesha-nfs in VMs (which >> will run in Overcloud, probably managed by manila ceph driver), in >> this deployment mode a user should have access to ganesha servers and >> only ganesha server VMs should have access to ceph public network. >> Ganesha VMs would run in a separate
Re: [openstack-dev] [tripleo][manila] Ganesha deployment
I'm not really an expert on composable roles so I'll leave that to someone else, but see my thoughts inline on the networking aspect. On 04/10/2017 03:22 AM, Jan Provaznik wrote: 2) define a new VIP (for IP failover) and 2 networks for NfsStorage role: a) a frontend network between users and ganesha servers (e.g. NfsNetwork name), used by tenants to mount nfs shares - this network should be accessible from user instances. Adding a new network is non-trivial today, so I think we want to avoid that if possible. Is there a reason the Storage network couldn't be used for this? That is already present on compute nodes by default so it would be available to user instances, and it seems like the intended use of the Storage network matches this use case. In a Ceph deployment today that's the network which exposes data to user instances. b) a backend network between ganesha servers ans ceph cluster - this could just map to the existing StorageNetwork I think. This actually sounds like a better fit for StorageMgmt to me. It's non-user-facing storage communication, which is what StorageMgmt is used for in the vanilla Ceph case. What i'm not sure at all is how network definition should look like. There are following Overcloud deployment options: 1) no network isolation is used - then both direct ceph mount and mount through ganesha should work because StorageNetwork and NfsNetwork are accessible from user instances (there is no restriction in accessing other networks it seems). There are no other networks without network-isolation. Everything runs over the provisioning network. The network-isolation templates should mostly handle this for you though. 2) network isolation is used: a) ceph is used directly - user instances need access to the ceph public network (which is StorageNetwork in Overcloud) - how should I enable access to this network? I filled a bug for this deployment variant here [3] So does this mean that the current manila implementation is completely broken in network-isolation? If so, that's rather concerning. If I'm understanding correctly, it sounds like what needs to happen is to make the Storage network routable so it's available from user instances. That's not actually something TripleO can do, it's an underlying infrastructure thing. I'm not sure what the security implications of it are either. Well, on second thought it might be possible to make the Storage network only routable within overcloud Neutron by adding a bridge mapping for the Storage network and having the admin configure a shared Neutron network for it. That would be somewhat more secure since it wouldn't require the Storage network to be routable by the world. I also think this would work today in TripleO with no changes. Alternatively I guess you could use ServiceNetMap to move the public Ceph traffic to the public network, which has to be routable. That seems like it might have a detrimental effect on the public network's capacity, but it might be okay in some instances. b) ceph is used through ganesha - user instances need access to ganesha servers (NfsNetwork in previous paragraph) - how should I enable access to this network? I think the answer here will be the same as for vanilla Ceph. You need to make the network routable to instances, and you'd have the same options as I discussed above. The ultimate (and future) plan is to deploy ganesha-nfs in VMs (which will run in Overcloud, probably managed by manila ceph driver), in this deployment mode a user should have access to ganesha servers and only ganesha server VMs should have access to ceph public network. Ganesha VMs would run in a separate tenant so I wonder if it's possible to manage access to the ceph public network (StorageNetwork in Overcloud) on per-tenant level? This would suggest that the bridged Storage network approach is the best. In that case access to the ceph public network is controlled by the overcloud Neutron, so you would just need to only give access to it to the tenant running the Ganesha VMs. User VMs would only get access to a separate shared network providing access to the public Ganesha API, and the Ganesha VMs would straddle both networks. Any thoughts and hints? Thanks, Jan [1] https://github.com/nfs-ganesha/nfs-ganesha/wiki [2] https://github.com/ceph/ceph-ansible/tree/master/roles/ceph-nfs [3] https://bugs.launchpad.net/tripleo/+bug/1680749 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev