[Gluster-devel] An Update on GlusterD-2.0
The first preview/dev release of GlusterD-2.0 is available now. A prebuilt binary is available for download from the release-page[1]. This is just a preview of what has been happening in GD2, to give users a taste of how GD2 is evolving. GD2 can now form a cluster, list peers, create/delete,(psuedo) start/stop and list volumes. Most of these will undergo changes and refined as we progress. More information on how to test this release can be found on the release page. We'll providing periodic (hopefully fortnightly) updates on the changes happening in GD2 from now on. We'll also be providing periodic dev builds for people to test. Currently builds are only available for Linux on amd64. Vagrant and docker releases are planned to make it easier to test GD2. Thanks, Kaushal [1] https://github.com/gluster/glusterd2/releases/tag/v4.0dev-1 ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] An update on GlusterD-2.0
Kaushal, This is really cool stuff! I think Heketi will provide the IVC service Glusterd-2.0 requires. I also look forward to working with you and KP if you guys have the time (and anyone else who want to help, for that matter). Just letting you know, that at least until October, Heketi will be focused on providing a solution for Manila and Kubernetes as the priority. There may be other features that Glusterd-2.0 requires which may have to wait, but I think it would be great to participate and get used to how Heketi works. Currently I am trying to finalize the API, and I will be sending a follow up email requesting a review. Also, like Deepak pointed out, Heketi will need to deal with GlusterFS Certificates, somehow. That is still TBD. I look forward to working with you guys. - Luis - Original Message - From: "Kaushal M" To: "Gluster Devel" Sent: Wednesday, June 17, 2015 11:05:14 AM Subject: [Gluster-devel] An update on GlusterD-2.0 At the Gluster Summit, everyone agreed that GlusterD should be the first component to be targeted for GlusterFS-4.0. We had good discussions on what GlusterD lacks currently and what is required for GlusterD-2.0. KP and I had promised to send an update to mailing list summarizing the discussions, and this is it. Along with the summary, we'll also be discussing our plans for Manila integration and how we are planning to do it with GlusterD-2.0. ## Gluster Summit Summary In the summit, KP and I presented a talk titled *GLUSTERD NEXT GEN OR: HOW WE CAN STOP BEING A ROADBLOCK AND INSTEAD BECOME THE ROAD TO SCALABILITY*. The slides can be viewed at [1][1]. There is no video recording of it unfortunately. The summary of the presentation is below. ### Problems GlusterD, as it is currently, is not scalable which will prevent GlusterFS as whole from scaling. The scaling issues can be classified into, - Node scalability - Maintenance scalability - Integration scalability - Contribution scalability Node scalability This is the general scalability we all think about, scale in terms of node/machine/clients. GlusterD scalability is help back in this because of the store, transaction and synchronization mechanisms used in GlusterD. Maintenance scalability Maintenance scalability is to with the problems we as GlusterD maintainers faced. This is mainly related to the huge, monolithic code base of the current GlusterD, which makes splitting of maintenance and ownership tasks hard. Integration scalability Integration scalability can split into internal and external integration. Internal integration is the integration dealing with new features being added to GlusterFS. Every new feature being added needs to touch GlusterD or CLI in some way. This has generally been done with copy/paste coding which has added to the maintenence overhead. External integration is the integration of Gluster with other projects or other projects with Gluster. This is hurt due to the non-availability of a proper API for GlusterD operations. All interaction with GlusterD currently happens only via the CLI. And the output we provide is generally inconsistent to be programatically parsed. Contribution scalability Getting new contributors for GlusterD is hard. New contributors are put off because GlusterD is hard to understand, and because there isn't enough documentation. So GlusterD-2.0 will need to - be scalable to 1000s of nodes - have lower maintenance costs - enable better external and internal integrations - make it easier for newer contributors ### Design characteristics for GlusterD-2.0 For GlusterD-2.0 to satisfy the above listed requirements and solve the problems listed before, it should have the following characteristics. Centralized store This should help with our numbers scalability issues. GlusterD-2.0 will be built around a centralized store. This means, instead of the GlusterD volume and peer information being persisted on all nodes, it will be stored only on a subset of the nodes in a trusted storage pool. We are looking at solutions like etcd and consul, both of which provide a distributed key/value store (and some more useful features on top), to provide the centralized store. The transaction mechanism for Gluster operations will be built around this centralized store. Moving to an external store provider and a transaction framework built around it will reduce a lot of the complexity in GlusterD. Plugins This mainly for the maintainability and internal integration aspects. GlusterD-2.0 will have a plug-gable, modular design. We expect all the commands of GlusterD to be implemented as plugins. Certain other parts of GlusterD, including things like volgen, volume-set, rest api, etc. This will allow new features to be integrated in to GlusterD easily. The code for these plugins is expected to live with the feature, and not in GlusterD. Doing a plugin design requires the defining of well defined plug
Re: [Gluster-devel] An update on GlusterD-2.0
On Thu, Jun 25, 2015 at 1:22 PM, Kaushal M wrote: > > One other design characteristic I can think of is to have GlusterD-2.0 > > be able to make best use of Nodes based on the Node topology, not sure > > if this is covered by any other feature below ? > > > > IOW, admin should be able to provide a hierarchy of how nodes are > > provisioned > > (rack, site, zone, region, geo etc) and glusterd should use this info > while > > provisioning bricks for volume creation > > > > The intelligent volume creation support that I mention later is nearly > this, without the additional markers of rack, site etc. For an initial > implementation, all we wanted was a way list available space, without > any special markers, and use this list to create volumes. > > The design characteristics I wrote about, are mainly with respect to > how we want the internal frameworks for GlusterD-2.0 to be. These > frameworks will be the ones which will be used to implement other > feature requirements, like volume creation. > IMHO i feel this should be a design char as in "Node Topology Awareness" I know this can be implemented on top of glusterd, but having this as part of the design ensures we don't do something that won't be then extensible as needed for glusterd to be topology aware. > > > I have 2 things to say here : > > > > 1) Manila uses GlusterFS SSL auth mode, so is there any discussion > happening > > around adding support in GlusterD-2.0 for managing SSL certificates ? > > > > For eg: Other (eg Netapp) storage arrays do provide basic digital cert > mgmt > > support for server and client auth. > > > > I feel it would be good to have glusterd provide support to generate, > > install, manage > > self-signed and/or CA signed digital certs. > > > > This will not just help Manila usecase, but also help GlusterFS provide a > > complete > > solution for digital cert management which will aid the SSL auth support > > feature of GlusterFS > > > > In fact, this can be done in a modular way where in the default implm > will > > be that of GlusterD > > cert module, but optionally one can use openstack Barbican service too > as a > > cert mgmt service > > This one management feature we need add to out list. > > > > > 2) In the manila community there was some intense discussion on the > > definition of multi-tenancy > > when applied to storage and network level multi-tenancy was voted as > being > > very important in > > Manila cloud usecase. Do we have any thoughts on how GlusterD-2.0 / > > GlusterFS-4.0 can > > look at providing tenant separation at network layer ? > > This is a goal for GlusterFS-4.0 and should be discussed along with > that. Our main priority for GlusterD-2.0 is to build a good management > framework first and implement the minimum set of functionality > required to manage a cluster. This will be the base for building the > other features of GlusterFS-4.0, like network multi-tenancy. > GlusterFS-4.0 will have this from the perspetive of providing ability to serve shares in a private/segmented network. But i think GlusterD should have the ability to provision/manage the private network/segmentation and prvoide this network ID ( say ) to GlusterFS as part of volume creation flow. Other than that, you also mentioned as part of REST API that GlusterD should provide ability to query the cluster size, i guess we also need ability to query the volume size too (currently client needs to mount , do df -h and get the free/used size, this should be part of REST API) as i see that it will be helpful in Manila usecase thanx, deepak > > > > > thanx, > > deepak > > > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] An update on GlusterD-2.0
> One other design characteristic I can think of is to have GlusterD-2.0 > be able to make best use of Nodes based on the Node topology, not sure > if this is covered by any other feature below ? > > IOW, admin should be able to provide a hierarchy of how nodes are > provisioned > (rack, site, zone, region, geo etc) and glusterd should use this info while > provisioning bricks for volume creation > The intelligent volume creation support that I mention later is nearly this, without the additional markers of rack, site etc. For an initial implementation, all we wanted was a way list available space, without any special markers, and use this list to create volumes. The design characteristics I wrote about, are mainly with respect to how we want the internal frameworks for GlusterD-2.0 to be. These frameworks will be the ones which will be used to implement other feature requirements, like volume creation. > I have 2 things to say here : > > 1) Manila uses GlusterFS SSL auth mode, so is there any discussion happening > around adding support in GlusterD-2.0 for managing SSL certificates ? > > For eg: Other (eg Netapp) storage arrays do provide basic digital cert mgmt > support for server and client auth. > > I feel it would be good to have glusterd provide support to generate, > install, manage > self-signed and/or CA signed digital certs. > > This will not just help Manila usecase, but also help GlusterFS provide a > complete > solution for digital cert management which will aid the SSL auth support > feature of GlusterFS > > In fact, this can be done in a modular way where in the default implm will > be that of GlusterD > cert module, but optionally one can use openstack Barbican service too as a > cert mgmt service This one management feature we need add to out list. > > 2) In the manila community there was some intense discussion on the > definition of multi-tenancy > when applied to storage and network level multi-tenancy was voted as being > very important in > Manila cloud usecase. Do we have any thoughts on how GlusterD-2.0 / > GlusterFS-4.0 can > look at providing tenant separation at network layer ? This is a goal for GlusterFS-4.0 and should be discussed along with that. Our main priority for GlusterD-2.0 is to build a good management framework first and implement the minimum set of functionality required to manage a cluster. This will be the base for building the other features of GlusterFS-4.0, like network multi-tenancy. > > thanx, > deepak > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] An update on GlusterD-2.0
On Wed, Jun 17, 2015 at 8:35 PM, Kaushal M wrote: > At the Gluster Summit, everyone agreed that GlusterD should be the > first component to be targeted for GlusterFS-4.0. We had good > discussions on what GlusterD lacks currently and what is required for > GlusterD-2.0. KP and I had promised to send an update to mailing list > summarizing the discussions, and this is it. > > Along with the summary, we'll also be discussing our plans for Manila > integration and how we are planning to do it with GlusterD-2.0. > > ## Gluster Summit Summary > In the summit, KP and I presented a talk titled *GLUSTERD NEXT GEN > OR: HOW WE CAN STOP BEING A ROADBLOCK AND INSTEAD BECOME THE ROAD TO > SCALABILITY*. The slides can be viewed at [1][1]. There is no video > recording of it unfortunately. > > The summary of the presentation is below. > > ### Problems > GlusterD, as it is currently, is not scalable which will prevent > GlusterFS as whole from scaling. The scaling issues can be classified > into, > - Node scalability > - Maintenance scalability > - Integration scalability > - Contribution scalability > > Node scalability > This is the general scalability we all think about, scale in terms of > node/machine/clients. GlusterD scalability is help back in this > because of the store, transaction and synchronization mechanisms used > in GlusterD. > > Maintenance scalability > Maintenance scalability is to with the problems we as GlusterD > maintainers faced. This is mainly related to the huge, monolithic code > base of the current GlusterD, which makes splitting of maintenance and > ownership tasks hard. > > Integration scalability > Integration scalability can split into internal and external integration. > Internal integration is the integration dealing with new features > being added to GlusterFS. Every new feature being added needs to touch > GlusterD or CLI in some way. This has generally been done with > copy/paste coding which has added to the maintenence overhead. > External integration is the integration of Gluster with other projects > or other projects with Gluster. This is hurt due to the > non-availability of a proper API for GlusterD operations. All > interaction with GlusterD currently happens only via the CLI. And the > output we provide is generally inconsistent to be programatically > parsed. > > Contribution scalability > Getting new contributors for GlusterD is hard. New contributors are > put off because GlusterD is hard to understand, and because there > isn't enough documentation. > > > So GlusterD-2.0 will need to > - be scalable to 1000s of nodes > - have lower maintenance costs > - enable better external and internal integrations > - make it easier for newer contributors > > ### Design characteristics for GlusterD-2.0 > For GlusterD-2.0 to satisfy the above listed requirements and solve > the problems listed before, it should have the following > characteristics. > One other design characteristic I can think of is to have GlusterD-2.0 be able to make best use of Nodes based on the Node topology, not sure if this is covered by any other feature below ? IOW, admin should be able to provide a hierarchy of how nodes are provisioned (rack, site, zone, region, geo etc) and glusterd should use this info while provisioning bricks for volume creation > Centralized store > This should help with our numbers scalability issues. GlusterD-2.0 > will be built around a centralized store. This means, instead of the > GlusterD volume and peer information being persisted on all nodes, it > will be stored only on a subset of the nodes in a trusted storage > pool. > > We are looking at solutions like etcd and consul, both of which > provide a distributed key/value store (and some more useful features > on top), to provide the centralized store. The transaction mechanism > for Gluster operations will be built around this centralized store. > > Moving to an external store provider and a transaction framework built > around it will reduce a lot of the complexity in GlusterD. > > Plugins > This mainly for the maintainability and internal integration aspects. > GlusterD-2.0 will have a plug-gable, modular design. We expect all the > commands of GlusterD to be implemented as plugins. Certain other parts > of GlusterD, including things like volgen, volume-set, rest api, etc. > This will allow new features to be integrated in to GlusterD easily. > The code for these plugins is expected to live with the feature, and > not in GlusterD. > > Doing a plugin design requires the defining of well defined plugin > interfaces to not just plug into GlusterD-2.0, but also interact with > GlusterD well. I'll be posting a document on this to the mailing list > soon. > > The GlusterD maintainers/team will be doing the implementation of a > few core commands required including volume create, start, stop, > delete, expand, shrink and set/reset options. > > Rest API > To provide better a better method for
[Gluster-devel] An update on GlusterD-2.0
At the Gluster Summit, everyone agreed that GlusterD should be the first component to be targeted for GlusterFS-4.0. We had good discussions on what GlusterD lacks currently and what is required for GlusterD-2.0. KP and I had promised to send an update to mailing list summarizing the discussions, and this is it. Along with the summary, we'll also be discussing our plans for Manila integration and how we are planning to do it with GlusterD-2.0. ## Gluster Summit Summary In the summit, KP and I presented a talk titled *GLUSTERD NEXT GEN OR: HOW WE CAN STOP BEING A ROADBLOCK AND INSTEAD BECOME THE ROAD TO SCALABILITY*. The slides can be viewed at [1][1]. There is no video recording of it unfortunately. The summary of the presentation is below. ### Problems GlusterD, as it is currently, is not scalable which will prevent GlusterFS as whole from scaling. The scaling issues can be classified into, - Node scalability - Maintenance scalability - Integration scalability - Contribution scalability Node scalability This is the general scalability we all think about, scale in terms of node/machine/clients. GlusterD scalability is help back in this because of the store, transaction and synchronization mechanisms used in GlusterD. Maintenance scalability Maintenance scalability is to with the problems we as GlusterD maintainers faced. This is mainly related to the huge, monolithic code base of the current GlusterD, which makes splitting of maintenance and ownership tasks hard. Integration scalability Integration scalability can split into internal and external integration. Internal integration is the integration dealing with new features being added to GlusterFS. Every new feature being added needs to touch GlusterD or CLI in some way. This has generally been done with copy/paste coding which has added to the maintenence overhead. External integration is the integration of Gluster with other projects or other projects with Gluster. This is hurt due to the non-availability of a proper API for GlusterD operations. All interaction with GlusterD currently happens only via the CLI. And the output we provide is generally inconsistent to be programatically parsed. Contribution scalability Getting new contributors for GlusterD is hard. New contributors are put off because GlusterD is hard to understand, and because there isn't enough documentation. So GlusterD-2.0 will need to - be scalable to 1000s of nodes - have lower maintenance costs - enable better external and internal integrations - make it easier for newer contributors ### Design characteristics for GlusterD-2.0 For GlusterD-2.0 to satisfy the above listed requirements and solve the problems listed before, it should have the following characteristics. Centralized store This should help with our numbers scalability issues. GlusterD-2.0 will be built around a centralized store. This means, instead of the GlusterD volume and peer information being persisted on all nodes, it will be stored only on a subset of the nodes in a trusted storage pool. We are looking at solutions like etcd and consul, both of which provide a distributed key/value store (and some more useful features on top), to provide the centralized store. The transaction mechanism for Gluster operations will be built around this centralized store. Moving to an external store provider and a transaction framework built around it will reduce a lot of the complexity in GlusterD. Plugins This mainly for the maintainability and internal integration aspects. GlusterD-2.0 will have a plug-gable, modular design. We expect all the commands of GlusterD to be implemented as plugins. Certain other parts of GlusterD, including things like volgen, volume-set, rest api, etc. This will allow new features to be integrated in to GlusterD easily. The code for these plugins is expected to live with the feature, and not in GlusterD. Doing a plugin design requires the defining of well defined plugin interfaces to not just plug into GlusterD-2.0, but also interact with GlusterD well. I'll be posting a document on this to the mailing list soon. The GlusterD maintainers/team will be doing the implementation of a few core commands required including volume create, start, stop, delete, expand, shrink and set/reset options. Rest API To provide better a better method for external projects to hook on to. GlusterD-2.0 will provide a HTTP REST API, with JSON output and proper API versioning. We will be looking for inspiration from other storage solutions to define the APIs presented to the users and admins. Logging To help with debuggability, GlusterD-2.0 will provide better logs that should allow easier tracking of operations across machines. We are currently looking at the oVirt logging mechanism, which tags all logs of an operation with a unique id, for inspiration. Documentation We hope to do a much better job of documenting the technical details of GlusterD-2.0, starting wi
