Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec

[Pranith Kumar Karampuri]

On 10/29/2015 06:11 PM, Jeff Darcy wrote:

I want to understand if there is a possibility of exposing these as
different modules that we can mix and match, using options.

It’s not only possible, but it’s easier than you might think.  If an
option is set (cluster.nsr IIRC) then we replace cluster/afr with
cluster/nsr-client and then add some translators to the server-side
stack.  A year ago that was just one nsr-server translator.  The
journaling part has already been split out, and I plan to do the same
with the leader-election parts (making them usable for server-side AFR
or EC) as well.  It shouldn’t be hard to control the addition and
removal of these and related translators (e.g. index) with multiple
options instead of just one.  The biggest stumbling block I’ve actually
hit when trying to do this with AFR on the server side is the *tests*,
many of which can’t handle delays on the client side while the server
side elects leaders and cross-connects peers.  That’s all solvable.  It
just would have taken more time than I had available for the experiment.


precisely. I think switching is not that difficult once we make sure
healing is complete. Switching is a rare operation IMO so we can
probably ask the users to do stop/choose-new-value/start the volume
after choosing the options. This way is simpler than to migrate
between the volumes where you have to probably copy the data.

The two sets of metadata are *entirely* disjoint, which puts us in a
good position compared e.g. to DHT/tiering which had overlaps.  As long
as the bricks are “clean” switching back and forth should be simple.  In
fact I expect to do this a lot when we get to characterizing performance
etc.

Good to hear this.



choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future
if we come up with better metadata journals/stores it should be
easy to plug them is what I'm thinking. The idea I have is based on
the workload, users should be able to decide which pair of
synchronization/metadata works best for them (Or we can also
recommend based on our tests). Wanted to seek your inputs.

Absolutely.  As I’m sure you’re tired of hearing, I believe NSR will
outperform AFR by a significant margin for most workloads and
configurations.  I wouldn’t be the project’s initiator/leader if I
didn’t believe that, but I’m OK if others disagree.  We’ll find out
eventually.  ;)  More importantly, “most” is still not “all”.  Even by
my own reckoning, there are cases in which AFR will perform better or be
preferable for other reasons.  EC’s durability and space-efficiency
advantages make an even stronger case for preserving both kinds of data
paths and metadata arrangements.  That’s precisely why I want to make
the journaling and leader-election parts more generic.

All the best for your endeavors! Lets make users happy.

Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec

[Jeff Darcy]

> >> I want to understand if there is a possibility of exposing these as
> >> different modules that we can mix and match, using options.

It’s not only possible, but it’s easier than you might think.  If an
option is set (cluster.nsr IIRC) then we replace cluster/afr with
cluster/nsr-client and then add some translators to the server-side
stack.  A year ago that was just one nsr-server translator.  The
journaling part has already been split out, and I plan to do the same
with the leader-election parts (making them usable for server-side AFR
or EC) as well.  It shouldn’t be hard to control the addition and
removal of these and related translators (e.g. index) with multiple
options instead of just one.  The biggest stumbling block I’ve actually
hit when trying to do this with AFR on the server side is the *tests*,
many of which can’t handle delays on the client side while the server
side elects leaders and cross-connects peers.  That’s all solvable.  It
just would have taken more time than I had available for the experiment.

> precisely. I think switching is not that difficult once we make sure
> healing is complete. Switching is a rare operation IMO so we can
> probably ask the users to do stop/choose-new-value/start the volume
> after choosing the options. This way is simpler than to migrate
> between the volumes where you have to probably copy the data.

The two sets of metadata are *entirely* disjoint, which puts us in a
good position compared e.g. to DHT/tiering which had overlaps.  As long
as the bricks are “clean” switching back and forth should be simple.  In
fact I expect to do this a lot when we get to characterizing performance
etc.

> >> choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future
> >> if we come up with better metadata journals/stores it should be
> >> easy to plug them is what I'm thinking. The idea I have is based on
> >> the workload, users should be able to decide which pair of
> >> synchronization/metadata works best for them (Or we can also
> >> recommend based on our tests). Wanted to seek your inputs.

Absolutely.  As I’m sure you’re tired of hearing, I believe NSR will
outperform AFR by a significant margin for most workloads and
configurations.  I wouldn’t be the project’s initiator/leader if I
didn’t believe that, but I’m OK if others disagree.  We’ll find out
eventually.  ;)  More importantly, “most” is still not “all”.  Even by
my own reckoning, there are cases in which AFR will perform better or be
preferable for other reasons.  EC’s durability and space-efficiency
advantages make an even stronger case for preserving both kinds of data
paths and metadata arrangements.  That’s precisely why I want to make
the journaling and leader-election parts more generic.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec

[Shyam]

On 10/29/2015 02:06 AM, Pranith Kumar Karampuri wrote:

hi,
  I want to understand how are you guys planning to integrate
NSR volumes to the existing CLIs. Here are some thoughts I had, wanted
to know your thoughts:
At the heart of both the replication/ec schemes we have
1) synchronization mechanisms
a) afr,ec does it using locks
b) nsr does it using leader election
2) Metadata to figure out the healing/reconciliation aspects
a) afr,ec does it using xattrs
b) nsr does it using journals

I want to understand if there is a possibility of exposing these as
different modules that we can mix and match, using options. If the users
choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we
come up with better metadata journals/stores it should be easy to plug
them is what I'm thinking. The idea I have is based on the workload,
users should be able to decide which pair of synchronization/metadata
works best for them (Or we can also recommend based on our tests).


I wanted to add some more *insights* (or stir up the mess) based on the 
DHT2 MDS/DS split. With this case, we could (that may even change to a 
would) have NSR/AFR *only* on the MDS side of things and no EC there, 
and have any of EC/AFR/NSR on the DS side of things.


The theory being, EC provides data space efficiency and replicates 
meta-data anyway, so instead of having that on the MDS side of things, 
which does not have any data, we may want to stick to replication 
methods for availability reasons.


I am adding this here, although this is orthogonal to the current 
discussion, just to provide a perspective into our thought process.



Wanted to seek your inputs.

Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] pluggability of some aspects in afr/nsr/ec

[Pranith Kumar Karampuri]

hi,
 I want to understand how are you guys planning to integrate 
NSR volumes to the existing CLIs. Here are some thoughts I had, wanted 
to know your thoughts:

At the heart of both the replication/ec schemes we have
1) synchronization mechanisms
   a) afr,ec does it using locks
   b) nsr does it using leader election
2) Metadata to figure out the healing/reconciliation aspects
   a) afr,ec does it using xattrs
   b) nsr does it using journals

I want to understand if there is a possibility of exposing these as 
different modules that we can mix and match, using options. If the users 
choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we 
come up with better metadata journals/stores it should be easy to plug 
them is what I'm thinking. The idea I have is based on the workload, 
users should be able to decide which pair of synchronization/metadata 
works best for them (Or we can also recommend based on our tests). 
Wanted to seek your inputs.


Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec

[Venky Shankar]

On Thu, Oct 29, 2015 at 11:36 AM, Pranith Kumar Karampuri
 wrote:
> hi,
>  I want to understand how are you guys planning to integrate NSR
> volumes to the existing CLIs. Here are some thoughts I had, wanted to know
> your thoughts:
> At the heart of both the replication/ec schemes we have
> 1) synchronization mechanisms
>a) afr,ec does it using locks
>b) nsr does it using leader election
> 2) Metadata to figure out the healing/reconciliation aspects
>a) afr,ec does it using xattrs
>b) nsr does it using journals
>
> I want to understand if there is a possibility of exposing these as
> different modules that we can mix and match, using options. If the users

Do you mean abstracting it out during volume creation? At a high level
this could be in the form of client or server
side replication. Not that AFR cannot be used on the server side
(you'd know better than me), but, if at all this level
of abstraction is used, we'd need to default to what fits best in what
use case (as you already mentioned below)
but still retaining the flexibility to override it.

> choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come
> up with better metadata journals/stores it should be easy to plug them is
> what I'm thinking. The idea I have is based on the workload, users should be
> able to decide which pair of synchronization/metadata works best for them
> (Or we can also recommend based on our tests). Wanted to seek your inputs.
>
> Pranith
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] pluggability of some aspects in afr/nsr/ec

[Pranith Kumar Karampuri]

On 10/29/2015 12:18 PM, Venky Shankar wrote:

On Thu, Oct 29, 2015 at 11:36 AM, Pranith Kumar Karampuri
 wrote:

hi,
  I want to understand how are you guys planning to integrate NSR
volumes to the existing CLIs. Here are some thoughts I had, wanted to know
your thoughts:
At the heart of both the replication/ec schemes we have
1) synchronization mechanisms
a) afr,ec does it using locks
b) nsr does it using leader election
2) Metadata to figure out the healing/reconciliation aspects
a) afr,ec does it using xattrs
b) nsr does it using journals

I want to understand if there is a possibility of exposing these as
different modules that we can mix and match, using options. If the users

Do you mean abstracting it out during volume creation? At a high level
this could be in the form of client or server
side replication. Not that AFR cannot be used on the server side
(you'd know better than me), but, if at all this level
of abstraction is used, we'd need to default to what fits best in what
use case (as you already mentioned below)
but still retaining the flexibility to override it.
precisely. I think switching is not that difficult once we make sure 
healing is complete. Switching is a rare operation IMO so we can 
probably ask the users to do stop/choose-new-value/start the volume 
after choosing the options. This way is simpler than to migrate between 
the volumes where you have to probably copy the data.


Pranith



choose 1b, 2b it becomes nsr and 1a, 2a becomes afr/ec. In future if we come
up with better metadata journals/stores it should be easy to plug them is
what I'm thinking. The idea I have is based on the workload, users should be
able to decide which pair of synchronization/metadata works best for them
(Or we can also recommend based on our tests). Wanted to seek your inputs.

Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel