Re: [Gluster-devel] sub-directory geo-replication, snapshot features

2016-03-09 Thread Shyam 

On 03/09/2016 09:05 AM, Kaushal M wrote:

On Wed, Mar 9, 2016 at 6:24 PM, Shyam  wrote:

On 03/09/2016 02:25 AM, Pranith Kumar Karampuri wrote:




On 03/09/2016 10:40 AM, Kaushal M wrote:


On Tue, Mar 8, 2016 at 11:58 PM, Atin Mukherjee 
wrote:



On 03/08/2016 07:32 PM, Pranith Kumar Karampuri wrote:


hi,
   Late last week I sent a solution for how to achieve
subdirectory-mount support with access-controls

(http://www.gluster.org/pipermail/gluster-devel/2016-March/048537.html).

What follows here is a short description of how other features of
gluster volumes are implemented for sub-directories.

Please note that the sub-directories are not allowed to be accessed by
normal mounts i.e. top-level volume mounts. All access to the
sub-directories goes only through sub-directory mounts.


Is this acceptable? If I have a,b,c sub directories in the volume and if
I mount the same volume in /mnt then do you mean to say I won't be able
to access /mnt/a or /mnt/b and I can only access them using sub
directory mounts? Or you are talking about some specific case here?


1) Geo-replication:
The direction in which we are going is to allow geo-replicating just
some sub-directories and not all of the volume based on options. When
these options are set, server xlators populate extra information in the
frames/xdata to write changelog for the fops coming from their
sub-directory mounts. changelog xlator on seeing this will only
geo-replicate the files/directories that are in the changelog. Thus
only
the sub-directories are geo-replicated. There is also a suggestion from
Vijay and Aravinda to have separate domains for operations inside
sub-directories for changelogs.

2) Sub-directory snapshots using lvm
Every time a sub-directory needs to be created, Our idea is that the
admin needs to execute subvolume creation command which creates a mount
to an empty snapshot at the given sub-directory name. All these
directories can be modified in parallel and we can take individual
snapshots of each of the directories. We will be providing a detailed
list commands to do the same once they are fleshed out. At the moment
these are the directions we are going to increase granularity from
volume to subdirectory for the main features.


We use hardlinks to the `.glusterfs` directory on bricks. So wouldn't
having multiple filesystems inside a brick break the brick?



You are right. I think we will have to do full separation, where it will
be more like multiple tenants :-/.



Ah! I had a half baked response to this yesterday, asking if subdirs were
their own FS instances and LVM instances etc. looks like that was the case.

Anyway, apart form the .glusterfs this idea merits some thinking, like a
virtual volume that chains up separate mounts on the local FS, consider it a
virtual namespace that links subvolumes together. So that we do not need to
create separate volumes and have that heavy weight process in place, at the
same time provide snapshots as well.


If we are thinking about subdirs as virtual volumes and want to avoid
a process per volume, why not just consider brick-multiplexing and
proper full fledged volumes.
We already have multiplexing on the agenda for 4.0, and it would seem
easier to just leverage the facilities provided by that to get similar
behavior, instead of attempting to add-in sub-directory support for
all sort of operations/features.


+100, I simply agree. This (daemon multiplexing) and volume management 
scalability with GD2.0 (i.e ability to manage larger number of volumes 
per trusted pool) would help a good deal. (I rather subscribe to this 
view myself anyway). Also, things like network isolation etc. but those 
can be worked at a volume level from there on.




One of the benefits of supporting sub directory mounts is that it
would make it easier to create and manage multiple shares for many
tenants. Trying to re-implement all volume operations for
sub-directories only makes it harder for us and makes the solution
really complex.


Yes agreed again.

The one potential (external?) issue is dm-thinp scalability in this 
scenario needs to be tested (i.e ability to carve out multiple LVs from 
a single thin-p, and their management).











Also, I'd prefer if sub-directory mounts and sub-directory snapshots
remained separate, and not tied with each other. This mail gives the
feeling that they will be tied together.



With the above point, I don't think it will be.




Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] sub-directory geo-replication, snapshot features

2016-03-09 Thread Shyam 

On 03/09/2016 02:25 AM, Pranith Kumar Karampuri wrote:



On 03/09/2016 10:40 AM, Kaushal M wrote:

On Tue, Mar 8, 2016 at 11:58 PM, Atin Mukherjee 
wrote:


On 03/08/2016 07:32 PM, Pranith Kumar Karampuri wrote:

hi,
  Late last week I sent a solution for how to achieve
subdirectory-mount support with access-controls
(http://www.gluster.org/pipermail/gluster-devel/2016-March/048537.html).

What follows here is a short description of how other features of
gluster volumes are implemented for sub-directories.

Please note that the sub-directories are not allowed to be accessed by
normal mounts i.e. top-level volume mounts. All access to the
sub-directories goes only through sub-directory mounts.

Is this acceptable? If I have a,b,c sub directories in the volume and if
I mount the same volume in /mnt then do you mean to say I won't be able
to access /mnt/a or /mnt/b and I can only access them using sub
directory mounts? Or you are talking about some specific case here?

1) Geo-replication:
The direction in which we are going is to allow geo-replicating just
some sub-directories and not all of the volume based on options. When
these options are set, server xlators populate extra information in the
frames/xdata to write changelog for the fops coming from their
sub-directory mounts. changelog xlator on seeing this will only
geo-replicate the files/directories that are in the changelog. Thus
only
the sub-directories are geo-replicated. There is also a suggestion from
Vijay and Aravinda to have separate domains for operations inside
sub-directories for changelogs.

2) Sub-directory snapshots using lvm
Every time a sub-directory needs to be created, Our idea is that the
admin needs to execute subvolume creation command which creates a mount
to an empty snapshot at the given sub-directory name. All these
directories can be modified in parallel and we can take individual
snapshots of each of the directories. We will be providing a detailed
list commands to do the same once they are fleshed out. At the moment
these are the directions we are going to increase granularity from
volume to subdirectory for the main features.

We use hardlinks to the `.glusterfs` directory on bricks. So wouldn't
having multiple filesystems inside a brick break the brick?


You are right. I think we will have to do full separation, where it will
be more like multiple tenants :-/.


Ah! I had a half baked response to this yesterday, asking if subdirs 
were their own FS instances and LVM instances etc. looks like that was 
the case.


Anyway, apart form the .glusterfs this idea merits some thinking, like a 
virtual volume that chains up separate mounts on the local FS, consider 
it a virtual namespace that links subvolumes together. So that we do not 
need to create separate volumes and have that heavy weight process in 
place, at the same time provide snapshots as well.






Also, I'd prefer if sub-directory mounts and sub-directory snapshots
remained separate, and not tied with each other. This mail gives the
feeling that they will be tied together.


With the above point, I don't think it will be.




Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] sub-directory geo-replication, snapshot features

2016-03-08 Thread Pranith Kumar Karampuri 



On 03/09/2016 10:40 AM, Kaushal M wrote:

On Tue, Mar 8, 2016 at 11:58 PM, Atin Mukherjee  wrote:


On 03/08/2016 07:32 PM, Pranith Kumar Karampuri wrote:

hi,
  Late last week I sent a solution for how to achieve
subdirectory-mount support with access-controls
(http://www.gluster.org/pipermail/gluster-devel/2016-March/048537.html).
What follows here is a short description of how other features of
gluster volumes are implemented for sub-directories.

Please note that the sub-directories are not allowed to be accessed by
normal mounts i.e. top-level volume mounts. All access to the
sub-directories goes only through sub-directory mounts.

Is this acceptable? If I have a,b,c sub directories in the volume and if
I mount the same volume in /mnt then do you mean to say I won't be able
to access /mnt/a or /mnt/b and I can only access them using sub
directory mounts? Or you are talking about some specific case here?

1) Geo-replication:
The direction in which we are going is to allow geo-replicating just
some sub-directories and not all of the volume based on options. When
these options are set, server xlators populate extra information in the
frames/xdata to write changelog for the fops coming from their
sub-directory mounts. changelog xlator on seeing this will only
geo-replicate the files/directories that are in the changelog. Thus only
the sub-directories are geo-replicated. There is also a suggestion from
Vijay and Aravinda to have separate domains for operations inside
sub-directories for changelogs.

2) Sub-directory snapshots using lvm
Every time a sub-directory needs to be created, Our idea is that the
admin needs to execute subvolume creation command which creates a mount
to an empty snapshot at the given sub-directory name. All these
directories can be modified in parallel and we can take individual
snapshots of each of the directories. We will be providing a detailed
list commands to do the same once they are fleshed out. At the moment
these are the directions we are going to increase granularity from
volume to subdirectory for the main features.

We use hardlinks to the `.glusterfs` directory on bricks. So wouldn't
having multiple filesystems inside a brick break the brick?


You are right. I think we will have to do full separation, where it will 
be more like multiple tenants :-/.




Also, I'd prefer if sub-directory mounts and sub-directory snapshots
remained separate, and not tied with each other. This mail gives the
feeling that they will be tied together.


With the above point, I don't think it will be.




Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] sub-directory geo-replication, snapshot features

2016-03-08 Thread Aravinda 


regards
Aravinda

On 03/08/2016 07:32 PM, Pranith Kumar Karampuri wrote:

hi,
 Late last week I sent a solution for how to achieve 
subdirectory-mount support with access-controls 
(http://www.gluster.org/pipermail/gluster-devel/2016-March/048537.html). 
What follows here is a short description of how other features of 
gluster volumes are implemented for sub-directories.


Please note that the sub-directories are not allowed to be accessed by 
normal mounts i.e. top-level volume mounts. All access to the 
sub-directories goes only through sub-directory mounts.


1) Geo-replication:
The direction in which we are going is to allow geo-replicating just 
some sub-directories and not all of the volume based on options. When 
these options are set, server xlators populate extra information in 
the frames/xdata to write changelog for the fops coming from their 
sub-directory mounts. changelog xlator on seeing this will only 
geo-replicate the files/directories that are in the changelog. Thus 
only the sub-directories are geo-replicated. There is also a 
suggestion from Vijay and Aravinda to have separate domains for 
operations inside sub-directories for changelogs.
We can additionally record subdir/client info in Changelog to 
differentiate I/O belonging to each subdirs instead of having separate 
domains for changelogs.


Just a note, Geo-replication expects target to be Gluster Volume and not 
any directory. If subdir 1 to be replicated to remote site A and subdir2 
to be replicated to remote site B, then we should have two Geo-rep 
session from Master to two remote volumes in remote site A and site B 
respectively.(with subdir filter accordingly)


2) Sub-directory snapshots using lvm
Every time a sub-directory needs to be created, Our idea is that the 
admin needs to execute subvolume creation command which creates a 
mount to an empty snapshot at the given sub-directory name. All these 
directories can be modified in parallel and we can take individual 
snapshots of each of the directories. We will be providing a detailed 
list commands to do the same once they are fleshed out. At the moment 
these are the directions we are going to increase granularity from 
volume to subdirectory for the main features.


Pranith


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel