Re: netapp-alike snapshots?

[Marc MERLIN]

On Sat, Sep 09, 2017 at 10:43:16PM +0300, Andrei Borzenkov wrote:
> 09.09.2017 16:44, Ulli Horlacher пишет:
> > 
> > Your tool does not create .snapshot subdirectories in EVERY directory like
> 
> Neither does NetApp. Those "directories" are magic handles that do not
> really exist.
 
Correct, thanks for saving me typing the same thing (I actually did work
at netapp many years back, so I'm familiar with how they work)

> > Netapp does.
> > Example:
> > 
> > framstag@fex:~: cd ~/Mail/.snapshot/
> > framstag@fex:~/Mail/.snapshot: l
> > lR-X - 2017-09-09 09:55 2017-09-09_.daily -> 
> > /local/home/.snapshot/2017-09-09_.daily/framstag/Mail
> 
> Apart from obvious problem with recursive directory traversal (NetApp
> .snapshot are not visible with normal directory list) those will also be
> captured in snapshots and cannot be removed. NetApp snapshots themselves
> do not expose .snapshot "directories".

Correct. Netapp knows this of course, which is why those .snapshot
directories are "magic" and hidden to ls(1), find(1) and others when
they do a readdir(3)

> > lR-X - 2017-09-09 14:00 2017-09-09_1400.hourly -> 
> > /local/home/.snapshot/2017-09-09_1400.hourly/framstag/Mail
> > lR-X - 2017-09-09 15:00 2017-09-09_1500.hourly -> 
> > /local/home/.snapshot/2017-09-09_1500.hourly/framstag/Mail
> > lR-X - 2017-09-09 15:18 2017-09-09_1518.single -> 
> > /local/home/.snapshot/2017-09-09_1518.single/framstag/Mail
> > lR-X - 2017-09-09 15:20 2017-09-09_1520.single -> 
> > /local/home/.snapshot/2017-09-09_1520.single/framstag/Mail
> > lR-X - 2017-09-09 15:22 2017-09-09_1522.single -> 
> > /local/home/.snapshot/2017-09-09_1522.single/framstag/Mail
> > 
> > My users (and I) need snapshots in this way.

You are used to them being there, I was too :)
While you could create lots of symlinks, I opted not to since it would
have littered the filesystem.
I can simply cd $(SNAPROOT)/volname_hourly/$(PWD)
and end up where I wanted to be.

I suppose you could make a snapcd shell function that does this for you.
The only issue is that volname_hourly comes before the rest of the path,
so you aren't given a list of all the snapshots available for a given
path, you have to cd into the given snapshot first, and then add the
path.
I agree it's not as nice as netapp, but honestly I don't think you can
do better with btrfs at this point.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems 
   what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[A L]

Perhaps netapp is using a VFS overlay. There is really only one snapshot but it 
is shown in the overlay on every folder. Kind of the same with samba Shadow 
Copies.

 From: Ulli Horlacher  -- Sent: 2017-09-09 - 
21:52 

> On Sat 2017-09-09 (22:43), Andrei Borzenkov wrote:
> 
>> > Your tool does not create .snapshot subdirectories in EVERY directory like
>> 
>> Neither does NetApp. Those "directories" are magic handles that do not
>> really exist.
> 
> I know.
> But symbolic links are the next close thing (I am not a kernel programmer).
> 
> 
>> Apart from obvious problem with recursive directory traversal (NetApp
>> .snapshot are not visible with normal directory list)
> 
> Yes, they are, at least sometimes, eg tar includes the snapshots.
> 
> 
> -- 
> Ullrich Horlacher  Server und Virtualisierung
> Rechenzentrum TIK 
> Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
> Allmandring 30aTel:++49-711-68565868
> 70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
> REF:<14c87878-a5a0-d7d3-4a76-c55812e75...@gmail.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Sat 2017-09-09 (22:43), Andrei Borzenkov wrote:

> > Your tool does not create .snapshot subdirectories in EVERY directory like
> 
> Neither does NetApp. Those "directories" are magic handles that do not
> really exist.

I know.
But symbolic links are the next close thing (I am not a kernel programmer).


> Apart from obvious problem with recursive directory traversal (NetApp
> .snapshot are not visible with normal directory list)

Yes, they are, at least sometimes, eg tar includes the snapshots.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<14c87878-a5a0-d7d3-4a76-c55812e75...@gmail.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Andrei Borzenkov]

09.09.2017 16:44, Ulli Horlacher пишет:
> 
> Your tool does not create .snapshot subdirectories in EVERY directory like

Neither does NetApp. Those "directories" are magic handles that do not
really exist.

> Netapp does.
> Example:
> 
> framstag@fex:~: cd ~/Mail/.snapshot/
> framstag@fex:~/Mail/.snapshot: l
> lR-X - 2017-09-09 09:55 2017-09-09_.daily -> 
> /local/home/.snapshot/2017-09-09_.daily/framstag/Mail

Apart from obvious problem with recursive directory traversal (NetApp
.snapshot are not visible with normal directory list) those will also be
captured in snapshots and cannot be removed. NetApp snapshots themselves
do not expose .snapshot "directories".

> lR-X - 2017-09-09 14:00 2017-09-09_1400.hourly -> 
> /local/home/.snapshot/2017-09-09_1400.hourly/framstag/Mail
> lR-X - 2017-09-09 15:00 2017-09-09_1500.hourly -> 
> /local/home/.snapshot/2017-09-09_1500.hourly/framstag/Mail
> lR-X - 2017-09-09 15:18 2017-09-09_1518.single -> 
> /local/home/.snapshot/2017-09-09_1518.single/framstag/Mail
> lR-X - 2017-09-09 15:20 2017-09-09_1520.single -> 
> /local/home/.snapshot/2017-09-09_1520.single/framstag/Mail
> lR-X - 2017-09-09 15:22 2017-09-09_1522.single -> 
> /local/home/.snapshot/2017-09-09_1522.single/framstag/Mail
> 
> My users (and I) need snapshots in this way.
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Sat 2017-09-09 (06:36), Marc MERLIN wrote:

> > On Tue 2017-08-22 (15:22), Ulli Horlacher wrote:
> > > With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> > > You can find these snapshots in every local directory (readonly).
> > 
> > I have found none, so I have implemented it by myself:
> > 
> > https://fex.rus.uni-stuttgart.de/snaprotate.html
> 
> Not sure how you looked :)
> http://marc.merlins.org/perso/btrfs/post_2014-03-21_Btrfs-Tips_-How-To-Setup-Netapp-Style-Snapshots.html
> 
> Might not be exactly what you wanted, but been using it for 3 years.

Your tool does not create .snapshot subdirectories in EVERY directory like
Netapp does.
Example:

framstag@fex:~: cd ~/Mail/.snapshot/
framstag@fex:~/Mail/.snapshot: l
lR-X - 2017-09-09 09:55 2017-09-09_.daily -> 
/local/home/.snapshot/2017-09-09_.daily/framstag/Mail
lR-X - 2017-09-09 14:00 2017-09-09_1400.hourly -> 
/local/home/.snapshot/2017-09-09_1400.hourly/framstag/Mail
lR-X - 2017-09-09 15:00 2017-09-09_1500.hourly -> 
/local/home/.snapshot/2017-09-09_1500.hourly/framstag/Mail
lR-X - 2017-09-09 15:18 2017-09-09_1518.single -> 
/local/home/.snapshot/2017-09-09_1518.single/framstag/Mail
lR-X - 2017-09-09 15:20 2017-09-09_1520.single -> 
/local/home/.snapshot/2017-09-09_1520.single/framstag/Mail
lR-X - 2017-09-09 15:22 2017-09-09_1522.single -> 
/local/home/.snapshot/2017-09-09_1522.single/framstag/Mail

My users (and I) need snapshots in this way.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<20170909133612.7iqwr6cbjxzvf...@merlins.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Marc MERLIN]

On Sat, Sep 09, 2017 at 03:26:14PM +0200, Ulli Horlacher wrote:
> On Tue 2017-08-22 (15:22), Ulli Horlacher wrote:
> > With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> > You can find these snapshots in every local directory (readonly).
> 
> > I would like to have something similar with btrfs.
> > Is there (where?) such a tool?
> 
> I have found none, so I have implemented it by myself:
> 
> https://fex.rus.uni-stuttgart.de/snaprotate.html

Not sure how you looked :)
https://www.google.com/search?q=btrfs+netapp+snapshot
http://marc.merlins.org/perso/btrfs/post_2014-03-21_Btrfs-Tips_-How-To-Setup-Netapp-Style-Snapshots.html

Might not be exactly what you wanted, but been using it for 3 years.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems 
   what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (15:22), Ulli Horlacher wrote:
> With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> You can find these snapshots in every local directory (readonly).

> I would like to have something similar with btrfs.
> Is there (where?) such a tool?

I have found none, so I have implemented it by myself:

https://fex.rus.uni-stuttgart.de/snaprotate.html

In contrast to Netapp, with snaprotate the local host administrator can
create a snapshot at any time or by cronjob.

Example:

root@fex:~# snaprotate single 3 /local/home
Create a readonly snapshot of '/local/home' in 
'/local/home/.snapshot/2017-09-09_1518.single'
Delete subvolume '/local/home/.snapshot/2017-09-09_1255.single'

root@fex:~# snaprotate -l
/local/home/.snapshot/2017-09-08_.daily
/local/home/.snapshot/2017-09-09_.daily
/local/home/.snapshot/2017-09-09_1331.single
/local/home/.snapshot/2017-09-09_1332.single
/local/home/.snapshot/2017-09-09_1400.hourly
/local/home/.snapshot/2017-09-09_1500.hourly
/local/home/.snapshot/2017-09-09_1518.single

root@fex:~# crontab -l | grep snaprotate
0 * * * * /root/bin/snaprotate -q hourly 2 /local/home
0 0 * * * /root/bin/snaprotate -q daily  3 /local/home
0 0 * * 1 /root/bin/snaprotate -q weekly 1 /local/home

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<20170822132208.gd14...@rus.uni-stuttgart.de>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (19:36), Peter Grandi wrote:

> Indeed and there is a fair description of some options for
> subvolume nesting policies here which may be interesting to the
> original poster:
> 
>   https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Layout
> 
> It is unsurprising to me that there are tradeoffs involved in
> every choice. I find the "Flat" layout particularly desirable.

My layout is already nearly "flat".
It seems my decision was right :-)



> Btrfs snapshots can only be done for a whole subvolume.

I know this.

> Subvolumes and snapshots can be created by users, but too many snapshots
> (see below) can cause trouble. For somewhat good reasons subvolumes
> including snapshots cannot be deleted by users though unless mount option
> 'user_subvol_rm_allowed' is used.

Ooops, this is new to me!

framstag@fex:~: btrfs subvolume create xx
Create subvolume './xx'

framstag@fex:~: btrfs subvolume delete xx
Delete subvolume '/local/home/framstag/xx'
ERROR: cannot delete '/local/home/framstag/xx' - Operation not permitted

This means, root has to remove the subvolme.
Is it possible to disallow creation of subvolumes for normal users?



> >>> Because Netapp do it this way - for at least 20 years and we
> >>> have a multi-PB Netapp storage environment. No chance to change
> >>> this.
> 
> Send patches :-).

For waffle or btrfs? :-)


> Assumptions that all Btrfs features such as snapshots are
> infinitely scalable at no cost may be optimistic:
> 
>   
> https://btrfs.wiki.kernel.org/index.php/Gotchas#Having_many_subvolumes_can_be_very_slow

"when you do device removes on file systems with a lot of snapshots, it
 is unbelievably slow ... took nearly a week to move 20GB of FS data from
 one device to the other using that method"
  
"a balance on 2TB of data that was heavily snapshotted - it took 3 months" 

ARGH!!
Thanks for this warning!
I will overthink my multi-snapshots plan!

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<22940.31139.194399.982...@tree.ty.sabi.co.uk>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Peter Grandi]

[ ... ]

 It is beneficial to not have snapshots in-place. With a local
 directory of snapshots, [ ... ]

Indeed and there is a fair description of some options for
subvolume nesting policies here which may be interesting to the
original poster:

  https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Layout

It is unsurprising to me that there are tradeoffs involved in
every choice. I find the "Flat" layout particularly desirable.

>>> Netapp snapshots are invisible for tools doing opendir()/
>>> readdir() One could simulate this with symlinks for the
>>> snapshot directory: store the snapshot elsewhere (not inplace)
>>> and create a symlink to it, in every directory.

More precisely in every subvolume root directory.

>>> My users want the snapshots locally in a .snapshot
>>> subdirectory.

Btrfs snapshots can only be done for a whole subvolume. Subvolumes
and snapshots can be created by users, but too many snapshots (see
below) can cause trouble. For somewhat good reasons subvolumes
including snapshots cannot be deleted by users though unless mount
option 'user_subvol_rm_allowed' is used.

>>> Because Netapp do it this way - for at least 20 years and we
>>> have a multi-PB Netapp storage environment. No chance to change
>>> this.

Send patches :-).

> Not only du works recursivly, but also find and with option
> also ls, grep, etc.

Note also that subvolume root directory inodes are indeed root
directory inodes so they can be 'mount'ed and therefore the
transition from a subvolume into a contained subvolume can be
detected at the mountpoint.

So 'find' has the '-xdev' option and 'du' has the '-x' options and
so similarly nearly all other tools, so perhaps someone expects
that to happen :-).

> And it would require a bind mount for EVERY directory. There can
> be hundreds... thousends!

Assumptions that all Btrfs features such as snapshots are
infinitely scalable at no cost may be optimistic:

  
https://btrfs.wiki.kernel.org/index.php/Gotchas#Having_many_subvolumes_can_be_very_slow
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (22:36), Roman Mamedov wrote:

> > My users want the snapshots locally in a .snapshot subdirectory.
> > Because Netapp do it this way - for at least 20 years and we have a
> > multi-PB Netapp storage environment.
> 
> Just a side note, you do know that only subvolumes can be snapshotted on 
> Btrfs,
> not any regular directory? And that snapshots are not recursive, i.e. if a
> subvolume "contains" other subvolumes (hint: it really doesn't), snapshots of
> the parent one will not include content of subvolumes below that in the tree.

Yes, I know this. But thanks for your hints! (Other readers here may be
not aware of this)


> I don't know how Netapp does this

I am only a Netapp/waffle user, so I know no internals.
Netapp is not Linux based and definitly a lot older than btrfs.


> from the way you describe that setup it feels like with Btrfs you're
> still in for some bad surprises and a part of your expectations will not
> be met.

I will take care :-)


> Do you plan to make each and every directory and subdirectory a subvolume

No. My idea is to place a symlink in every subdirectory pointing to the
snapshot directory. Not yet programmed...
I was hoping someone already has implemented such a feature.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<2017083647.350ca27d@natsu>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (19:19), A L wrote:
> Perhaps using a bind mount? It would look and work the same as a ordinary fs. 
> Just need to make sure du uses one filesystem.
> 
>  From: Ulli Horlacher  -- Sent: 2017-08-22 
> - 18:57 
> 
> > On Tue 2017-08-22 (21:45), Roman Mamedov wrote:
> > 
> >> It is beneficial to not have snapshots in-place. With a local directory of
> >> snapshots, issuing things like "find", "grep -r" or even "du" will take an
> >> inordinate amount of time and will produce a result you do not expect.
> > 
> > Netapp snapshots are invisible for tools doing opendir()/readdir()
> > One could simulate this with symlinks for the snapshot directory:
> > store the snapshot elsewhere (not inplace) and create a symlink to it, in
> > every directory.

Not only du works recursivly, but also find and with option also ls, grep,
etc.

And it would require a bind mount for EVERY directory. There can be
hundreds... thousends!


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Roman Mamedov]

On Tue, 22 Aug 2017 18:57:25 +0200
Ulli Horlacher  wrote:

> On Tue 2017-08-22 (21:45), Roman Mamedov wrote:
> 
> > It is beneficial to not have snapshots in-place. With a local directory of
> > snapshots, issuing things like "find", "grep -r" or even "du" will take an
> > inordinate amount of time and will produce a result you do not expect.
> 
> Netapp snapshots are invisible for tools doing opendir()/readdir()
> One could simulate this with symlinks for the snapshot directory:
> store the snapshot elsewhere (not inplace) and create a symlink to it, in
> every directory.
> 
> 
> > Personally I prefer to have a /snapshots directory on every FS
> 
> My users want the snapshots locally in a .snapshot subdirectory.
> Because Netapp do it this way - for at least 20 years and we have a
> multi-PB Netapp storage environment.
> No chance to change this.

Just a side note, you do know that only subvolumes can be snapshotted on Btrfs,
not any regular directory? And that snapshots are not recursive, i.e. if a
subvolume "contains" other subvolumes (hint: it really doesn't), snapshots of
the parent one will not include content of subvolumes below that in the tree.

I don't know how Netapp does this, from the way you describe that setup it
feels like with Btrfs you're still in for some bad surprises and a part of
your expectations will not be met.

Do you plan to make each and every directory and subdirectory a subvolume (so
that it could have a trail of its own snapshots)? There will be performance
implications to that. Also deleting subvolumes can only be done via the
"btrfs" tool, they won't delete like normal dirs, e.g. when trying to do that
remotely via NFS or Samba share.

-- 
With respect,
Roman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[A L]

Perhaps using a bind mount? It would look and work the same as a ordinary fs. 
Just need to make sure du uses one filesystem.

 From: Ulli Horlacher  -- Sent: 2017-08-22 - 
18:57 

> On Tue 2017-08-22 (21:45), Roman Mamedov wrote:
> 
>> It is beneficial to not have snapshots in-place. With a local directory of
>> snapshots, issuing things like "find", "grep -r" or even "du" will take an
>> inordinate amount of time and will produce a result you do not expect.
> 
> Netapp snapshots are invisible for tools doing opendir()/readdir()
> One could simulate this with symlinks for the snapshot directory:
> store the snapshot elsewhere (not inplace) and create a symlink to it, in
> every directory.
> 
> 
>> Personally I prefer to have a /snapshots directory on every FS
> 
> My users want the snapshots locally in a .snapshot subdirectory.
> Because Netapp do it this way - for at least 20 years and we have a
> multi-PB Netapp storage environment.
> No chance to change this.
> 
> -- 
> Ullrich Horlacher  Server und Virtualisierung
> Rechenzentrum TIK 
> Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
> Allmandring 30aTel:++49-711-68565868
> 70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
> REF:<20170822214531.44538589@natsu>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (21:45), Roman Mamedov wrote:

> It is beneficial to not have snapshots in-place. With a local directory of
> snapshots, issuing things like "find", "grep -r" or even "du" will take an
> inordinate amount of time and will produce a result you do not expect.

Netapp snapshots are invisible for tools doing opendir()/readdir()
One could simulate this with symlinks for the snapshot directory:
store the snapshot elsewhere (not inplace) and create a symlink to it, in
every directory.


> Personally I prefer to have a /snapshots directory on every FS

My users want the snapshots locally in a .snapshot subdirectory.
Because Netapp do it this way - for at least 20 years and we have a
multi-PB Netapp storage environment.
No chance to change this.

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<20170822214531.44538589@natsu>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (18:08), Peter Becker wrote:
> This is possible. Use the -b or -B option.
> 
> -b basedir places the snapshot in basedir with a directory structure
> that mimics the mountpoint
> -B basedir places the snapshots in basedir with NO additional
> subdirectory structure
> 
> 2017-08-22 16:24 GMT+02:00 Ulli Horlacher :
> > On Tue 2017-08-22 (15:44), Peter Becker wrote:
> >> Is use: https://github.com/jf647/btrfs-snap
> >>
> >> 2017-08-22 15:22 GMT+02:00 Ulli Horlacher :
> >> > With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> >> > You can find these snapshots in every local directory (readonly).
> >> > Example:
> >> >
> >> > framstag@fex:/sw/share: ll .snapshot/
> >> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> >> > .snapshot/daily.2017-08-15_0010
> >> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> >> > .snapshot/daily.2017-08-16_0010
> >> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> >> > .snapshot/daily.2017-08-17_0010
> >> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> >> > .snapshot/daily.2017-08-18_0010
> >> > drwxr-xr-x  framstag root - 2017-08-18 23:59:29 
> >> > .snapshot/daily.2017-08-19_0010
> >> > drwxr-xr-x  framstag root - 2017-08-19 21:01:25 
> >> > .snapshot/daily.2017-08-20_0010
> >> > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> >> > .snapshot/daily.2017-08-21_0010
> >> > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> >> > .snapshot/hourly.2017-08-20_1210
> >> > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> >> > .snapshot/hourly.2017-08-20_1610
> >> > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> >> > .snapshot/hourly.2017-08-20_2010
> >> > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> >> > .snapshot/hourly.2017-08-21_0810
> >> > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> >> > .snapshot/hourly.2017-08-21_1210
> >> > drwxr-xr-x  framstag root - 2017-08-21 13:05:28 
> >> > .snapshot/hourly.2017-08-21_1610
> >
> > btrfs-snap does not create local .snapshot/ sub-directories, but saves the
> > snapshots in the toplevel root volume directory.

No, I want in EVERY directory of the sourcetree a subdirectory named
snapshot, example:

framstag@fex:/sw/share: ll .snapshot a*/.snapshot a*/*/.snapshot
drwxrwxrwx  root root - 2017-08-22 16:10:01 .snapshot
drwxrwxrwx  root root - 2017-08-22 16:10:01 aggis-1.0/.snapshot
drwxrwxrwx  root root - 2017-08-22 16:10:01 aggis-1.0/bin/.snapshot
drwxrwxrwx  root root - 2017-08-22 16:10:01 aggis-1.0/man/.snapshot

(this is on a Netapp NFS volume)

btrfs-snap creates a snapshot directory tree on a different path.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Roman Mamedov]

On Tue, 22 Aug 2017 16:24:51 +0200
Ulli Horlacher  wrote:

> On Tue 2017-08-22 (15:44), Peter Becker wrote:
> > Is use: https://github.com/jf647/btrfs-snap
> > 
> > 2017-08-22 15:22 GMT+02:00 Ulli Horlacher :
> > > With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> > > You can find these snapshots in every local directory (readonly).
> > > Example:
> > >
> > > framstag@fex:/sw/share: ll .snapshot/
> > > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > > .snapshot/daily.2017-08-15_0010
> > > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > > .snapshot/daily.2017-08-16_0010
> > > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > > .snapshot/daily.2017-08-17_0010
> > > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > > .snapshot/daily.2017-08-18_0010
> > > drwxr-xr-x  framstag root - 2017-08-18 23:59:29 
> > > .snapshot/daily.2017-08-19_0010
> > > drwxr-xr-x  framstag root - 2017-08-19 21:01:25 
> > > .snapshot/daily.2017-08-20_0010
> > > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> > > .snapshot/daily.2017-08-21_0010
> > > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> > > .snapshot/hourly.2017-08-20_1210
> > > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> > > .snapshot/hourly.2017-08-20_1610
> > > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> > > .snapshot/hourly.2017-08-20_2010
> > > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> > > .snapshot/hourly.2017-08-21_0810
> > > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> > > .snapshot/hourly.2017-08-21_1210
> > > drwxr-xr-x  framstag root - 2017-08-21 13:05:28 
> > > .snapshot/hourly.2017-08-21_1610
> 
> btrfs-snap does not create local .snapshot/ sub-directories, but saves the
> snapshots in the toplevel root volume directory.

It is beneficial to not have snapshots in-place. With a local directory of
snapshots, issuing things like "find", "grep -r" or even "du" will take an
inordinate amount of time and will produce a result you do not expect.

For some of those tools the problem can be avoided (by always keeping in mind
to use "-x" with du, or "--one-file-system" with tar), but not for all of them.

Personally I prefer to have a /snapshots directory on every FS, and e.g. timed
snapshots of /home/username/src will live in /snapshots/home-username-src/. No
point to hide it there with a dot either, as it's convenient to be able to
browse older snapshots with GUI filemanagers (which hide dot-files by default).

-- 
With respect,
Roman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: netapp-alike snapshots?

[Peter Becker]

This is possible. Use the -b or -B option.

-b basedir places the snapshot in basedir with a directory structure
that mimics the mountpoint
-B basedir places the snapshots in basedir with NO additional
subdirectory structure

2017-08-22 16:24 GMT+02:00 Ulli Horlacher :
> On Tue 2017-08-22 (15:44), Peter Becker wrote:
>> Is use: https://github.com/jf647/btrfs-snap
>>
>> 2017-08-22 15:22 GMT+02:00 Ulli Horlacher :
>> > With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
>> > You can find these snapshots in every local directory (readonly).
>> > Example:
>> >
>> > framstag@fex:/sw/share: ll .snapshot/
>> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
>> > .snapshot/daily.2017-08-15_0010
>> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
>> > .snapshot/daily.2017-08-16_0010
>> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
>> > .snapshot/daily.2017-08-17_0010
>> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
>> > .snapshot/daily.2017-08-18_0010
>> > drwxr-xr-x  framstag root - 2017-08-18 23:59:29 
>> > .snapshot/daily.2017-08-19_0010
>> > drwxr-xr-x  framstag root - 2017-08-19 21:01:25 
>> > .snapshot/daily.2017-08-20_0010
>> > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
>> > .snapshot/daily.2017-08-21_0010
>> > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
>> > .snapshot/hourly.2017-08-20_1210
>> > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
>> > .snapshot/hourly.2017-08-20_1610
>> > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
>> > .snapshot/hourly.2017-08-20_2010
>> > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
>> > .snapshot/hourly.2017-08-21_0810
>> > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
>> > .snapshot/hourly.2017-08-21_1210
>> > drwxr-xr-x  framstag root - 2017-08-21 13:05:28 
>> > .snapshot/hourly.2017-08-21_1610
>
> btrfs-snap does not create local .snapshot/ sub-directories, but saves the
> snapshots in the toplevel root volume directory.
>
>
>
> --
> Ullrich Horlacher  Server und Virtualisierung
> Rechenzentrum TIK
> Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
> Allmandring 30aTel:++49-711-68565868
> 70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
> 

Re: netapp-alike snapshots?

[Ulli Horlacher]

On Tue 2017-08-22 (15:44), Peter Becker wrote:
> Is use: https://github.com/jf647/btrfs-snap
> 
> 2017-08-22 15:22 GMT+02:00 Ulli Horlacher :
> > With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> > You can find these snapshots in every local directory (readonly).
> > Example:
> >
> > framstag@fex:/sw/share: ll .snapshot/
> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > .snapshot/daily.2017-08-15_0010
> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > .snapshot/daily.2017-08-16_0010
> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > .snapshot/daily.2017-08-17_0010
> > drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> > .snapshot/daily.2017-08-18_0010
> > drwxr-xr-x  framstag root - 2017-08-18 23:59:29 
> > .snapshot/daily.2017-08-19_0010
> > drwxr-xr-x  framstag root - 2017-08-19 21:01:25 
> > .snapshot/daily.2017-08-20_0010
> > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> > .snapshot/daily.2017-08-21_0010
> > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> > .snapshot/hourly.2017-08-20_1210
> > drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> > .snapshot/hourly.2017-08-20_1610
> > drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> > .snapshot/hourly.2017-08-20_2010
> > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> > .snapshot/hourly.2017-08-21_0810
> > drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> > .snapshot/hourly.2017-08-21_1210
> > drwxr-xr-x  framstag root - 2017-08-21 13:05:28 
> > .snapshot/hourly.2017-08-21_1610

btrfs-snap does not create local .snapshot/ sub-directories, but saves the
snapshots in the toplevel root volume directory.



-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/

Re: netapp-alike snapshots?

[Peter Becker]

Is use: https://github.com/jf647/btrfs-snap

2017-08-22 15:22 GMT+02:00 Ulli Horlacher :
> With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
> You can find these snapshots in every local directory (readonly).
> Example:
>
> framstag@fex:/sw/share: ll .snapshot/
> drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> .snapshot/daily.2017-08-15_0010
> drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> .snapshot/daily.2017-08-16_0010
> drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> .snapshot/daily.2017-08-17_0010
> drwxr-xr-x  framstag root - 2017-08-14 10:21:47 
> .snapshot/daily.2017-08-18_0010
> drwxr-xr-x  framstag root - 2017-08-18 23:59:29 
> .snapshot/daily.2017-08-19_0010
> drwxr-xr-x  framstag root - 2017-08-19 21:01:25 
> .snapshot/daily.2017-08-20_0010
> drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> .snapshot/daily.2017-08-21_0010
> drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> .snapshot/hourly.2017-08-20_1210
> drwxr-xr-x  framstag root - 2017-08-20 02:50:18 
> .snapshot/hourly.2017-08-20_1610
> drwxr-xr-x  framstag root - 2017-08-20 19:48:40 
> .snapshot/hourly.2017-08-20_2010
> drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> .snapshot/hourly.2017-08-21_0810
> drwxr-xr-x  framstag root - 2017-08-21 00:42:28 
> .snapshot/hourly.2017-08-21_1210
> drwxr-xr-x  framstag root - 2017-08-21 13:05:28 
> .snapshot/hourly.2017-08-21_1610
>
> I would like to have something similar with btrfs.
> Programming such a feature is not a general problem for me, but I think I
> am not the first one who wants this kind of auto-snapshooting.
> Is there (where?) such a tool?
>
> I know snapper, but it has a totally different approach.
>
> --
> Ullrich Horlacher  Server und Virtualisierung
> Rechenzentrum TIK
> Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
> Allmandring 30aTel:++49-711-68565868
> 70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
> REF:<20170822132208.gd14...@rus.uni-stuttgart.de>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

netapp-alike snapshots?

[Ulli Horlacher]

With Netapp/waffle you have automatic hourly/daily/weekly snapshots.
You can find these snapshots in every local directory (readonly).
Example:

framstag@fex:/sw/share: ll .snapshot/
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-15_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-16_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-17_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-18_0010
drwxr-xr-x  framstag root - 2017-08-18 23:59:29 .snapshot/daily.2017-08-19_0010
drwxr-xr-x  framstag root - 2017-08-19 21:01:25 .snapshot/daily.2017-08-20_0010
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/daily.2017-08-21_0010
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1210
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1610
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/hourly.2017-08-20_2010
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_0810
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_1210
drwxr-xr-x  framstag root - 2017-08-21 13:05:28 .snapshot/hourly.2017-08-21_1610

I would like to have something similar with btrfs.
Programming such a feature is not a general problem for me, but I think I
am not the first one who wants this kind of auto-snapshooting.
Is there (where?) such a tool?

I know snapper, but it has a totally different approach.

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<20170822132208.gd14...@rus.uni-stuttgart.de>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html