Re: [gpfsug-discuss] Tracking deleted files

Mon, 27 Feb 2017 05:25:49 -0800

I would say the snapshot is not a light process. on heavy I/O and hundreds of nodes is pretty hard actually.
 
the quiescence needed to do the actual snapshot is not to be taken lightly on such conditions.
 
Furthermore the deletion of snapshots can be really fun too when it comes to how heavy it is.
 
For the rest of the points I wont argue, and is some cases this product might be the best way to go, just wanted to jump in about the snapshot part.

--
Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations

Luis Bolinches
Lab Services
http://www-03.ibm.com/systems/services/labservices/

IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland
Phone: +358 503112585

"If you continually give you will continually have." Anonymous
 
 
----- Original message -----
From: Jez Tucker <[email protected]>
Sent by: [email protected]
To: gpfsug main discussion list <[email protected]>
Cc:
Subject: Re: [gpfsug-discuss] Tracking deleted files
Date: Mon, Feb 27, 2017 3:12 PM
 
Hi
 
  Whilst it does use snapshots, I'd argue that snapshot creation is pretty lightweight - and always consistent. 
 
Your alternative via the mmbackup 'tracking' route is to parse out the mmbackup shadow file.  AFAIK to do this /properly in a timely fashion/ you'd need to do this as an inline post process after the scan phase of mmbackup has run, else you're instead looking at the outdated view of the shadow file post previous mmbackup run. 
 
mmbackup does not 'track' file changes, it performs a comparison pass between the filesystem contents and what TSM _believes_ is the known state of the file system during each run.  If a change is made oob of TSM then you need to re-generate the show file to regain total consistency.
 
Sensibly you should be running any mmbackup process from a snapshot to perform consistent backups without dsmc errors. 
 
So all things being equal, using snapshots for exact consistency and not having to regenerate (very heavyweight) or parse out a shadow file periodically is a lighter weight, smoother and reliably consistent workflow. 
 
YMMV with either approach depending on your management of TSM and your interpretation of 'consistent view' vs 'good enough'. 
 
Jez
 
On Mon, 27 Feb 2017 at 12:39, Simon Thompson (Research Computing - IT Services) <[email protected]> wrote:
Yeah but that uses snapshots, which is pretty heavy-weight for what I want to do, particularly given mmbackup seems to have a way of tracking deletes...
 
Simon
 
From: <[email protected]> on behalf of Jez Tucker <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Monday, 27 February 2017 at 11:59
To: "[email protected]" <[email protected]>
Subject: Re: [gpfsug-discuss] Tracking deleted files
 
 
Hi Simon
 
  I presented exactly this (albeit briefly) at the 2016 UG. 
 
See the snapdiff section of the presentation at:
 
 
We can track creations, modifications, deletions and moves (from, to) for files and directories between one point in time and another. 
 
The selections can be returned via a manner of your choice. 
 
If anyone wants to know more, hit me up directly. 
 
Incidentally - I will be at BVE this week (http://www.bvexpo.com/) showing new things driven by the Python API and GPFS - so if anyone is in the area and wants to chat about technicals in person rather than on mail, drop me a line and we can sort that out. 
 
Best,
 
Jez
 
 
On Mon, 27 Feb 2017 at 11:30, Simon Thompson (Research Computing - IT Services) <[email protected]> wrote:
Hi,

Is there a way to track files which have been deleted easily? I'm assuming
that we can't easily use a policy scan as they files are no longer in the
file-system unless we do some sort of diff?

I'm assuming there must be a way of doing this as mmbackup must track
deleted files to notify TSM of expired objects.

Basically I want a list of new files, changed files and deleted files
since a certain time. I'm assuming the first two will be relatively simple
with a policyscan, but the latter I'm not sure about.

Thanks

Simon

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
--

This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
--
 

This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
 

Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Reply via email to