Hi
Thanks very much everyone for this.
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
[1505984389175_twitter.png] @imperialRCS @imperialRSE
[1505983882959_Imperial-RCS.png]
________________________________
From: [email protected]
<[email protected]> on behalf of Hai Zhong HZ Zhou
<[email protected]>
Sent: 29 November 2019 01:55
To: IBM Spectrum Scale <[email protected]>
Cc: [email protected]
<[email protected]>; gpfsug main discussion list
<[email protected]>; Leo Luan <[email protected]>
Subject: Re: [gpfsug-discuss] Compression question
Caution - This email from [email protected] originated outside Imperial
The statement from Olaf and Alex in below emails are correct. Firstly,
compressing and decompressing files in active file system doesn't not trigger
data blocks copy-on-write, that is just deallocating the unnecessary original
data blocks and put compressed data in few data blocks, and no data blocks copy
to snapshot. Secondly, when reading the file from snapshot, it will be
redirected to active file system because there's no data blocks in snapshot,
and then doing decompression in-memory for snapshot read, while the data on
disk is still kept compressed.
Regards,
Hai Zhong Zhou
-----Huzefa H Pancha/India/IBM wrote: -----
To: Hai Zhong HZ Zhou/China/IBM@IBMCN
From: IBM Spectrum Scale/Poughkeepsie/IBM
Sent by: Huzefa H Pancha/India/IBM
Date: 11/29/2019 02:33AM
Cc:
[email protected]<mailto:[email protected]>,
gpfsug main discussion list
<[email protected]<mailto:[email protected]>>,
Leo Luan/Almaden/IBM@IBMUS
Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Hai Zhong,
Can you please help the customer with their compression related query.
Regards, The Spectrum Scale (GPFS) team
------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479.
If your query concerns a potential software error in Spectrum Scale (GPFS) and
you have an IBM software maintenance contract please contact 1-800-237-5511 in
the United States or your local IBM Service Center in other countries.
The forum is informally monitored as time permits and should not be used for
priority messages to the Spectrum Scale (GPFS) team.
[Inactive hide details for "Daniel Kidger" ---28-11-2019 20:58:12---Olaf's
explanation makes excellent sense. So is this defi]"Daniel Kidger"
---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is
this definitely the case? .. The now snapshot
From: "Daniel Kidger"
<[email protected]<mailto:[email protected]>>
To: [email protected]<mailto:[email protected]>
Cc: [email protected]<mailto:[email protected]>
Date: 28-11-2019 20:58
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by:
[email protected]<mailto:[email protected]>
________________________________
Olaf's explanation makes excellent sense.
So is this definitely the case? ..
The now snapshotted inode is not updated when a live file is compressed, as
it point to the current inode which itself has compressed blocks (and the
compressed flag set).
And likewise for deeper (older) snapshots.
sDaniel
_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
+44-(0)7818 522 266
[email protected]<mailto:[email protected]>
<https://www.youracclaim.com/badges/687cf790-fe65-4a92-b129-d23ae41862ac/public_url>
<https://www.youracclaim.com/badges/8153c6a7-3e02-40be-87ee-24e27ae9459c/public_url>
<https://www.youracclaim.com/badges/78197e2c-4277-4ec9-808b-ad6abe1e1b16/public_url>
----- Original message -----
From: "Olaf Weiser" <[email protected]<mailto:[email protected]>>
Sent by:
[email protected]<mailto:[email protected]>
To: gpfsug main discussion list
<[email protected]<mailto:[email protected]>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 15:01
Hi Alex,
not 100% sure about my answer.. but so far as I see it.. it is working, because
of the so called "dito resolution " .. In the snaphost's inode .. die pointer
to the DA's point the the next (more recent) inode information ..
so accessing a file in a snapshot- "redirects" the request to the origin inode
- and there ..the information about compression is given and points to the
origin DA
(of course.. only as long nobody changed the file since the snapshot was taken)
From: "Alexander Wolf"
<[email protected]<mailto:[email protected]>>
To:
[email protected]<mailto:[email protected]>
Date: 11/28/2019 07:03 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by:
[email protected]<mailto:[email protected]>
________________________________
I see the same behavior of mmlsattr on my system (with some post 5.0.4
development build). Funny enough if I look at the file content in the snapshot
it gets properly decompressed.
Mit freundlichen Grüßen / Kind regards
[cid:1574992361973]
[IBM Spectrum Scale]
Dr. Alexander Wolf-Reber
Spectrum Scale Release Lead Architect
Department M069 / Spectrum Scale Software Development
+49-160-90540880
[email protected]<mailto:[email protected]>
[cid:1574992361975]
IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/>
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats:
Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB
243294
----- Original message -----
From: "Luis Bolinches"
<[email protected]<mailto:[email protected]>>
Sent by:
[email protected]<mailto:[email protected]>
To: "gpfsug main discussion list"
<[email protected]<mailto:[email protected]>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 14:00
Which version are you running?
I was involved on a big for compressed file sets and snapshots that were
related to what you see.
--
Cheers
On 28. Nov 2019, at 14.57, Cregan, Bob
<[email protected]<mailto:[email protected]>> wrote:
Hi
Sounds logical - except the snap metadata does not have the compression
flag set. So if the inode now points to a set of compressed blocks how does the
client know to decompress it?
After compression of an existing file we get in the snap
-bash-4.2$ mmlsattr -L
.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf<mailto:.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf>
file name:
.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf<mailto:.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf>
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sata1
fileset name: userdirs
snapshot name: @GMT-2019.11.27-19.30.14
creation time: Tue Mar 5 16:16:40 2019
Misc attributes: ARCHIVE
Encrypted: no
and the original file is definitely compressed.
-bash-4.2$ mmlsattr -L UserGuide_13.06.pdf
file name: UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sata1
fileset name: userdirs
snapshot name:
creation time: Tue Mar 5 16:16:40 2019
Misc attributes: ARCHIVE COMPRESSION (library z)
Encrypted: no
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]<mailto:[email protected]>
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
<Outlook-1505984389.png>
@imperialRCS @imperialRSE
<Outlook-1505983882.png>
________________________________
From:
[email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
on behalf of Daniel Kidger
<[email protected]<mailto:[email protected]>>
Sent: 28 November 2019 12:30
To: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
Subject: Re: [gpfsug-discuss] Compression question
Caution - This email from
[email protected]<mailto:[email protected]> originated outside
Imperial
Alexander,
Can you then confirm then that the inodes in the snapshot will now point to
fewer but compressed blocks ?
Daniel
_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
+44-(0)7818 522 266
[email protected]<mailto:[email protected]>
----- Original message -----
From: "Alexander Wolf" <[email protected]<mailto:[email protected]>>
Sent by:
[email protected]<mailto:[email protected]>
To: [email protected]<mailto:[email protected]>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:21
I just tested this. Compressing a file did free up space in the file system.
Looks like our compression code does not trigger COW on the snapshot. You can
test this yourself by looking into mmlssnapshot -d (please not on a large
production fs, this command is expensive).
Mit freundlichen Grüßen / Kind regards
<Image.15749424191280.png>
<Image.15749424191281.png>
Dr. Alexander Wolf-Reber
Spectrum Scale Release Lead Architect
Department M069 / Spectrum Scale Software Development
+49-160-90540880
[email protected]<mailto:[email protected]>
<Image.15749424191282.png>
IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/>
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats:
Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB
243294
----- Original message -----
From: "Luis Bolinches"
<[email protected]<mailto:[email protected]>>
Sent by:
[email protected]<mailto:[email protected]>
To: [email protected]<mailto:[email protected]>
Cc: [email protected]<mailto:[email protected]>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 11:47
Hi
Same principle COW. The data blocks do not get modified.
--
Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations /
Salutacions
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches
"If you always give you will always have" -- Anonymous
----- Original message -----
From: "Cregan, Bob" <[email protected]<mailto:[email protected]>>
Sent by:
[email protected]<mailto:[email protected]>
To: gpfsug main discussion list
<[email protected]<mailto:[email protected]>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:06 PM
Just to clarify this is SS compression, so
mmchattr --compression yes <filename>
or an ILM equivalent
So not a regular modification.
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]<mailto:[email protected]>
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
<Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png>
@imperialRCS @imperialRSE
<Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png>
________________________________
From: Cregan, Bob
Sent: 28 November 2019 09:43
To: gpfsug main discussion list
<[email protected]<mailto:[email protected]>>
Subject: Compression question
Hi All,
Can someone answer the following question on compression in a
snapshot context? We have tried various experiments and they are inconclusive -
too tedious to go into the detail.
What happens to the snapshot when a file is compressed in SS? The logic as I
see it is
####### In a non compressed situation ###############
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old
blocks are then written to the snap.
######In a compressed situation ############
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is
compressed. So do the old uncompressed blocks go into the snap? If so we now
have 2 copies of the file and unless the compression > 50% we have used more
space until the snap is deleted.
You get the space back in the end, but if you are in a tight situation then
potentially compression might not work for you in the short term.
Thanks
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]<mailto:[email protected]>
W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>
<Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png>
@imperialRCS @imperialRSE
<Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
