Hai Zhong Zhou
-----Huzefa H Pancha/India/IBM wrote: -----
From: IBM Spectrum Scale/Poughkeepsie/IBM
Sent by: Huzefa H Pancha/India/IBM
Date: 11/29/2019 02:33AM
Cc: [email protected], gpfsug main discussion list <[email protected]>, Leo Luan/Almaden/IBM@IBMUS
Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Hai Zhong,
Can you please help the customer with their compression related query.
Regards, The Spectrum Scale (GPFS) team
------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479.
If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.
The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.
From: "Daniel Kidger" <[email protected]>
To: [email protected]
Cc: [email protected]
Date: 28-11-2019 20:58
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by: [email protected]
Olaf's explanation makes excellent sense.
So is this definitely the case? ..
The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set).
And likewise for deeper (older) snapshots.
sDaniel
_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
+44-(0)7818 522 266
[email protected]
- ----- Original message -----
From: "Olaf Weiser" <[email protected]>
Sent by: [email protected]
To: gpfsug main discussion list <[email protected]>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 15:01
Hi Alex,
not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information ..
so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA
(of course.. only as long nobody changed the file since the snapshot was taken)
From: "Alexander Wolf" <[email protected]>
To: [email protected]
Date: 11/28/2019 07:03 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by: [email protected]
I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed.
Mit freundlichen Grüßen / Kind regards
Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 [email protected] |
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
----- Original message -----
From: "Luis Bolinches" <[email protected]>
Sent by: [email protected]
To: "gpfsug main discussion list" <[email protected]>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 14:00
Which version are you running?
I was involved on a big for compressed file sets and snapshots that were related to what you see.
--
Cheers
On 28. Nov 2019, at 14.57, Cregan, Bob <[email protected]> wrote:
Hi
Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it?
After compression of an existing file we get in the snap
-bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sata1
fileset name: userdirs
snapshot name: @GMT-2019.11.27-19.30.14
creation time: Tue Mar 5 16:16:40 2019
Misc attributes: ARCHIVE
Encrypted: no
and the original file is definitely compressed.
-bash-4.2$ mmlsattr -L UserGuide_13.06.pdf
file name: UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sata1
fileset name: userdirs
snapshot name:
creation time: Tue Mar 5 16:16:40 2019
Misc attributes: ARCHIVE COMPRESSION (library z)
Encrypted: no
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]
W: www.imperial.ac.uk/ict/rcs
<Outlook-1505984389.png>
@imperialRCS @imperialRSE
<Outlook-1505983882.png>
From: [email protected] <[email protected]> on behalf of Daniel Kidger <[email protected]>
Sent: 28 November 2019 12:30
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [gpfsug-discuss] Compression question
|
Alexander,
Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?
Daniel
_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
+44-(0)7818 522 266
[email protected]
----- Original message -----
From: "Alexander Wolf" <[email protected]>
Sent by: [email protected]
To: [email protected]
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:21
I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).
Mit freundlichen Grüßen / Kind regards
<Image.15749424191280.png>
|
Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 [email protected] |
<Image.15749424191282.png>
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
----- Original message -----
From: "Luis Bolinches" <[email protected]>
Sent by: [email protected]
To: [email protected]
Cc: [email protected]
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 11:47
Hi
Same principle COW. The data blocks do not get modified.
--
Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches
"If you always give you will always have" -- Anonymous
----- Original message -----
From: "Cregan, Bob" <[email protected]>
Sent by: [email protected]
To: gpfsug main discussion list <[email protected]>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:06 PM
Just to clarify this is SS compression, so
mmchattr --compression yes <filename>
or an ILM equivalent
So not a regular modification.
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]
W: www.imperial.ac.uk/ict/rcs
<Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png>
@imperialRCS @imperialRSE
<Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png>
From: Cregan, Bob
Sent: 28 November 2019 09:43
To: gpfsug main discussion list <[email protected]>
Subject: Compression question
Hi All,
Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.
What happens to the snapshot when a file is compressed in SS? The logic as I see it is
####### In a non compressed situation ###############
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap.
######In a compressed situation ############
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted.
You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
Thanks
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: [email protected]
W: www.imperial.ac.uk/ict/rcs
<Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png>
@imperialRCS @imperialRSE
<Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
