Hai Zhong, Can you please help the customer with their compression related query.
Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Daniel Kidger" <[email protected]> To: [email protected] Cc: [email protected] Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: [email protected] Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 [email protected] ----- Original message ----- From: "Olaf Weiser" <[email protected]> Sent by: [email protected] To: gpfsug main discussion list <[email protected]> Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" <[email protected]> To: [email protected] Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: [email protected] I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Grüßen / Kind regards IBM Spectrum Scale Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 [email protected] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" <[email protected]> Sent by: [email protected] To: "gpfsug main discussion list" <[email protected]> Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob <[email protected]> wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: [email protected] W: www.imperial.ac.uk/ict/rcs <Outlook-1505984389.png> @imperialRCS @imperialRSE <Outlook-1505983882.png> From: [email protected] <[email protected]> on behalf of Daniel Kidger <[email protected]> Sent: 28 November 2019 12:30 To: [email protected] <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [gpfsug-discuss] Compression question |------------------------------------------------------------------------------| |Caution - This email from [email protected] originated outside Imperial| | | |------------------------------------------------------------------------------| Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 [email protected] ----- Original message ----- From: "Alexander Wolf" <[email protected]> Sent by: [email protected] To: [email protected] Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Grüßen / Kind regards <Image.15749424191280.png> <Image.15749424191281.png> Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 [email protected] <Image.15749424191282.png> IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" <[email protected]> Sent by: [email protected] To: [email protected] Cc: [email protected] Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" <[email protected]> Sent by: [email protected] To: gpfsug main discussion list <[email protected]> Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes <filename> or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: [email protected] W: www.imperial.ac.uk/ict/rcs <Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png> @imperialRCS @imperialRSE <Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png> From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list <[email protected]> Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: [email protected] W: www.imperial.ac.uk/ict/rcs <Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png> @imperialRCS @imperialRSE <Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png> _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edellä ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edellä ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=6F5tNsgC3a8tnTmVCVxGFJgNmWLUYm_MUO-zYDgeQ5o&s=F1aJcNbQxqosWD5WimhGS19MpeszOyIp9CxuUib-c2Q&e=
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
