Re: [gpfsug-discuss] Compression question

[Cregan, Bob]

Hi
 Thanks very much everyone for this.

Bob



Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cre...@imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>

[1505984389175_twitter.png] @imperialRCS @imperialRSE

[1505983882959_Imperial-RCS.png]



From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Hai Zhong HZ Zhou 

Sent: 29 November 2019 01:55
To: IBM Spectrum Scale 
Cc: gpfsug-discuss-boun...@spectrumscale.org 
; gpfsug main discussion list 
; Leo Luan 
Subject: Re: [gpfsug-discuss] Compression question


Caution - This email from zho...@cn.ibm.com originated outside Imperial



The statement from Olaf and Alex in below emails are correct. Firstly, 
compressing and decompressing files in active file system doesn't not trigger 
data blocks copy-on-write, that is just deallocating the unnecessary original 
data blocks and put compressed data in few data blocks, and no data blocks copy 
to snapshot. Secondly, when reading the file from snapshot, it will be 
redirected to active file system because there's no data blocks in snapshot, 
and then doing decompression in-memory for snapshot read, while the data on 
disk is still kept compressed.

Regards,
Hai Zhong Zhou


-Huzefa H Pancha/India/IBM wrote: -
To: Hai Zhong HZ Zhou/China/IBM@IBMCN
From: IBM Spectrum Scale/Poughkeepsie/IBM
Sent by: Huzefa H Pancha/India/IBM
Date: 11/29/2019 02:33AM
Cc: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>,
 gpfsug main discussion list 
mailto:gpfsug-discuss@spectrumscale.org>>, 
Leo Luan/Almaden/IBM@IBMUS
Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] Compression question

Hai Zhong,

Can you please help the customer with their compression related query.


Regards, The Spectrum Scale (GPFS) team

--
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=----0479.

If your query concerns a potential software error in Spectrum Scale (GPFS) and 
you have an IBM software maintenance contract please contact  1-800-237-5511 in 
the United States or your local IBM Service Center in other countries.

The forum is informally monitored as time permits and should not be used for 
priority messages to the Spectrum Scale (GPFS) team.

[Inactive hide details for "Daniel Kidger" ---28-11-2019 20:58:12---Olaf's 
explanation makes excellent sense.So is this defi]"Daniel Kidger" 
---28-11-2019 20:58:12---Olaf's explanation makes excellent sense.So is 
this definitely the case? .. The now snapshot

From: "Daniel Kidger" 
mailto:daniel.kid...@uk.ibm.com>>
To: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>
Cc: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>
Date: 28-11-2019 20:58
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>




Olaf's explanation makes excellent sense.

So is this definitely the case? ..
The now snapshotted inode is not updated when a live file is compressed, as 
it point to the current inode which itself has compressed blocks (and the 
compressed flag set).

And likewise for deeper (older) snapshots.
sDaniel

_
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store

+44-(0)7818 522 266
daniel.kid...@uk.ibm.com<mailto:daniel.kid...@uk.ibm.com>

<https://www.youracclaim.com/badges/687cf790-fe65-4a92-b129-d23ae41862ac/public_url>

<https://www.youracclaim.com/badges/8153c6a7-3e02-40be-87ee-24e27ae9459c/public_url>

<https://www.youracclaim.com/badges/78197e2c-4277-4ec9-808b-ad6abe1e1b16/public_url>





- Original message -
From: "Olaf Weiser" mailto:olaf.wei...@de.ibm.com>>
Sent by: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>
To: gpfsug main discussion list 
mailto:gpfsug-discuss@spectrumscale.org>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 15:01



Hi Alex,
not 100% sure about my answer.. but so far as I see it.. it is working, because 
of the so called "dito resolution " .. In the snaphost's inode .. die pointer 
to the DA's point the the next (more recent) inode information ..
so accessing a file in a snapshot- "redirects" the request 

Re: [gpfsug-discuss] Compression question

[Hai Zhong HZ Zhou]

c.uk> wrote:Hi       Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdffile name:            .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdfmetadata replication: 2 max 2data replication:     1 max 3immutable:            noappendOnly:           noflags:                storage pool name:    sata1fileset name:         userdirssnapshot name:        @GMT-2019.11.27-19.30.14creation time:        Tue Mar  5 16:16:40 2019Misc attributes:      ARCHIVEEncrypted:            no  and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name:            UserGuide_13.06.pdfmetadata replication: 2 max 2data replication:     1 max 3immutable:            noappendOnly:           noflags:                storage pool name:    sata1fileset name:         userdirssnapshot name:        creation time:        Tue Mar  5 16:16:40 2019Misc attributes:      ARCHIVE COMPRESSION (library z)Encrypted:            noBobBob CreganHPC Systems AnalystInformation & Communication TechnologiesImperial College London,South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.ukW: www.imperial.ac.uk/ict/rcs  @imperialRCS @imperialRSE            From: gpfsug-discuss-boun...@spectrumscale.org <gpfsug-discuss-boun...@spectrumscale.org> on behalf of Daniel Kidger <daniel.kid...@uk.ibm.com>Sent: 28 November 2019 12:30To: gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org>Cc: gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org>Subject: Re: [gpfsug-discuss] Compression question  Caution - This email from daniel.kid...@uk.ibm.com originated outside Imperial  Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?Daniel _Daniel KidgerIBM Technical Sales SpecialistSpectrum Scale, Spectrum NAS and IBM Cloud Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com   - Original message -From: "Alexander Wolf" <a.wolf-re...@de.ibm.com>Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Grüßen / Kind regards      Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.comIBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294  - Original message -From: "Luis Bolinches" <luis.bolinc...@fi.ibm.com>Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified.--Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / SalutacionsLuis BolinchesConsultant IT SpecialistMobile Phone: +358503112585https://www.youracclaim.com/user/luis-bolinches"If you always give you will always have" --  Anonymous   - Original message -From: "Cregan, Bob" <b.cre...@imperial.ac.uk>Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so  mmchattr --compression yes  or an ILM equivalent So not a regular modification. Bob   Bob CreganHPC Systems AnalystInformation & Communication TechnologiesImperial College London,South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.ukW: www.imperial.ac.uk/ict/rcs  @imperialRCS @imperialRSE          From: Cregan, BobSent: 28 November 2019 09:43To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>Subject: Compression question  Hi All,           Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is  #

Re: [gpfsug-discuss] Compression question

[IBM Spectrum Scale]

Hai Zhong,

Can you please help the customer with their compression related query.


Regards, The Spectrum Scale (GPFS) team

--

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=----0479.


If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.



From:   "Daniel Kidger" 
To: gpfsug-discuss@spectrumscale.org
Cc: gpfsug-discuss@spectrumscale.org
Date:   28-11-2019 20:58
Subject:[EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by:gpfsug-discuss-boun...@spectrumscale.org



Olaf's explanation makes excellent sense.

So is this definitely the case? ..
The now snapshotted inode is not updated when a live file is
compressed, as it point to the current inode which itself has compressed
blocks (and the compressed flag set).

And likewise for deeper (older) snapshots.
sDaniel

_
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store

+44-(0)7818 522 266
daniel.kid...@uk.ibm.com
 
 
 




 - Original message -
 From: "Olaf Weiser" 
 Sent by: gpfsug-discuss-boun...@spectrumscale.org
 To: gpfsug main discussion list 
 Cc:
 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
 Date: Thu, Nov 28, 2019 15:01



 Hi Alex,
 not 100% sure about my answer.. but so far as I see it.. it is working,
 because of the so called "dito resolution " .. In the snaphost's inode ..
 die pointer to the DA's point the the next (more recent) inode
 information ..
 so accessing a file in a snapshot- "redirects" the request to the origin
 inode - and there ..the information about compression is given and points
 to the origin DA

 (of course.. only as long nobody changed the file since the snapshot was
 taken)




 From:"Alexander Wolf" 
 To:gpfsug-discuss@spectrumscale.org
 Date:    11/28/2019 07:03 AM
 Subject:    [EXTERNAL] Re: [gpfsug-discuss] Compression question
 Sent by:gpfsug-discuss-boun...@spectrumscale.org


 I see the same behavior of mmlsattr on my system (with some post 5.0.4
 development build). Funny enough if I look at the file content in the
 snapshot it gets properly decompressed.

 Mit freundlichen Grüßen / Kind regards









  
 IBM Spectrum Scale Dr. Alexander Wolf-Reber
  
Spectrum Scale Release Lead Architect   
  
Department M069 / Spectrum Scale Software 
Development 

  
+49-160-90540880
  
a.wolf-re...@de.ibm.com 
  

  




 IBM Data Privacy Statement


 IBM Deutschland Research & Development GmbH / Vorsitzende des
 Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
 Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
 HRB 243294




 - Original message -
 From: "Luis Bolinches" 
 Sent by: gpfsug-discuss-boun...@spectrumscale.org
 To: "gpfsug main discussion list" 
 Cc:
 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
 Date: Thu, Nov 28, 2019 14:00

 Which version are you running?

 I was involved on a big for compressed file sets and snapshots that were
 related to what you see.

 --
 Cheers

 On 28. Nov 2019, at 14.57, Cregan, Bob  wrote:


 Hi
   Sounds logical - except the snap metadata does not have the
 compression flag set. So if the inode now points to a set of compressed
 blocks how does the client know to decompress it?

 After compression of an existing file we get in the snap

 -bash-4.2$ mmlsattr
 -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
 file
 name:.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
 metadata replication: 2 max 2
 data replication: 1 max 3
 immutable:no
 appendOnly:   no
 flags:
 

Re: [gpfsug-discuss] Compression question

[Daniel Kidger]

Olaf's explanation makes excellent sense. 
So is this definitely the case? ..
    The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). 
 
And likewise for deeper (older) snapshots.
sDaniel
 
_Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com

 
 
 
- Original message -From: "Olaf Weiser" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA(of course.. only as long nobody changed the file since the snapshot was taken)From:        "Alexander Wolf" To:        gpfsug-discuss@spectrumscale.orgDate:        11/28/2019 07:03 AMSubject:        [EXTERNAL] Re: [gpfsug-discuss] Compression questionSent by:        gpfsug-discuss-boun...@spectrumscale.org
I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Grüßen / Kind regards

 
      Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.comIBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
  - Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug main discussion list" Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 14:00Which version are you running?  I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers  On 28. Nov 2019, at 14.57, Cregan, Bob  wrote:Hi       Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdffile name:            .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdfmetadata replication: 2 max 2data replication:     1 max 3immutable:            noappendOnly:           noflags:                storage pool name:    sata1fileset name:         userdirssnapshot name:        @GMT-2019.11.27-19.30.14creation time:        Tue Mar  5 16:16:40 2019Misc attributes:      ARCHIVEEncrypted:            no
  and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name:            UserGuide_13.06.pdfmetadata replication: 2 max 2data replication:     1 max 3immutable:            noappendOnly:           noflags:                storage pool name:    sata1fileset name:         userdirssnapshot name:        creation time:        Tue Mar  5 16:16:40 2019Misc attributes:      ARCHIVE COMPRESSION (library z)Encrypted:            no
BobBob CreganHPC Systems AnalystInformation & Communication TechnologiesImperial College London,South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.ukW: www.imperial.ac.uk/ict/rcs 
 @imperialRCS @imperialRSE 
  
  
  
    
 
From: gpfsug-discuss-boun...@spectrumscale.org  on behalf of Daniel Kidger Sent: 28 November 2019 12:30To: gpfsug-discuss@spectrumscale.org Cc: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question 
Caution - This email from daniel.kid...@uk.ibm.com originated outside Imperial 
 
Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?Daniel _Daniel KidgerIBM Technical Sales Speciali

Re: [gpfsug-discuss] Compression question

[Olaf Weiser]

Hi Alex, not 100% sure about my answer.. but
so far as I see it.. it is working, because of the so called "dito
resolution " .. In the snaphost's inode .. die pointer to the DA's
point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects"
the request to the origin inode - and there ..the information about compression
is given and points to the origin DA(of course.. only as long nobody changed
the file since the snapshot was taken)From:      
 "Alexander Wolf"
To:      
 gpfsug-discuss@spectrumscale.orgDate:      
 11/28/2019 07:03 AMSubject:    
   [EXTERNAL] Re:
[gpfsug-discuss] Compression questionSent by:    
   gpfsug-discuss-boun...@spectrumscale.orgI see the same behavior of mmlsattr on my
system (with some post 5.0.4 development build). Funny enough if I look
at the file content in the snapshot it gets properly decompressed. Mit freundlichen Grüßen / Kind regards  
   Dr.
Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.comIBM
Data Privacy StatementIBM Deutschland Research &
Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung:
Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294  - Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug main discussion list" Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 14:00 Which version are you running?  I was involved on a big for compressed file
sets and snapshots that were related to what you see.   -- Cheers  On 28. Nov 2019, at 14.57, Cregan, Bob 
wrote:  Hi       Sounds logical - except
the snap metadata does not have the compression flag set. So if the inode
now points to a set of compressed blocks how does the client know to decompress
it? After compression of an existing file we
get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdffile name:        
   .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdfmetadata replication: 2 max 2data replication:     1 max 3immutable:        
   noappendOnly:        
  noflags:          
     storage pool name:    sata1fileset name:        
userdirssnapshot name:        @GMT-2019.11.27-19.30.14creation time:        Tue
Mar  5 16:16:40 2019Misc attributes:      ARCHIVEEncrypted:        
   no  and the original file is definitely
compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf
file name:        
   UserGuide_13.06.pdfmetadata replication: 2 max 2data replication:     1 max 3immutable:        
   noappendOnly:        
  noflags:          
     storage pool name:    sata1fileset name:        
userdirssnapshot name:        creation time:        Tue
Mar  5 16:16:40 2019Misc attributes:      ARCHIVE
COMPRESSION (library z)Encrypted:        
   noBobBob CreganHPC Systems AnalystInformation & Communication TechnologiesImperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.ukW: www.imperial.ac.uk/ict/rcs  @imperialRCS
@imperialRSE            From: gpfsug-discuss-boun...@spectrumscale.org
 on behalf of Daniel Kidger
Sent: 28 November 2019 12:30To: gpfsug-discuss@spectrumscale.org Cc: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution
- This email from daniel.kid...@uk.ibm.com
originated outside Imperial  Alexander, Can you then confirm then that the inodes
in the snapshot will now point to fewer but compressed blocks ?Daniel _Daniel KidgerIBM Technical Sales SpecialistSpectrum Scale, Spectrum NAS and IBM Cloud
Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com    - Original message -From: "Alexander Wolf" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:21  I just tested this. Compressing a file did
free up space in the file system. Looks like our compression code does
not trigger COW on the snapshot. You can test this yourself by looking
into mmlssnapshot -d (please not on a large production fs, this command
is expensive). Mit freundlichen Grüßen / Kind regards
   Dr.
Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.comIBM
Data Privacy StatementIBM Deutschland Research &
Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung:
Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294  - Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gp

Re: [gpfsug-discuss] Compression question

[Luis Bolinches]

That is the fix. Before it was an error. 


So blocks that are pointed from active get compressed. Ones that are no longer 
linked are not modified. 

--
Cheers

> On 28. Nov 2019, at 16.03, Alexander Wolf  wrote:
> 
> 
> I see the same behavior of mmlsattr on my system (with some post 5.0.4 
> development build). Funny enough if I look at the file content in the 
> snapshot it gets properly decompressed.
>  
> Mit freundlichen Grüßen / Kind regards
> 
> 
> 
>  
>  
> Dr. Alexander Wolf-Reber
> Spectrum Scale Release Lead Architect
> Department M069 / Spectrum Scale Software Development
> 
> +49-160-90540880
> a.wolf-re...@de.ibm.com
> 
> IBM Data Privacy Statement
> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: 
> Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, 
> HRB 243294
> 
>  
>  
> - Original message -
> From: "Luis Bolinches" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: "gpfsug main discussion list" 
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
> Date: Thu, Nov 28, 2019 14:00
>  
> Which version are you running?
>  
> I was involved on a big for compressed file sets and snapshots that were 
> related to what you see. 
>  
> --
> Cheers
>  
>> 
>>> On 28. Nov 2019, at 14.57, Cregan, Bob  wrote:
>>>  
>> 
>> Hi 
>>   Sounds logical - except the snap metadata does not have the 
>> compression flag set. So if the inode now points to a set of compressed 
>> blocks how does the client know to decompress it?
>>  
>> After compression of an existing file we get in the snap
>>  
>> -bash-4.2$ mmlsattr -L 
>> .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
>> file name:.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
>> metadata replication: 2 max 2
>> data replication: 1 max 3
>> immutable:no
>> appendOnly:   no
>> flags:
>> storage pool name:sata1
>> fileset name: userdirs
>> snapshot name:@GMT-2019.11.27-19.30.14
>> creation time:Tue Mar  5 16:16:40 2019
>> Misc attributes:  ARCHIVE
>> Encrypted:no
>>  
>>  and the original file is definitely compressed.
>>  
>> -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf 
>> file name:UserGuide_13.06.pdf
>> metadata replication: 2 max 2
>> data replication: 1 max 3
>> immutable:no
>> appendOnly:   no
>> flags:
>> storage pool name:sata1
>> fileset name: userdirs
>> snapshot name:
>> creation time:Tue Mar  5 16:16:40 2019
>> Misc attributes:  ARCHIVE COMPRESSION (library z)
>> Encrypted:no
>> Bob
>>  
>>  
>>  
>>  
>> Bob Cregan
>> HPC Systems Analyst
>> Information & Communication Technologies
>> Imperial College London, 
>> South Kensington Campus London, SW7 2AZ
>> T: 07712388129
>> E: b.cre...@imperial.ac.uk
>> W: www.imperial.ac.uk/ict/rcs
>>  
>> 
>>  @imperialRCS @imperialRSE
>>  
>> 
>>  
>> 
>>  
>>  
>> 
>>  
>>  
>>  
>>  
>>  
>> From: gpfsug-discuss-boun...@spectrumscale.org 
>>  on behalf of Daniel Kidger 
>> 
>> Sent: 28 November 2019 12:30
>> To: gpfsug-discuss@spectrumscale.org 
>> Cc: gpfsug-discuss@spectrumscale.org 
>> Subject: Re: [gpfsug-discuss] Compression question
>>  
>> Caution - This email from daniel.kid...@uk.ibm.com originated outside 
>> Imperial
>>  
>> 
>> Alexander,
>>  
>> Can you then confirm then that the inodes in the snapshot will now point to 
>> fewer but compressed blocks ?
>> Daniel
>>  
>> _
>> Daniel Kidger
>> IBM Technical Sales Specialist
>> Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
>> 
>> +44-(0)7818 522 266 
>> daniel.kid...@uk.ibm.com
>>  
>>  
>>  
>>  
>> - Original message -
>> From: "Alexander Wolf" 
>> Sent by: gpfsug-discuss-boun...@spectrumscale.org
>> To: gpfsug-discuss@spectrumscale.org
>> Cc:
>> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
>> Date: Thu, Nov 28, 2019 12:21
>>  
>> I just tested this. Compressing a file did free up sp

Re: [gpfsug-discuss] Compression question

[Alexander Wolf]

I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed.
 
Mit freundlichen Grüßen / Kind regards  
   
  
        Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.com
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294  
 
 
- Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug main discussion list" Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 14:00 Which version are you running?
 
I was involved on a big for compressed file sets and snapshots that were related to what you see.  
--
Cheers
 
On 28. Nov 2019, at 14.57, Cregan, Bob  wrote: 
 
Hi 
      Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it?
 
After compression of an existing file we get in the snap
 
-bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
file name:            .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication:     1 max 3
immutable:            no
appendOnly:           no
flags:                
storage pool name:    sata1
fileset name:         userdirs
snapshot name:        @GMT-2019.11.27-19.30.14
creation time:        Tue Mar  5 16:16:40 2019
Misc attributes:      ARCHIVE
Encrypted:            no
  and the original file is definitely compressed.
 
-bash-4.2$ mmlsattr -L UserGuide_13.06.pdf 
file name:            UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication:     1 max 3
immutable:            no
appendOnly:           no
flags:                
storage pool name:    sata1
fileset name:         userdirs
snapshot name:        
creation time:        Tue Mar  5 16:16:40 2019
Misc attributes:      ARCHIVE COMPRESSION (library z)
Encrypted:            no
Bob
 
 
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 
 @imperialRCS @imperialRSE
 
 
 

 
  

 
 
 
From: gpfsug-discuss-boun...@spectrumscale.org  on behalf of Daniel Kidger Sent: 28 November 2019 12:30To: gpfsug-discuss@spectrumscale.org Cc: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question
 
Caution - This email from daniel.kid...@uk.ibm.com originated outside Imperial 
 
Alexander,
 
Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?
Daniel
 
_Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com

 
 
 
- Original message -From: "Alexander Wolf" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:21 
I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).
 
Mit freundlichen Grüßen / Kind regards
 

 

         

Re: [gpfsug-discuss] Compression question

[Cregan, Bob]

Hi
 5.0.2 with a hotfix for a lz4 compression bug

Bob



Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cre...@imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>

[1505984389175_twitter.png] @imperialRCS @imperialRSE

[1505983882959_Imperial-RCS.png]



From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Luis Bolinches 

Sent: 28 November 2019 13:00
To: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] Compression question


Caution - This email from luis.bolinc...@fi.ibm.com originated outside Imperial



Which version are you running?

I was involved on a big for compressed file sets and snapshots that were 
related to what you see.

--
Cheers

On 28. Nov 2019, at 14.57, Cregan, Bob  wrote:


Hi
  Sounds logical - except the snap metadata does not have the compression 
flag set. So if the inode now points to a set of compressed blocks how does the 
client know to decompress it?

After compression of an existing file we get in the snap


-bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf

file name:.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf

metadata replication: 2 max 2

data replication: 1 max 3

immutable:no

appendOnly:   no

flags:

storage pool name:sata1

fileset name: userdirs

snapshot name:@GMT-2019.11.27-19.30.14

creation time:Tue Mar  5 16:16:40 2019

Misc attributes:  ARCHIVE

Encrypted:no

 and the original file is definitely compressed.


-bash-4.2$ mmlsattr -L UserGuide_13.06.pdf

file name:UserGuide_13.06.pdf

metadata replication: 2 max 2

data replication: 1 max 3

immutable:no

appendOnly:   no

flags:

storage pool name:sata1

fileset name: userdirs

snapshot name:

creation time:Tue Mar  5 16:16:40 2019

Misc attributes:  ARCHIVE COMPRESSION (library z)

Encrypted:no

Bob




Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cre...@imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>


 @imperialRCS @imperialRSE






From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Daniel Kidger 

Sent: 28 November 2019 12:30
To: gpfsug-discuss@spectrumscale.org 
Cc: gpfsug-discuss@spectrumscale.org 
Subject: Re: [gpfsug-discuss] Compression question


Caution - This email from daniel.kid...@uk.ibm.com originated outside Imperial



Alexander,

Can you then confirm then that the inodes in the snapshot will now point to 
fewer but compressed blocks ?
Daniel

_
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store

+44-(0)7818 522 266
daniel.kid...@uk.ibm.com
[https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png]<https://www.youracclaim.com/badges/687cf790-fe65-4a92-b129-d23ae41862ac/public_url>
   
[https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png]
 
<https://www.youracclaim.com/badges/8153c6a7-3e02-40be-87ee-24e27ae9459c/public_url>
   
[https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png]
 
<https://www.youracclaim.com/badges/78197e2c-4277-4ec9-808b-ad6abe1e1b16/public_url>



- Original message -
From: "Alexander Wolf" 
Sent by: gpfsug-discuss-boun...@spectrumscale.org
To: gpfsug-discuss@spectrumscale.org
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:21

I just tested this. Compressing a file did free up space in the file system. 
Looks like our compression code does not trigger COW on the snapshot. You can 
test this yourself by looking into mmlssnapshot -d (please not on a large 
production fs, this command is expensive).

Mit freundlichen Grüßen / Kind regards





  *

  *

  *   Dr. Alexander Wolf-Reber
Spectrum Scale Release Lead Architect
Department M069 / Spectrum Scale Software Development

+49-160-90540880
a.wolf-re...@de.ibm.com


IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/>

IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: 
Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 
243294



- Original message -
From: "Luis Bolinches" 
Sent by: gpfsug-discuss-boun...@spectrumscale.org
To: gpfsug-discuss@spectrumscale.org
Cc: gpfsug-discuss@spectrumscale.org
Subject: [EXTERNAL] Re: [gpfsug-d

Re: [gpfsug-discuss] Compression question

[Luis Bolinches]

Which version are you running?

I was involved on a big for compressed file sets and snapshots that were 
related to what you see. 

--
Cheers

> On 28. Nov 2019, at 14.57, Cregan, Bob  wrote:
> 
> 
> Hi 
>   Sounds logical - except the snap metadata does not have the compression 
> flag set. So if the inode now points to a set of compressed blocks how does 
> the client know to decompress it?
> 
> After compression of an existing file we get in the snap
> 
> -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
> file name:.snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
> metadata replication: 2 max 2
> data replication: 1 max 3
> immutable:no
> appendOnly:   no
> flags:
> storage pool name:sata1
> fileset name: userdirs
> snapshot name:@GMT-2019.11.27-19.30.14
> creation time:Tue Mar  5 16:16:40 2019
> Misc attributes:  ARCHIVE
> Encrypted:no
> 
>  and the original file is definitely compressed.
> 
> -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf 
> file name:UserGuide_13.06.pdf
> metadata replication: 2 max 2
> data replication: 1 max 3
> immutable:no
> appendOnly:   no
> flags:
> storage pool name:sata1
> fileset name: userdirs
> snapshot name:
> creation time:Tue Mar  5 16:16:40 2019
> Misc attributes:  ARCHIVE COMPRESSION (library z)
> Encrypted:no
> 
> Bob
> 
> 
> 
> Bob Cregan
> HPC Systems Analyst
> Information & Communication Technologies
> Imperial College London, 
> South Kensington Campus London, SW7 2AZ
> T: 07712388129
> E: b.cre...@imperial.ac.uk
> W: www.imperial.ac.uk/ict/rcs
> 
>  @imperialRCS @imperialRSE
> 
> 
> 
> From: gpfsug-discuss-boun...@spectrumscale.org 
>  on behalf of Daniel Kidger 
> 
> Sent: 28 November 2019 12:30
> To: gpfsug-discuss@spectrumscale.org 
> Cc: gpfsug-discuss@spectrumscale.org 
> Subject: Re: [gpfsug-discuss] Compression question
>  
> Caution - This email from daniel.kid...@uk.ibm.com originated outside Imperial
>  
> 
> Alexander,
>  
> Can you then confirm then that the inodes in the snapshot will now point to 
> fewer but compressed blocks ?
> Daniel
>  
> _
> Daniel Kidger
> IBM Technical Sales Specialist
> Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
> 
> +44-(0)7818 522 266 
> daniel.kid...@uk.ibm.com
>   
>  
>  
>  
> - Original message -
> From: "Alexander Wolf" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: gpfsug-discuss@spectrumscale.org
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
> Date: Thu, Nov 28, 2019 12:21
>  
> I just tested this. Compressing a file did free up space in the file system. 
> Looks like our compression code does not trigger COW on the snapshot. You can 
> test this yourself by looking into mmlssnapshot -d (please not on a large 
> production fs, this command is expensive).
>  
> Mit freundlichen Grüßen / Kind regards
> 
> 
> 
>  
>  
> Dr. Alexander Wolf-Reber
> Spectrum Scale Release Lead Architect
> Department M069 / Spectrum Scale Software Development
> 
> +49-160-90540880
> a.wolf-re...@de.ibm.com
> 
> IBM Data Privacy Statement
> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: 
> Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, 
> HRB 243294
> 
>  
>  
> - Original message -
> From: "Luis Bolinches" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: gpfsug-discuss@spectrumscale.org
> Cc: gpfsug-discuss@spectrumscale.org
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
> Date: Thu, Nov 28, 2019 11:47
>  
> Hi
>  
> Same principle COW. The data blocks do not get modified.
> --
> Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / 
> Salutacions
> Luis Bolinches
> Consultant IT Specialist
> Mobile Phone: +358503112585
> https://www.youracclaim.com/user/luis-bolinches
> 
> "If you always give you will always have" --  Anonymous
>  
>  
>  
> - Original message -
> From: "Cregan, Bob" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: gpfsug main discussion list 
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
> Date: Thu, Nov 28, 2019 12:06 PM
>  
> Just to clarify this is SS compression,

Re: [gpfsug-discuss] Compression question

[Alexander Wolf]

Actually I am not aware of an easy way to do this. But to back my statement: I had created a file of 1G. Made a snapshot "snap1". Wrote new data over the whole file. Created "snap2". Compressed file. This is what stat says:
 
[root@alex-11 ~]# stat /mnt/gpfs0/testfile   File: /mnt/gpfs0/testfile  Size: 1073741824    Blocks: 209920     IO Block: 262144 regular fileDevice: 2dh/45d    Inode: 10432       Links: 1[root@alex-11 ~]# stat /mnt/gpfs0/.snapshots/snap1/testfile   File: /mnt/gpfs0/.snapshots/snap1/testfile  Size: 1073741824    Blocks: 2097152    IO Block: 262144 regular fileDevice: 2dh/45d    Inode: 10432       Links: 1
 
[root@alex-11 ~]# stat /mnt/gpfs0/.snapshots/snap2/testfile   File: /mnt/gpfs0/.snapshots/snap2/testfile  Size: 1073741824    Blocks: 1          IO Block: 262144 regular fileDevice: 2dh/45d    Inode: 10432       Links: 1 
As expected in snap1 the number of blocks corresponds with the file size. In snap2 we have just a single block. And the "active" file is about 10% of the blocks you would normally expect (looks like my test data compresses nicely).
 
Mit freundlichen Grüßen / Kind regards  
   
  
        Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.com
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294  
 
 
- Original message -From: "Daniel Kidger" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 13:30 
Alexander,
 
Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?
Daniel
 
_Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com

 
 
 
- Original message -From: "Alexander Wolf" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:21 
I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).
 
Mit freundlichen Grüßen / Kind regards  
   
  
        Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.com
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294  
 
 
- Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 11:47 
Hi
 
Same principle COW. The data blocks do not get modified.
--Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / SalutacionsLuis Bolinches
Consultant IT SpecialistMobile Phone: +358503112585https://www.youracclaim.com/user/luis-bolinches"If you always give you will always have" --  

Re: [gpfsug-discuss] Compression question

[Daniel Kidger]

Alexander,
 
Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?
Daniel
 
_Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com

 
 
 
- Original message -From: "Alexander Wolf" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:21 
I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).
 
Mit freundlichen Grüßen / Kind regards  
   
  
        Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.com
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294  
 
 
- Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 11:47 
Hi
 
Same principle COW. The data blocks do not get modified.
--Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / SalutacionsLuis Bolinches
Consultant IT SpecialistMobile Phone: +358503112585https://www.youracclaim.com/user/luis-bolinches"If you always give you will always have" --  Anonymous
 
 
 
- Original message -From: "Cregan, Bob" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:06 PM 
Just to clarify this is SS compression, so 
 
mmchattr --compression yes 
 
or an ILM equivalent
 
So not a regular modification.
 
Bob
 
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
 
From: Cregan, BobSent: 28 November 2019 09:43To: gpfsug main discussion list Subject: Compression question
 
Hi All,
           Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.
 
What happens to the snapshot when a file is compressed in SS? The logic as I see it is 
 
### In a non compressed situation ###
 
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. 
 
##In a compressed situation 
 
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50%  we have used more space until the snap is deleted.
 
You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
Thanks
 
Bob
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 Ellei edellä ole toisin mainittu: / Unless stated otherwise above:Oy IBM Finland AbPL 265, 00101 Helsinki, FinlandBusiness ID, Y-tunnus: 0195876-3Registered in Finland 
___gpfsug-discuss mailing 

Re: [gpfsug-discuss] Compression question

[Alexander Wolf]

I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).
 
Mit freundlichen Grüßen / Kind regards  
   
  
        Dr. Alexander Wolf-ReberSpectrum Scale Release Lead ArchitectDepartment M069 / Spectrum Scale Software Development+49-160-90540880a.wolf-re...@de.ibm.com
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294  
 
 
- Original message -From: "Luis Bolinches" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 11:47 
Hi
 
Same principle COW. The data blocks do not get modified.
--Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / SalutacionsLuis Bolinches
Consultant IT SpecialistMobile Phone: +358503112585https://www.youracclaim.com/user/luis-bolinches"If you always give you will always have" --  Anonymous
 
 
 
- Original message -From: "Cregan, Bob" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:06 PM 
Just to clarify this is SS compression, so 
 
mmchattr --compression yes 
 
or an ILM equivalent
 
So not a regular modification.
 
Bob
 
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
 
From: Cregan, BobSent: 28 November 2019 09:43To: gpfsug main discussion list Subject: Compression question
 
Hi All,
           Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.
 
What happens to the snapshot when a file is compressed in SS? The logic as I see it is 
 
### In a non compressed situation ###
 
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. 
 
##In a compressed situation 
 
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50%  we have used more space until the snap is deleted.
 
You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
Thanks
 
Bob
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 Ellei edellä ole toisin mainittu: / Unless stated otherwise above:Oy IBM Finland AbPL 265, 00101 Helsinki, FinlandBusiness ID, Y-tunnus: 0195876-3Registered in Finland 
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Compression question

[Luis Bolinches]

Hi
 
Same principle COW. The data blocks do not get modified.
--Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / SalutacionsLuis Bolinches
Consultant IT SpecialistMobile Phone: +358503112585https://www.youracclaim.com/user/luis-bolinches"If you always give you will always have" --  Anonymous
 
 
 
- Original message -From: "Cregan, Bob" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 12:06 PM 
Just to clarify this is SS compression, so 
 
mmchattr --compression yes 
 
or an ILM equivalent
 
So not a regular modification.
 
Bob
 
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
 
From: Cregan, BobSent: 28 November 2019 09:43To: gpfsug main discussion list Subject: Compression question
 
Hi All,
           Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.
 
What happens to the snapshot when a file is compressed in SS? The logic as I see it is 
 
### In a non compressed situation ###
 
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. 
 
##In a compressed situation 
 
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50%  we have used more space until the snap is deleted.
 
You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
Thanks
 
Bob
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 

Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Compression question

[Cregan, Bob]

Just to clarify this is SS compression, so

mmchattr --compression yes 

or an ILM equivalent

So not a regular modification.

Bob



Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cre...@imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs

[1505984389175_twitter.png] @imperialRCS @imperialRSE

[1505983882959_Imperial-RCS.png]



From: Cregan, Bob
Sent: 28 November 2019 09:43
To: gpfsug main discussion list 
Subject: Compression question

Hi All,
   Can someone answer the following question on compression in a 
snapshot context? We have tried various experiments and they are inconclusive - 
too tedious to go into the detail.

What happens to the snapshot when a file is compressed in SS? The logic as I 
see it is

### In a non compressed situation ###

1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old 
blocks are then written to the snap.


##In a compressed situation 


1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is 
compressed. So do the old uncompressed blocks go into the snap? If so we now 
have 2 copies of the file and unless the compression > 50%  we have used more 
space until the snap is deleted.

You get the space back in the end, but if you are in a tight situation then 
potentially compression might not work for you in the short term.

Thanks

Bob


Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cre...@imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs

[1505984389175_twitter.png] @imperialRCS @imperialRSE

[1505983882959_Imperial-RCS.png]

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Compression question

[Luis Bolinches]

Hi
 
snaps are copy on write (COW). No blocks are written to snap compressed or not compressed situation
 
###
Create a file
Take snap
Modify file, only the blocks that are modified are linked from the active i-nodes, snap i-nodes point to the original blocks. you only free space when a data block has no snaps (and active) pointers from i-nodes
###
 
That is the same if data blocks are compressed or not
 
Hope that helps
--Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / SalutacionsLuis Bolinches
Consultant IT SpecialistMobile Phone: +358503112585https://www.youracclaim.com/user/luis-bolinches"If you always give you will always have" --  Anonymous
 
 
 
- Original message -From: "Cregan, Bob" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] [gpfsug-discuss] Compression questionDate: Thu, Nov 28, 2019 11:44 AM 
Hi All,
           Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.
 
What happens to the snapshot when a file is compressed in SS? The logic as I see it is 
 
### In a non compressed situation ###
 
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. 
 
##In a compressed situation 
 
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50%  we have used more space until the snap is deleted.
 
You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
Thanks
 
Bob
 
 
Bob CreganHPC Systems Analyst
Information & Communication Technologies
Imperial College London, South Kensington Campus London, SW7 2AZT: 07712388129E: b.cre...@imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
 @imperialRCS @imperialRSE

  

 
 
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 

Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss