Re: [PR] KVM incremental snapshot feature [cloudstack]

[via GitHub]

JoaoJandre commented on PR #9270:
URL: https://github.com/apache/cloudstack/pull/9270#issuecomment-2754366861

   > > > Noticed incremental snapshots taking more physical size as earlier 
(you mentioned here, these are the extra bytes to store the metadata, but it 
varies - please check/confirm).
   > > 
   > > 
   > > > A file added with 300MB for each snapshot taken
   > > 
   > > 
   > > @sureshanaparti I'm not @JoaoJandre, but: your 4 first full snapshots 
(taken with `kvm.incremental.snapshot` set to `false`) are all using 1.6 GB, 
even though you are adding 300 MB every time you take a new snapshot. This 
means that the files you created are still cached; they were not written to 
disk yet. Therefore, the snapshots you took do not contain these files. You 
need to execute `sync` before taking a snapshot so that the cached data is 
written to disk.
   > > If you consider how many times you added 300 MB before taking the first 
full snapshot with `kvm.incremental.snapshot` set to `true` (4 times), you'll 
see that you created a total of 1.2 GB. This matches the size of the first full 
snapshot with incremental snapshots enabled (1.6 GB originally + 1.2 GB created 
= 2.8 GB of the first full snapshot with incremental snapshots enabled).
   > > Could you repeat your tests executing a `sync` before taking the 
snapshots?
   > 
   > @winterhazel @JoaoJandre I've repeated these tests ( with `sync` cmd 
execution before taking the snapshots), still there is difference in physical 
size for normal/full and incremental snapshots. [A file added with 300MB and 
executed `sync` cmd before each snapshot taken. Incremental snapshots with incr 
name are taken with kvm.incremental.snapshot true, and snapshot.delta.max set 
to 3.] Maybe, the disk-only based snapshots introduced here: #5297, applies 
some compression for the full snapshots taken (@GutoVeronezi can you confirm?)
   > 
   > Snapshot records starting with full snapshots first =>
   > <img alt="full-and-incr-snapshot-records-starting-with-full-snapshots" 
width="1160" 
src="https://private-user-images.githubusercontent.com/12028987/426890397-0617ae06-413f-495a-8861-8db9f0adc951.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDI5ODk0MzcsIm5iZiI6MTc0Mjk4OTEzNywicGF0aCI6Ii8xMjAyODk4Ny80MjY4OTAzOTctMDYxN2FlMDYtNDEzZi00OTVhLTg4NjEtOGRiOWYwYWRjOTUxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzI2VDExMzg1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJlYzc1NjM1M2E4MTljNWJjNDFlNmMyN2FiN2IzMmM5NDI2MWU5MWU1MDlhMzk3NTVmMGU4MWEyOTg2Y2Y1YzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.sfVU0hnMFbqrPbIZ41tWPX1MaLzUMqg-t1u30N6chlc";>
   > 
   > Snapshot records starting with increment snapshots first =>
   > <img alt="full-and-incr-snapshot-records-starting-with-incr-snapshots" 
width="1159" 
src="https://private-user-images.githubusercontent.com/12028987/426890565-e33980ab-30b6-4a15-b2de-239a98a3e6a3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDI5ODk0MzcsIm5iZiI6MTc0Mjk4OTEzNywicGF0aCI6Ii8xMjAyODk4Ny80MjY4OTA1NjUtZTMzOTgwYWItMzBiNi00YTE1LWIyZGUtMjM5YTk4YTNlNmEzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzI2VDExMzg1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThhMGUyNmFhN2RjOTBhZTI4ZmExZWJmYjYwMmRiMjkwNjhlNTllMzA4NTM4YWE2NmMyZTM1MTE1MTgyMjU3OWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.07X8hV86rqbqo4S0aV3mwRi0WlCZ__QmoGYcHoLYRus";>
   
   Hello, @sureshanaparti 
   
   Thank you for testing. The only way I found to reproduce your results, is if 
you're writing zeroed files to your volume (with `dd if=/dev/zero` for 
example), if this is the case, when converting the snapshot (a step done in the 
full snap process but not in the incremental), qemu will find the zeroed 
sectors and will not copy them (see 
https://qemu-project.gitlab.io/qemu/tools/qemu-img.html#cmdoption-qemu-img-commands-arg-convert
 for more information), this is why the full snapshots do not grow in size when 
writing zeroed files. **However**, this is an unrealistic test, as I expect 
that most people will not be creating many zeroed files to backup.
   
   A more realistic scenario would be writing files with random data:
   
   1. Create a VM, attach a volume to it and format it:
   
![2025-03-26_09-44](https://github.com/user-attachments/assets/100c15cd-5512-4b58-bf9e-ee0c1e295cd7)
   2. Take a snapshot;
   3. Write a file to the volume and sync it;
   4. Repeat steps 2 and 3.
   
![2025-03-26_09-51](https://github.com/user-attachments/assets/3cdadf0e-38d1-483d-aeaf-012c1e2a0037)
   
   I have taken 5 snapshots total, the first three using full snapshots, the 
second two using incremental snapshots.
   Here are the results:
   
![2025-03-26_09-50](https://github.com/user-attachments/assets/e7155efc-126c-48d5-927c-6c08fd644f6c)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org