Re: [gentoo-user] backing up a partition
On 24/08/18 20:53, Rich Freeman wrote: > On Fri, Aug 24, 2018 at 6:09 AM Mick wrote: >> However, you may prefer to use clonezilla instead of dd. The dd command will >> copy each and every bit and byte of the partition whether it has data on it >> or >> not. It is not particularly efficient. Clonezilla will perform better at >> this task. >> >> Personally, I would only keep a back up of the filesystem contents with e.g. >> rsync, and reformat the partition and restore its contents in the case of a >> disaster recovery scenario. > Just to summarize the sorts of options you have: > > dd = bit level copy. Output is the size of the partition, period, > though you could compress the output by piping it into a compression > utility/etc. Restored partition is identical to original, including > unallocated space, file fragmentation, etc. > > clonezilla/partimage/etc = sparse bit level copy. Output is the size > of all blocks that contain useful data, and can be further compressed. > Restored partition will contain zeros in the place of free space, but > will still preserve file fragmentation, special filesystem features, > etc. Basically these tools operate like dd at a block level, but they > first identify which blocks are used/unused. Savings is minimal for a > full filesystem, and substantial for a near-empty one. These tools > will fall back to dd if they can't identify free space, and can > support a wider variety of filesystems quickly because they don't have > to be able to mount/read the filesystem, just figure out which blocks > matter. I'll also note that with clonezilla you get a fairly nice > all-in-one bootable image that can store these images remotely via > ssh/samba/etc, which makes restoring images onto bare metal very easy. > > tar/rsync/etc = file level copy. Output is the logical size of all > the files on the filesystem. Restore partition will only contain file > contents - details like fragmentation, trailing unused space in > blocks, unused space in general, or many filesystem-specific features > like snapshots/etc will NOT be preserved. On the other hand it is > trivial to restore this data to any filesystem of any type of any > sufficient size. The other solutions make resizing or changing > filesystems more-or-less impossible unless you can mount the image > files and then do a subsequent file-level copy (which is no different > than doing a file level copy in the first place). > > I'd toss in one other general category: > > dump/send/etc - filesystem-specific serializing tools. The tools are > specific to the filesystem, so you can't just point them at a whole > hard drive with varying partition types like you can with clonezilla. > They may or may not reproduce details like fragmentation, but they > will efficiently store the actual data and will reproduce all > filesystem-specific features (snapshots, special attributes, etc). > They may also contain features that make them more efficient > (especially for incremental backups) because they can use an algorithm > suited for the low-level data structures employed by the filesystem, > instead of doing scanning at the file/directory level. For example, > it could just read all the metadata on the disk sequentially as it is > physically stored on the disk, instead of traversing it from root down > to leaf in the directory hierarchy which could result in lots of > seeks. Filesystems like btrfs/zfs have data structures that make it > VERY efficient to compare two related snapshots and find just the > differences between them, including differences of one block in the > middle of a large file without having to read the whole file. > Restoration usually is flexible with regard to filesystem size, but > not type. That is, if you have a 100GB filesystem with 20GB of data, > you could restore it to a 30GB filesystem of the same type, but not > one of a different type as with tar. > > The best solution for you obviously depends on your needs. I try to > go with the last category in general as it is far more efficient. > But, clonezilla is my general tool for replicating whole systems/etc > since it does that so well and works with anything. For partial > backups of high-value data I use duplicity, which is file-level (and > supports various cloud/etc options for storage). > and another category - do a proper backup using backup software. Copying the raw partition has some disadvantages - difficult (but not impossible) if you want to restore to new hardware (on failure), harder to restore if you decide to change the underlying filesystem, takes far longer to copy than a backup but the biggest problem is no versioning. Raw copies are great if you want to do an immediate restore, but hard work or useless after a few days of changes. Think of it this way - in almost all cases its the data that's important, not whats holding the data. BillK
Re: [gentoo-user] /boot filesystem, SSDs, TRIM
Thus spoke Adam Carter (adamcart...@gmail.com): > For a long time people recommended ext2 for /boot. The Gentoo wiki still > does. Is there any compelling reason to use ext2 for /boot (on a system > whose other filesystems are ext4) these days? AFAIK for systems that have > /boot on an SSD, ext4 makes more sense due to discard support, and for > non-SSD it doesn't matter either way. Have I missed something? Why not just disable journaling on those partitions altogether? # Delete has_journal option tune2fs -O ^has_journal # Issue required fsck e2fsck -f # Check fs options dumpe2fs | less -- malloc1337 mailto: dis...@mm-no.de
Re: [gentoo-user] backing up a partition
On Fri, Aug 24, 2018 at 6:09 AM Mick wrote: > > However, you may prefer to use clonezilla instead of dd. The dd command will > copy each and every bit and byte of the partition whether it has data on it or > not. It is not particularly efficient. Clonezilla will perform better at > this task. > > Personally, I would only keep a back up of the filesystem contents with e.g. > rsync, and reformat the partition and restore its contents in the case of a > disaster recovery scenario. Just to summarize the sorts of options you have: dd = bit level copy. Output is the size of the partition, period, though you could compress the output by piping it into a compression utility/etc. Restored partition is identical to original, including unallocated space, file fragmentation, etc. clonezilla/partimage/etc = sparse bit level copy. Output is the size of all blocks that contain useful data, and can be further compressed. Restored partition will contain zeros in the place of free space, but will still preserve file fragmentation, special filesystem features, etc. Basically these tools operate like dd at a block level, but they first identify which blocks are used/unused. Savings is minimal for a full filesystem, and substantial for a near-empty one. These tools will fall back to dd if they can't identify free space, and can support a wider variety of filesystems quickly because they don't have to be able to mount/read the filesystem, just figure out which blocks matter. I'll also note that with clonezilla you get a fairly nice all-in-one bootable image that can store these images remotely via ssh/samba/etc, which makes restoring images onto bare metal very easy. tar/rsync/etc = file level copy. Output is the logical size of all the files on the filesystem. Restore partition will only contain file contents - details like fragmentation, trailing unused space in blocks, unused space in general, or many filesystem-specific features like snapshots/etc will NOT be preserved. On the other hand it is trivial to restore this data to any filesystem of any type of any sufficient size. The other solutions make resizing or changing filesystems more-or-less impossible unless you can mount the image files and then do a subsequent file-level copy (which is no different than doing a file level copy in the first place). I'd toss in one other general category: dump/send/etc - filesystem-specific serializing tools. The tools are specific to the filesystem, so you can't just point them at a whole hard drive with varying partition types like you can with clonezilla. They may or may not reproduce details like fragmentation, but they will efficiently store the actual data and will reproduce all filesystem-specific features (snapshots, special attributes, etc). They may also contain features that make them more efficient (especially for incremental backups) because they can use an algorithm suited for the low-level data structures employed by the filesystem, instead of doing scanning at the file/directory level. For example, it could just read all the metadata on the disk sequentially as it is physically stored on the disk, instead of traversing it from root down to leaf in the directory hierarchy which could result in lots of seeks. Filesystems like btrfs/zfs have data structures that make it VERY efficient to compare two related snapshots and find just the differences between them, including differences of one block in the middle of a large file without having to read the whole file. Restoration usually is flexible with regard to filesystem size, but not type. That is, if you have a 100GB filesystem with 20GB of data, you could restore it to a 30GB filesystem of the same type, but not one of a different type as with tar. The best solution for you obviously depends on your needs. I try to go with the last category in general as it is far more efficient. But, clonezilla is my general tool for replicating whole systems/etc since it does that so well and works with anything. For partial backups of high-value data I use duplicity, which is file-level (and supports various cloud/etc options for storage). -- Rich
Re: [gentoo-user] backing up a partition
On Friday, 24 August 2018 09:32:23 BST Philip Webb wrote: > I want to make a copy of a partition which I can use to replace it, > if some catastrophe damages the partition or wipes it out ; > it needs to be byte-byte identical, incl all permissions. You are looking to clone a partition. This is a bit by bit identical copy of the original partition. You could clone a partition into another, so you end up with two identical partitions including their boot record, fs and contents, or save the cloned partition into an image file. However, you may prefer to use clonezilla instead of dd. The dd command will copy each and every bit and byte of the partition whether it has data on it or not. It is not particularly efficient. Clonezilla will perform better at this task. Personally, I would only keep a back up of the filesystem contents with e.g. rsync, and reformat the partition and restore its contents in the case of a disaster recovery scenario. I would prefer a fs backup, especially in the case of a USB stick being the backup storage media. This is because rsync will only copy data which has changed since the last backup, rather than overwriting what is on the USB already. I expect using something like dd repeatedly will reduce considerably the life of the USB stick. > Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy', > where the partition has been mounted at /mnt/xxx > & a USB stick has been mounted at /mnt/yyy . Will that do the job ? Not like that. If you intend to create a clone of the partition, rather than its filesystem, you need to copy the whole block device, not its mount point. dd if=/dev/sdXx of=/media/$USER//clone_of_sdXx.img bs=4096 conv=notrunc,fsync status=progress Where /media/$USER// is assumed to be the mountpoint of your mounted USB storage device. The above will save an image file of the cloned partition as 'clone_of_sdXx.img' within the existing filesystem of the USB stick. You could then copy this partition image to a secondary storage if you wish, encrypt it, compress it, etc. You'll be able to mount this image with '-o loop' and examing or use its contents. mount /media/$USER//clone_of_sdXx.img /mnt/My_partition_image -o loop ls -l /mnt/My_partition_image To create a full partition clone on the USB stick rather than an image file change the of= operand into the block device your USB stick is recognised as; e.g. dd if=/dev/sdXx of=/dev/sdYy bs=4096 conv=notrunc,fsync status=progress CAUTION: If you get /dev/sdYy wrong you *will* lose data on the device you overwrite. You may have problems when you try to mount the cloned USB stick with the original partition in place, because they will both have identical filesystem UUIDs. I haven't tried this to know what errors it may produce. > There seems also to be an issue re 'bs=' : > what size is best ? i plan to use USB 3.0 for quicker copying. You can test what block size will transfer data faster on your particular hardware setup. I assume in the example below the I/O bottleneck is your USB, rather than your HDD. dd if=/dev/zero of=/media/$USER//testfile bs=1024 count=100 rm /media/$USER//testfile && sync dd if=/dev/zero of=/media/$USER//testfile bs=2048 count=50 rm /media/$USER//testfile && sync and so on, each time increasing the bs to 4096, 8192, ... and halving the count to copy the same amount of data. You will soon find the transfer speed arrives at a peak and then reduces when an I/O bottleneck is reached. In the above example I have used a 1GB file, but you may want to increase this to a larger size to make sure you saturate your buffers. > Does it matter how the USB stick is formatted ? > Can I use a raw stick with the usual default VFAT formatting ? > Might it be better to replace that with a Linux FS, eg Ext2 ? If you are using the USB stick to save an image file to it, then you will come across the VFAT file size limit of 4GB. So, use exFAT, or ext2. If you are using a USB stick to create a partition clone it doesn't matter what the USB fs type is. It will be replaced with that of the original partition. HTH. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] backing up a partition
On Fri, 24 Aug 2018 04:32:23 -0400, Philip Webb wrote: > I want to make a copy of a partition which I can use to replace it, > if some catastrophe damages the partition or wipes it out ; > it needs to be byte-byte identical, incl all permissions. > > Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy', > where the partition has been mounted at /mnt/xxx > & a USB stick has been mounted at /mnt/yyy . Will that do the job ? It will, provided the USB stick is large enough. > There seems also to be an issue re 'bs=' : > what size is best ? i plan to use USB 3.0 for quicker copying. I usually use 4M but anything over 1M gives similar results. These days I use dcfldd rather than dd and it gives a nice progress display and also works out the block size for itself. > Does it matter how the USB stick is formatted ? > Can I use a raw stick with the usual default VFAT formatting ? > Might it be better to replace that with a Linux FS, eg Ext2 ? It doesn't matter how the stick *was* formatted because you are overwriting it when you dd. The resulting stick won't even have a partition table. Alternatively, format the stick and dd to a file on the stick. Bear in mind that dd copies the whole partition, including the bits (and bytes) not currently in use by the filesystem, so it is inefficient in space terms. If you just want a backup of the contents of the partition, would tar not suffice? -- Neil Bothwick If Yoda so strong in force is, why words in right order he cannot put? pgp16LwOCEmjJ.pgp Description: OpenPGP digital signature
Re: [gentoo-user] backing up a partition
Dear Philip, On Fri 24 Aug 2018 at 04:32:23 -0400, Philip Webb wrote: > I want to make a copy of a partition which I can use to replace it, > if some catastrophe damages the partition or wipes it out ; > it needs to be byte-byte identical, incl all permissions. > > Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy', > where the partition has been mounted at /mnt/xxx > & a USB stick has been mounted at /mnt/yyy . Will that do the job ? If you want to make a byte-by-byte copy of a partition, the easiest way in my opinion is to dd the (unmounted) partition, e.g. if=/dev/sda1. You could also do that with the entire drive, e.g. if=/dev/sda. > There seems also to be an issue re 'bs=' : > what size is best ? i plan to use USB 3.0 for quicker copying. For optimal performance the block size should be a multiple of the device’s physical block size. Old hard drives had 512-B blocks, more recent ones have larger 4-KiB physical blocks that are sometimes software-emulated to appear as 512-B blocks. In my experience it is faster to use even larger block sizes; I usually use bs=1M. > Does it matter how the USB stick is formatted ? > Can I use a raw stick with the usual default VFAT formatting ? > Might it be better to replace that with a Linux FS, eg Ext2 ? The lowest overhead is when you write the back-up to a raw partition of the same size, e.g. of=/dev/sdb1. You can also write it to a filesystem, which might make it easier to manage multiple back-ups. Since all permissions etc. are stored in the back-up file, the choice of file system does not really matter, though the back-up file might be too large for VFAT (max ~4 GiB). Sincerely, Bas -- Sebastiaan L. Zoutendijk | slzoutend...@gmail.com
[gentoo-user] backing up a partition
I want to make a copy of a partition which I can use to replace it, if some catastrophe damages the partition or wipes it out ; it needs to be byte-byte identical, incl all permissions. Can I use 'dd' ? -- eg 'dd if=/mnt/xxx of=/mnt/yyy', where the partition has been mounted at /mnt/xxx & a USB stick has been mounted at /mnt/yyy . Will that do the job ? There seems also to be an issue re 'bs=' : what size is best ? i plan to use USB 3.0 for quicker copying. Does it matter how the USB stick is formatted ? Can I use a raw stick with the usual default VFAT formatting ? Might it be better to replace that with a Linux FS, eg Ext2 ? -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca
