Re: disk change-out and packages
On Thu, Mar 05, 2015 at 10:12:09AM +, Craig Skinner wrote: 18.10.7 Which Backup Program Is Best? dump(8) Period. Elizabeth D. Zwicky torture tested all the backup programs discussed here. The clear choice for preserving all your data and all the peculiarities of UNIX file systems is dump. Elizabeth created file systems containing a large variety of unusual conditions (and some not so unusual ones) and tested each program by doing a backup and restore of those file systems. The peculiarities included: files with holes, files with holes and a block of nulls, files with funny characters in their names, unreadable and unwritable files, devices, files that change size during the backup, files that are created/deleted during the backup and more. It would be rather interesting to see the results of equivalent tests performed today, with modern systems. For one thing, the possible set of 'funny' characters in file names have expanded quite a bit since 1991, and $DEITY only knows what new and exciting file system bugs have been introduced with the various newer file systems on Linuxes, Solaris and others. But the paper is still a very good read. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: disk change-out and packages
On 2015-03-04 Wed 07:22 AM |, Ed Ahlsen-Girard wrote: I thought tar | tar would get everything. 18.10.7 Which Backup Program Is Best? dump(8) Period. Elizabeth D. Zwicky torture tested all the backup programs discussed here. The clear choice for preserving all your data and all the peculiarities of UNIX file systems is dump. Elizabeth created file systems containing a large variety of unusual conditions (and some not so unusual ones) and tested each program by doing a backup and restore of those file systems. The peculiarities included: files with holes, files with holes and a block of nulls, files with funny characters in their names, unreadable and unwritable files, devices, files that change size during the backup, files that are created/deleted during the backup and more. 5.0 Conclusions (Zwicky): These results are in most cases stunningly appalling. dump comes out ahead, which is no great surprise. Tools tested were: tar, gnutar, bar, cpio, pax, afio, fbackup, and bru. Almost all backup utilities are based on these tools underneath. Others use rsync, which is also not as reliable as dump as like the other tools, it does not work with the raw binary data of an (unmounted) disk. References: http://archive-org.com/page/3589638/2014-01-25/http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html http://www.coredumps.de/doc/dump/zwicky/testdump.doc.html -- Everybody's Free To Wear Sunscreen http://youtu.be/sTJ7AzBIJoI
Re: disk change-out and packages
For backup software, I usually make a program or script to make all 64k (minus a few) combination of two-byte-filenames in a dir, then ask for a backup and later a restore of it. Surprising results when backup programs decide that certain combinations of bytes are illegal and won't back them up for you, when they in reality are valid unix file names. Most of the time I guess they know Our gui will have a hard time painting the creative chars needed in a certain way. Or just malice. 2015-03-05 11:38 GMT+01:00 Peter N. M. Hansteen pe...@bsdly.net: On Thu, Mar 05, 2015 at 10:12:09AM +, Craig Skinner wrote: 18.10.7 Which Backup Program Is Best? dump(8) Period. Elizabeth D. Zwicky torture tested all the backup programs discussed here. The clear choice for preserving all your data and all the peculiarities of UNIX file systems is dump. Elizabeth created file systems containing a large variety of unusual conditions (and some not so unusual ones) and tested each program by doing a backup and restore of those file systems. The peculiarities included: files with holes, files with holes and a block of nulls, files with funny characters in their names, unreadable and unwritable files, devices, files that change size during the backup, files that are created/deleted during the backup and more. It would be rather interesting to see the results of equivalent tests performed today, with modern systems. For one thing, the possible set of 'funny' characters in file names have expanded quite a bit since 1991, and $DEITY only knows what new and exciting file system bugs have been introduced with the various newer file systems on Linuxes, Solaris and others. But the paper is still a very good read. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds. -- May the most significant bit of your life be positive.
Re: disk change-out and packages
On 2015-03-05 Thu 11:38 AM |, Peter N. M. Hansteen wrote: It would be rather interesting to see the results of equivalent tests performed today, with modern systems. Yes. Until a new paper comes out, go for a dump! Where possible, unmount partitions before dumping dump the RAW character device: operator@cedar:~ 0$ ls -l /dev/sd4h /dev/rsd4h crw-r- 1 root operator 13, 71 Jan 22 11:31 /dev/rsd4h brw-r- 1 root operator4, 71 Jan 22 11:31 /dev/sd4h Note that operator's home is /operator (not /home/operator) so /home can be sudo unmounted by operator for dumping, with ssh keys other dot files still available. -- Glasgow University has great Scottish balls with live folk music: https://twitter.com/Craig_Skinner/status/573437436119912448 Programme (diagrams/text): http://my.strathspey.org/dd/list/4012/
Re: disk change-out and packages
On Wed, Mar 04, 2015 at 01:22:55PM GMT, Ed Ahlsen-Girard wrote:
First method: mount all the slices in /tree and run a series of cp -R
as root. Files seemed to get there but something was not right with
permissions when I tried booting the new disk, so I dropped back and
did some research.
You have forgotten '-p' option - from 'man 1 cp':
Preserve in the copy as many of the modification time, access time,
file flags, file mode, user ID, and group ID as allowed by
permissions.
In the same manual, however, it reads:
However, cp copies hard linked files as separate files. To
preserve hard links, use a utility such as pax(1) or tar(1)
instead.
So using 'cp' to, effectively, mirror the disk, is not the best of
ideas.
Reinstalled, mounted the new slices as before, and ran:
#!/bin/sh
tar -cXf - /* | tar -xpf - -C /tree
tar -cXf - /home/* | tar -xpf - -C /tree/home
tar -cXf - /usr/* | tar -xpf - -C /tree/usr
tar -cXf - /usr/X11R6/* | tar -xpf - -C /tree/usr/X11R6
tar -cXf - /usr/local/* | tar -xpf - -C /tree/usr/local
tar -cXf - /usr/obj/* | tar -xpf - -C /tree/usr/obj
tar -cXf - /usr/src/* | tar -xpf - -C /tree/usr/src
tar -cXf - /var/* | tar -xpf - -C /tree/var
I had copied the new disk's fstab so that the duids were right when I
started from it.
Results were interesting. I got another copy of /home inside
/tree/home, as well as what I wanted in it, and youtube-dl turns out
to make filenames too long for tar. Nevertheless, I could log in as
myself. But running my usual packages at login didn't work: file not
found.
That's because the syntax is wrong - your working directory ('-C')
should have been '/tree' in all of the above.
Also, drop the asterisk '*'.
You could have shortened it all to:
tar -cXf - / | tar -xpf - -C /tree
for i in /home /usr /var ; do tar -cf - $i | tar -xpf - -C /tree ; done
Should I have not tried to save that much time?
You could have simply reinstalled OpenBSD on the new disk, copied the
crucial data (i.e. '/home') and reinstalled ports/packages, but copying
the data using 'tar' would have taken much less time... provided you got
it right the first time ;^)
I thought tar | tar would get everything. Do I need to install the
packages on the new disk? Is this a time that pkg_check is my friend?
Fix it using the above info, and reinstalling packages should not be
necessary.
Regards,
Raf
Re: disk change-out and packages
On Wed, 4 Mar 2015, Raf Czlonka wrote: In the same manual, however, it reads: However, cp copies hard linked files as separate files. To preserve hard links, use a utility such as pax(1) or tar(1) instead. So using 'cp' to, effectively, mirror the disk, is not the best of ideas. rsync -avH will copy hard linked files, .. and is a much better option - especially if using a network connection [e.g. to the backup disk mounted on another machine]. Lee
Re: disk change-out and packages
Ed Ahlsen-Girard eagir...@cox.net writes: I decided to upgrade the internal drive, so I hooked up the new on on the CD's usual SATA channel and installed, having adjust the disklabel more to suit me (the auto partition of /usr left it really tight on space, and home was not big enough). First method: mount all the slices in /tree and run a series of cp -R as root. Files seemed to get there but something was not right with permissions when I tried booting the new disk, so I dropped back and did some research. For this kind of things, dump/restore is a good way too that won't mess anything. AFAIK your differents source directories (/, /home, ...) have to already be differents partitions, then you can go like this: # mount -o async /dev/sd?a /tree # cd /tree # dump -0a -f - / | restore -rf - # mount -o async /dev/sd?d /tree/home # cd /tree/home # dump -0a -f - /home | restore -rf - -- Manuel Giraud
Re: disk change-out and packages
On Wed, Mar 04, 2015 at 03:22:45PM GMT, L. V. Lammert wrote: rsync -avH will copy hard linked files, .. and is a much better option - especially if using a network connection [e.g. to the backup disk mounted on another machine]. Hi Lee, I was meant to mention it but given the issues the OP had with (seemingly) simple 'tar' syntax, I did not want to get into explaining yet another tool (i.e. trailing slash, etc.). Besides, it's not in base. Regards, Raf
Re: disk change-out and packages
On Mar 04 07:22:55, eagir...@cox.net wrote: I decided to upgrade the internal drive, so I hooked up the new on on the CD's usual SATA channel and installed, having adjust the disklabel more to suit me (the auto partition of /usr left it really tight on space, and home was not big enough). First method: mount all the slices in /tree and run a series of cp -R as root. You said you installed on the new disk already. Boot form it, and treat your old disk as backup. What I usually do in a situation like this is cd /home ; dump /old/home | restore rf - Files seemed to get there but something was not right with permissions when I tried booting the new disk dump | restore beats almost anything else (when on the same machine). Boot from the freshly installed new disk and copy from the old. not the other way round. Nevertheless, I could log in as myself. Again, boot your new system as soon as it is bootable, and copy user data from the old disk; that includes vipw(8) to retain the old accounts, the old /etc/*, the old crontabs, the old /root. But running my usual packages at login didn't work: file not found. Install them from scratch, and only copy over their configs fom /etc. Hint: ls /old/var/db/pkg (only those you actually want, don't make the package system think that you explicitly want e.g. libdvdcss-1.3.0 if it is just a dependency of what you actually want). Should I have not tried to save that much time? I thought tar | tar would get everything. Do I need to install the packages on the new disk? It's the easiest way, and better that cp /usr/local Also, you are running a different version of the OS now. Jan
Re: disk change-out and packages
I decided to upgrade the internal drive, so I hooked up the new on on the CD's usual SATA channel and installed, having adjust the disklabel more to suit me (the auto partition of /usr left it really tight on space, and home was not big enough). First method: mount all the slices in /tree and run a series of cp -R as root. Files seemed to get there but something was not right with permissions when I tried booting the new disk, so I dropped back and did some research. Reinstalled, mounted the new slices as before, and ran: #!/bin/sh tar -cXf - /* | tar -xpf - -C /tree tar -cXf - /home/* | tar -xpf - -C /tree/home tar -cXf - /usr/* | tar -xpf - -C /tree/usr tar -cXf - /usr/X11R6/* | tar -xpf - -C /tree/usr/X11R6 tar -cXf - /usr/local/* | tar -xpf - -C /tree/usr/local tar -cXf - /usr/obj/* | tar -xpf - -C /tree/usr/obj tar -cXf - /usr/src/* | tar -xpf - -C /tree/usr/src tar -cXf - /var/* | tar -xpf - -C /tree/var I had copied the new disk's fstab so that the duids were right when I started from it. Results were interesting. I got another copy of /home inside /tree/home, as well as what I wanted in it, and youtube-dl turns out to make filenames too long for tar. Nevertheless, I could log in as myself. But running my usual packages at login didn't work: file not found. Should I have not tried to save that much time? I thought tar | tar would get everything. Do I need to install the packages on the new disk? Is this a time that pkg_check is my friend? pax -rw -pe was what you wanted. Possibly with the -k option too. If you are intending to do this filesystem-by-filesystem, you may experience better results using the dump restore tools, in a form something like: cd /new/filesystem dump 0af - /filesystem | restore -f -
Re: disk change-out and packages
On Wed, Mar 04, 2015 at 06:54:20PM GMT, L. V. Lammert wrote: Raf, Lee, No issue there, .. but rsync is much better than anything IN base for file synchronization (cross-filesystem, works over ssh [et al], properly handles permissions and hardlinks, . ). If there is an issue installing packages I would think that should probably be addressed first. For *file synchronisation*, 'rsync' is, indeed, *very* good - no doubt! I often use it myself. However, here we're talking about a disk swap-over. 'tar' or, as others have already suggested, 'dump/restore' will do just fine. 'rsync', as I have previously mentioned, will only contribute to OP's further confusion (as he was already struggling with tar's syntax) so helping to fix the issues at hand seemed like a batter idea than (potentially) introducing new ones. rsync's availability as a package/port is, indeed, as secondary matter. Raf
Re: disk change-out and packages
On Wed, 4 Mar 2015, Raf Czlonka wrote: Hi Lee, I was meant to mention it but given the issues the OP had with (seemingly) simple 'tar' syntax, I did not want to get into explaining yet another tool (i.e. trailing slash, etc.). Besides, it's not in base. Regards, Raf Raf, No issue there, .. but rsync is much better than anything IN base for file synchronization (cross-filesystem, works over ssh [et al], properly handles permissions and hardlinks, . ). If there is an issue installing packages I would think that should probably be addressed first. Lee
Re: disk change-out and packages
On 4 March 2015, Manuel Giraud man...@ledu-giraud.fr wrote: Ed Ahlsen-Girard eagir...@cox.net writes: I decided to upgrade the internal drive, so I hooked up the new on on the CD's usual SATA channel and installed, having adjust the disklabel more to suit me (the auto partition of /usr left it really tight on space, and home was not big enough). First method: mount all the slices in /tree and run a series of cp -R as root. Files seemed to get there but something was not right with permissions when I tried booting the new disk, so I dropped back and did some research. For this kind of things, dump/restore is a good way too that won't mess anything. AFAIK your differents source directories (/, /home, ...) have to already be differents partitions, then you can go like this: # mount -o async /dev/sd?a /tree # cd /tree # dump -0a -f - / | restore -rf - # mount -o async /dev/sd?d /tree/home # cd /tree/home # dump -0a -f - /home | restore -rf - +1 for dump / restore. cp doesn't handle hardlinks, tar / pax have limitations on maximum path length, rsync is resource-hungry if you tell it to deal with hardlinks, and cpio has other limitations on file names. Really, dump / restore is the only viable choice for this kind of task. Regards, Liviu Daia
disk change-out and packages
I decided to upgrade the internal drive, so I hooked up the new on on the CD's usual SATA channel and installed, having adjust the disklabel more to suit me (the auto partition of /usr left it really tight on space, and home was not big enough). First method: mount all the slices in /tree and run a series of cp -R as root. Files seemed to get there but something was not right with permissions when I tried booting the new disk, so I dropped back and did some research. Reinstalled, mounted the new slices as before, and ran: #!/bin/sh tar -cXf - /* | tar -xpf - -C /tree tar -cXf - /home/* | tar -xpf - -C /tree/home tar -cXf - /usr/* | tar -xpf - -C /tree/usr tar -cXf - /usr/X11R6/* | tar -xpf - -C /tree/usr/X11R6 tar -cXf - /usr/local/* | tar -xpf - -C /tree/usr/local tar -cXf - /usr/obj/* | tar -xpf - -C /tree/usr/obj tar -cXf - /usr/src/* | tar -xpf - -C /tree/usr/src tar -cXf - /var/* | tar -xpf - -C /tree/var I had copied the new disk's fstab so that the duids were right when I started from it. Results were interesting. I got another copy of /home inside /tree/home, as well as what I wanted in it, and youtube-dl turns out to make filenames too long for tar. Nevertheless, I could log in as myself. But running my usual packages at login didn't work: file not found. Should I have not tried to save that much time? I thought tar | tar would get everything. Do I need to install the packages on the new disk? Is this a time that pkg_check is my friend? -- Edward Ahlsen-Girard Ft Walton Beach, FL
Re: disk change-out and packages
On Wed, 4 Mar 2015, Ed Ahlsen-Girard wrote: I decided to upgrade the internal drive, so I hooked up the new on on the CD's usual SATA channel and installed, having adjust the disklabel more to suit me (the auto partition of /usr left it really tight on space, and home was not big enough). First method: mount all the slices in /tree and run a series of cp -R as root. Files seemed to get there but something was not right with permissions when I tried booting the new disk, so I dropped back and did some research. Reinstalled, mounted the new slices as before, and ran: #!/bin/sh tar -cXf - /* | tar -xpf - -C /tree tar -cXf - /home/* | tar -xpf - -C /tree/home tar -cXf - /usr/* | tar -xpf - -C /tree/usr tar -cXf - /usr/X11R6/* | tar -xpf - -C /tree/usr/X11R6 tar -cXf - /usr/local/* | tar -xpf - -C /tree/usr/local tar -cXf - /usr/obj/* | tar -xpf - -C /tree/usr/obj tar -cXf - /usr/src/* | tar -xpf - -C /tree/usr/src tar -cXf - /var/* | tar -xpf - -C /tree/var I had copied the new disk's fstab so that the duids were right when I started from it. Results were interesting. I got another copy of /home inside /tree/home, as well as what I wanted in it, and youtube-dl turns out to make filenames too long for tar. Nevertheless, I could log in as myself. But running my usual packages at login didn't work: file not found. Should I have not tried to save that much time? I thought tar | tar would get everything. Do I need to install the packages on the new disk? Is this a time that pkg_check is my friend? pax -rw -pe was what you wanted. Possibly with the -k option too. Regards, David
