Re: [Dvdrtools-users] 2GB file limit workaround
Hi, If you only want to store files 2GB DVDsize, use an ext2 filesystem (or udf, with mkudffs), and the disk can be mounted in the usual fashion (hint: loop mount). Whether it can be read on Microshite is none of my concern, if you use DVDs as tar tapes it sounds like it's none of yours either. discs created with mkudffs do work fine on windows XP but not on 2000 or less. Regards, Richard pgp0.pgp Description: signature ___ Dvdrtools-users mailing list [EMAIL PROTECTED] http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
[Dvdrtools-users] 2GB file limit workaround
While this is not a mkisofs related topic, it does relate to dvdrecord. I see so many problems, and few solutions on this list that I decided to post a solution, of sorts. I was trying to save multi gigabyte database dumps to dvd, but limitations withing mkisofs (more specifically the joliet file system) prevented me from storing these fat files in an iso image. The workaround is to save the files to a tar file directly, and save this to the DVD, sans any file system. While this does not result in a DVD that can be mounted in the normal fashion, it does result in a DVD that can be treated as a tape, and used for backups. The big win of course is that there is no 32 bit floating integer limit on the file size. To illustrate: 0.) md5sum the files of interest: md5sum /data/multiGigFile.dump | tee /data/multiGigFile.md5sum 1.) Create backup tar file: tar cf /data/multiGigFile.tar /data/multiGigFile.dump 2.) Shoot this onto the DVD using dvdrecord: dvdrecord -v -pad -dao dev=1,0,0 speed=4 /data/multiGigFile.tar 3.) Extract it again using dd. It will be bigger due to padding. dd if=/dev/cdrom of=/data/multiGigFile_restored.tar 4.) Now compare the contents, by changing to another directory, and using tar xvf /data/multiGigFile_restored.tar and then md5sum of the contents of the restored file. If the sums compare then the files should be identical (or try cmp - diff griped about memory being exhausted) So far everything has been good. I'm sure this could be streamlined with pipes, but I have the disk space, and am relatively short of RAM, so I'm leaving the files around for now. As we've seen before, it's best to compress component files _before_ placing them in the archive. I save the uncompressed md5sum file in the archive as well. The bzip2 man pages seem to imply that it has some sort of error detection, that I have not read about for gzip, so perhaps it's better for big files for that reason. -- Allan Peda Programmer, Gene Array Resource Center Rockefeller University Box 203 1230 York Ave New York, NY 10021-6399 (tel) 212-327-7064 (fax) 212-327-7065 ___ Dvdrtools-users mailing list [EMAIL PROTECTED] http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
Re: [Dvdrtools-users] 2GB file limit workaround
What I did myself, was to split my large files into smaller chunks.
I use GNU 'split' to split it into 50MB chunks, then I use parity
archives ('par' utility) to generate redundancy data for my split
volumes.
This way I can recover the large file even if data errors on the disc
prevent me from reading a few of the split volumes.
The par program also makes it easy to verify the integrity of all of the
split volumes with one command.
If they're all OK, I can just use gnu 'cat' to concatenate all the files
to disk.
I also include the md5sum of the large file so I can verify it is OK
after concatenating the small files.
What I can't do, is repair/recover bad volumes without copying all the
split volumes to disk first.
On Fri, Jun 13, 2003 at 04:17:57PM -0400, Allan Peda wrote:
While this is not a mkisofs related topic, it does relate to dvdrecord.
I see so many problems, and few solutions on this list that I decided to
post a solution, of sorts.
I was trying to save multi gigabyte database dumps to dvd, but
limitations withing mkisofs (more specifically the joliet file system)
prevented me from storing these fat files in an iso image.
The workaround is to save the files to a tar file directly, and save
this to the DVD, sans any file system. While this does not result in a
DVD that can be mounted in the normal fashion, it does result in a DVD
that can be treated as a tape, and used for backups. The big win of
course is that there is no 32 bit floating integer limit on the file
size.
To illustrate:
0.) md5sum the files of interest:
md5sum /data/multiGigFile.dump | tee /data/multiGigFile.md5sum
1.) Create backup tar file:
tar cf /data/multiGigFile.tar /data/multiGigFile.dump
2.) Shoot this onto the DVD using dvdrecord:
dvdrecord -v -pad -dao dev=1,0,0 speed=4 /data/multiGigFile.tar
3.) Extract it again using dd. It will be bigger due to padding.
dd if=/dev/cdrom of=/data/multiGigFile_restored.tar
4.) Now compare the contents, by changing to another directory,
and using tar xvf /data/multiGigFile_restored.tar
and then md5sum of the contents of the restored file. If the sums
compare then the files should be identical (or try cmp - diff griped
about memory being exhausted)
So far everything has been good.
I'm sure this could be streamlined with pipes, but I have the disk
space, and am relatively short of RAM, so I'm leaving the files around
for now. As we've seen before, it's best to compress component files
_before_ placing them in the archive. I save the uncompressed md5sum
file in the archive as well.
The bzip2 man pages seem to imply that it has some sort of error
detection, that I have not read about for gzip, so perhaps it's better
for big files for that reason.
--
Allan Peda
Programmer, Gene Array Resource Center
Rockefeller University
Box 203
1230 York Ave
New York, NY 10021-6399
(tel) 212-327-7064
(fax) 212-327-7065
___
Dvdrtools-users mailing list
[EMAIL PROTECTED]
http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
--
Andre Dalle [EMAIL PROTECTED]
Space Monkey
___
Dvdrtools-users mailing list
[EMAIL PROTECTED]
http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
Re: [Dvdrtools-users] 2GB file limit workaround
Thanks for posting that! I was wondering what other people do. I was trying to save multi gigabyte database dumps to dvd, but limitations withing mkisofs (more specifically the joliet file system) Are you sure? Joliet is an optional extra, on unix it's just a waste of space. The workaround is to save the files to a tar file directly, and save this to the DVD, sans any file system. The DVD is just another block device, whatever filesystem can be put on a harddisk can also go on DVD. Ext2 works well, and the disk space requirements are the same as for mkisofs. Obviously you can't store files larger than a DVD, but you can with your method - interesting idea. If you only want to store files 2GB DVDsize, use an ext2 filesystem (or udf, with mkudffs), and the disk can be mounted in the usual fashion (hint: loop mount). Whether it can be read on Microshite is none of my concern, if you use DVDs as tar tapes it sounds like it's none of yours either. I always(!!) create md5sums of all files first and store them with the data, just before mastering, and the master img isn't deleted until the burnt disk verifies ok. It's trivial to do, and allows bad media detection later (though not correction). I also always store an md5sum of the burnt img elsewhere, this allows a quick verify of the entire disk. All scripted of course, called md5 in http://volker.dnsalias.net/soft/scriptutils.tar.gz (rpms there too). md5sum /data/multiGigFile.dump | tee /data/multiGigFile.md5sum tar cf /data/multiGigFile.tar /data/multiGigFile.dump Unless you have specific reason to have a tar format on disk, you could just skip making the tar file. If you're dealing with a file larger than a DVD, the raw method is to use split (and copious quantities of md5sum and disk space). 3.) Extract it again using dd. It will be bigger due to padding. dd if=/dev/cdrom of=/data/multiGigFile_restored.tar Am I the only one who finds that Linux is now too crappy to read CDs/DVDs? Most of the time I get an error with that. Kernel 2.4.20. cat /dev/cdrom is identical to your dd. I have found in the past that some kernels panic at EOM (end of media recording). I have also found that, until recently, it returns too much data (some of the zeroes added by cdrecord -pad), and that on CDs written without -pad (e.g. all of the commercial ones) it craps out with an I/O error even before it has read all of the blocks belonging to the isofs. On a current 2.4.20, it craps out well before the end of the isofs at least half of the time. THIS SUCKS. It's independent of filesystem, happens with ext2 as well. Same for CD + DVD. The only fix is to append 2Mbyte worth of zeroes to whatever gets burnt. cdrecord -pad or -nopad is irrelevant (-pad only adds 15k). compare then the files should be identical (or try cmp - diff griped about memory being exhausted) md5sum is twice as fast as cmp, as cmp needs to read both sets of data, md5sum only one. Compared with the time it takes only one set of data, the time of computing the md5 is insignificant. diff is no good as it is line-based, it will need to read a full line into memory first before it can compare. Depending on the data, this can kill you before the line is finished. Volker -- Volker Kuhlmann is possibly list0570 with the domain in header http://volker.dnsalias.net/ Please do not CC list postings to me. ___ Dvdrtools-users mailing list [EMAIL PROTECTED] http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
Re: [Dvdrtools-users] 2GB file limit workaround
On Fri, 13 Jun 2003, André Dalle wrote:
I can make upto 4.7 Gig udf iso's without problems here and possibly even
bigger upto the filesize limit on current ext2/ext3 filesystems in
linux systems running kernel 2.4.18 and higher :
/usr/bin/mkisofs -o /mnt/backup/disc1.raw
-l \# Allow full 31 character filenames.
-r \# Generate SUSP and RR records using the Rock
\# Ridge protocol to further describe the files on
\# the iso9660 filesystem.
-L \# Allow ISO9660 filenames to begin with a period.
-J \# Generate Joliet directory records in addition to
\# regular iso9660 file names.
-V Archive disc1 \
-P STOCK-2003012301 \
-p stock \
-A Archive disc1 \
/mnt/data1/disc1
[jackson:root]:(~)# mkisofs --version
mkisofs 2.0 (i586-mandrake-linux-gnu)
this mkisofs is a component of cdrtools-2.0 which i patched see :
ftp://crashrecovery.org/pub/linux/cdrtools/
Robert
What I did myself, was to split my large files into smaller chunks.
I use GNU 'split' to split it into 50MB chunks, then I use parity
archives ('par' utility) to generate redundancy data for my split
volumes.
This way I can recover the large file even if data errors on the disc
prevent me from reading a few of the split volumes.
The par program also makes it easy to verify the integrity of all of the
split volumes with one command.
If they're all OK, I can just use gnu 'cat' to concatenate all the files
to disk.
I also include the md5sum of the large file so I can verify it is OK
after concatenating the small files.
What I can't do, is repair/recover bad volumes without copying all the
split volumes to disk first.
On Fri, Jun 13, 2003 at 04:17:57PM -0400, Allan Peda wrote:
While this is not a mkisofs related topic, it does relate to dvdrecord.
I see so many problems, and few solutions on this list that I decided to
post a solution, of sorts.
I was trying to save multi gigabyte database dumps to dvd, but
limitations withing mkisofs (more specifically the joliet file system)
prevented me from storing these fat files in an iso image.
The workaround is to save the files to a tar file directly, and save
this to the DVD, sans any file system. While this does not result in a
DVD that can be mounted in the normal fashion, it does result in a DVD
that can be treated as a tape, and used for backups. The big win of
course is that there is no 32 bit floating integer limit on the file
size.
To illustrate:
0.) md5sum the files of interest:
md5sum /data/multiGigFile.dump | tee /data/multiGigFile.md5sum
1.) Create backup tar file:
tar cf /data/multiGigFile.tar /data/multiGigFile.dump
2.) Shoot this onto the DVD using dvdrecord:
dvdrecord -v -pad -dao dev=1,0,0 speed=4 /data/multiGigFile.tar
3.) Extract it again using dd. It will be bigger due to padding.
dd if=/dev/cdrom of=/data/multiGigFile_restored.tar
4.) Now compare the contents, by changing to another directory,
and using tar xvf /data/multiGigFile_restored.tar
and then md5sum of the contents of the restored file. If the sums
compare then the files should be identical (or try cmp - diff griped
about memory being exhausted)
So far everything has been good.
I'm sure this could be streamlined with pipes, but I have the disk
space, and am relatively short of RAM, so I'm leaving the files around
for now. As we've seen before, it's best to compress component files
_before_ placing them in the archive. I save the uncompressed md5sum
file in the archive as well.
The bzip2 man pages seem to imply that it has some sort of error
detection, that I have not read about for gzip, so perhaps it's better
for big files for that reason.
--
Allan Peda
Programmer, Gene Array Resource Center
Rockefeller University
Box 203
1230 York Ave
New York, NY 10021-6399
(tel) 212-327-7064
(fax) 212-327-7065
___
Dvdrtools-users mailing list
[EMAIL PROTECTED]
http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
--
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org [EMAIL PROTECTED]
___
Dvdrtools-users mailing list
[EMAIL PROTECTED]
http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
Re: [Dvdrtools-users] 2GB file limit workaround
Quoting André Dalle [EMAIL PROTECTED]:
What I did myself, was to split my large files into smaller chunks.
I use GNU 'split' to split it into 50MB chunks, then I use parity
archives ('par' utility) to generate redundancy data for my split
volumes.
This way I can recover the large file even if data errors on the disc
prevent me from reading a few of the split volumes.
The par program also makes it easy to verify the integrity of all of
the split volumes with one command.
If they're all OK, I can just use gnu 'cat' to concatenate all the
files to disk.
I also include the md5sum of the large file so I can verify it is OK
after concatenating the small files.
Is there a good par HOWTO somewhere?
What I can't do, is repair/recover bad volumes without copying all the
split volumes to disk first.
That's the price.
--
Bryan J. Smith, E.I. mailto:[EMAIL PROTECTED] http://thebs.org
___
Dvdrtools-users mailing list
[EMAIL PROTECTED]
http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
Re: [Dvdrtools-users] 2GB file limit workaround
Is there a good par HOWTO somewhere? Do you need one? There's a lot of info on the sourceforge site, and with the version 2 source code. Version 1 seems pretty simple and just compiles. Version 2 doesn't compile with current versiosn of automake/autoconf gcc 3.3. What I can't do, is repair/recover bad volumes without copying all the split volumes to disk first. That's the price. Seems to be on the todo for version 2. Volker -- Volker Kuhlmann is possibly list0570 with the domain in header http://volker.dnsalias.net/ Please do not CC list postings to me. ___ Dvdrtools-users mailing list [EMAIL PROTECTED] http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
Re: [Dvdrtools-users] 2GB file limit workaround
On Sat 14 Jun 2003 14:00:07 NZST +1200, Robert M. Stockmann wrote: this mkisofs is a component of cdrtools-2.0 which i patched see : ftp://crashrecovery.org/pub/linux/cdrtools/ I wish people would give some useful information with their patches, like some basic info, e.g. what the patch actually tries to achieve, instead of just sticking it on an ftp server somewhere in between the other 87834.8734^98784 files on the internet with nothing to go with it. Yes one can even put this info at the top of the patch file without patch complaining./rant Volker -- Volker Kuhlmann is possibly list0570 with the domain in header http://volker.dnsalias.net/ Please do not CC list postings to me. ___ Dvdrtools-users mailing list [EMAIL PROTECTED] http://mail.nongnu.org/mailman/listinfo/dvdrtools-users
