On 7/28/2010 12:46 PM, Aaron Burt wrote: > On Tue, Jul 27, 2010 at 07:17:57PM -0700, mike item wrote: > >> Hi All, >> >> A project on my list at home is consolidation of unorganized backups >> smeared across a bunch of ide hdds. Partions are ext2, ext3, vfat, >> reiser, etc ... No backups are incremental, so there are lots of >> duplicates.<snip> >> > I've used findup and fslint to consolidate duplicate files like that. They > basically automate the process you descriubed. It's a sloooow process. > > >> Any suggestions on the choice of target filesystem? The larger issue >> issue is a few hundred cd and dvd backups, but for now, getting rid of >> the stack of old hdds would be good. >> > If you want to just get a server up quick, I'd recommend Ext3, which can be > upgraded to Ext4 or Btrfs. If you want an adventure, build a NAS using ZFS > (on freeNAS, BSD 8 or Nexenta), which can do deduplication without all the > hard-linking. > > > __
Please do not use Ext3 it is very slow compared to Reiser or XFS. XFS is the fastest for large files Reiser will handle lots of small files, both are about 3X faster than Ext3. I did the same md5sum and Perl script approach which worked reasonably well, I may try fslint next. _______________________________________________ PLUG mailing list [email protected] http://lists.pdxlinux.org/mailman/listinfo/plug
