since file count is the major issue with rsync and i can rsync almost
3million files pretty easily with 2Gb i think you should be ok. I transfer
files that average about 20KB each so my file sizes are much smaller on
average than yours are but my file count is much closer. that number of
files is going to extend your rsync out quite a bit though. while 4hours of
raw data copy is about right for 4TB, i think the rsync checksum process
could double that time.
On Dec 28, 2007 6:30 PM, Bryan Penney <[EMAIL PROTECTED]> wrote:
>
> The pool consists of about 3.8 million files, so there are a lot of
> small files.
>
> On 12/28/2007 5:37 PM, dan wrote:
> > then you should be able to rsync that accross in 4 or 5 hours. is it
> > mostly large files or small files? if it is large files then you
> > should be fine, but a ton of small files might be rough. just give it
> > a shot, only way to know for sure :)
> >
> > On Dec 28, 2007 4:29 PM, Bryan Penney <[EMAIL PROTECTED]
> > <mailto:[EMAIL PROTECTED]>> wrote:
> >
> > Yeah we will have them plugged into the same gigabit switch.
> >
> > On 12/28/2007 5:17 PM, Daniel Denson wrote:
> > > Bryan Penney wrote:
> > >> The original document I quoted was for an older version, but I
> > found
> > >> one for 2.9.1 and is still says it doesn't understand hardlinks
> > >>
> > >>
> >
> http://www.seas.upenn.edu/~bcpierce/unison//download/releases/unison-2.9.1/unison-manual.pdf<http://www.seas.upenn.edu/%7Ebcpierce/unison//download/releases/unison-2.9.1/unison-manual.pdf>
> > <
> http://www.seas.upenn.edu/%7Ebcpierce/unison//download/releases/unison-2.9.1/unison-manual.pdf
> >
> > >>
> > >>
> > >> I've copied a much smaller pool (150GB) using rsync when we first
> > >> went to a production server.
> > >>
> > >> Both of the servers have 2GB of RAM.
> > >> After I get the drives for the new server, I will try rsync.
> > It will
> > >> be interesting to see how long it takes to copy all of this
> > data with
> > >> all of those hardlinks.
> > >>
> > >> thanks for the help.
> > >>
> > >> Bryan
> > >>
> > >>
> > >>
> > >> On 12/28/2007 4:50 PM, dan wrote:
> > >>> no it wouldnt, but i though it did. is that statement for an
> > older
> > >>> version? it may just not handle it. rsync should work if you
> > have
> > >>> enough RAM
> > >>>
> > >>> On Dec 28, 2007 3:10 PM, Bryan Penney < [EMAIL PROTECTED]
> > <mailto:[EMAIL PROTECTED]>
> > >>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote:
> > >>>
> > >>> In reading about Unison I found a statement in the Caveats
> and
> > >>> Shortcomings section that said "Unison does not understand
> > hard
> > >>> links"
> > >>>
> > >>> If this is true, would Unison work in this situation?
> > >>>
> > >>> On 12/28/2007 2:28 PM, dan wrote:
> > >>> > no, you will have to copy the entire 'pool' or 'cpool'
> > over. you
> > >>> > could copy individual pc backups, BUT when backuppc
> nightly
> > >>> runs it
> > >>> > will remove any hardlinks from the pool that are not
> needed
> > >>> > elsewhere. when you copy over pc backups after that,
> > the will
> > >>> not use
> > >>> > hardlinks and so your filesystem usage will go up a lot.
> i
> > >>> would very
> > >>> > much suggest you do it all in one shot.
> > >>> >
> > >>> > i know that time is against you on this and that 2TB
> > even over
> > >>> gigabit
> > >>> > is 5 hours so i would suggest that you rsync the files
> over
> > >>> once and
> > >>> > leave your other machine up running backups, then once
> > it has
> > >>> > finished, turn backups off and rsync the source to the
> > target
> > >>> again.
> > >>> > then you will have the bulk of the data over and only
> > have to
> > >>> pull
> > >>> > changes. i worry about the file count for 2TB being
> > too much
> > >>> for
> > >>> > rsync so consider Unison for the transfers. In my
> > reading i have
> > >>> > found that though unison has the same issue as rsync(same
> > >>> algorythms)
> > >>> > for a high number for files, it can handle more files in
> > less
> > >>> memory.
> > >>> >
> > >>> > I have done this method to push about 800GB over and it
> > worked
> > >>> well,
> > >>> > but my backup server has 2GB of RAM and runs gigabit.
> > >>> >
> > >>> > maybe consider adding some network interfaces and
> > channel bonding
> > >>> > them. i dont know if you have parts lying around but
> > channel
> > >>> bonding
> > >>> > in linux is pretty easy and you have agrigate each NICs
> > >>> bandwidth to
> > >>> > reduce that transfer time though i suspect that your
> drives
> > >>> are not
> > >>> > much faster than 1 gigabit NIC so you might not get much
> > >>> benefit on
> > >>> > gigabit.
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Dec 28, 2007 10:17 AM, Bryan Penney <
> > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>> > <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>> wrote:
> > >>> >
> > >>> > We have a server running BackupPC that has filled up
> > it's 2TB
> > >>> > partition
> > >>> > (96% full anyway). We are planning on moving
> > BackupPC to
> > >>> another
> > >>> > server
> > >>> > but would like bring the history of backups over
> > without
> > >>> waiting the
> > >>> > extended period of time (days?) for the entire pool
> > to copy.
> > >>> Is there
> > >>> > any way to copy "pieces" of the pool, maybe per PC,
> > at a
> > >>> time? This
> > >>> > would allow us to migrate over the course of a few
> weeks
> > >>> without
> > >>> > having
> > >>> > days at a time with no backups.
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> -------------------------------------------------------------------------
> > >>>
> > >>>
> > >>> > This SF.net email is sponsored by: Microsoft
> > >>> > Defy all challenges. Microsoft(R) Visual Studio 2005.
> > >>> >
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > >>> <http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > <http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/>>
> > >>> > _______________________________________________
> > >>> > BackupPC-users mailing list
> > >>> > [email protected]
> > <mailto:[email protected]>
> > >>> <mailto:[email protected]
> > <mailto:[email protected]>>
> > >>> > <mailto: [email protected]
> > <mailto:[email protected]>
> > >>> <mailto:[email protected]
> > <mailto:[email protected]>>>
> > >>> > List:
> > >>> >
> > https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > >>> > <
> > https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > >>>
> > <https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > <https://lists.sourceforge.net/lists/listinfo/backuppc-users>>>
> > >>> > Wiki: http://backuppc.wiki.sourceforge.net
> > >>> > Project: http://backuppc.sourceforge.net/
> > >>> <http://backuppc.sourceforge.net/>
> > >>> > < http://backuppc.sourceforge.net/>
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>
> > > a long time. you got gigabit?
> > >
> > >
> > >
> >
> >
>
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
BackupPC-users mailing list
[email protected]
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
