RE: 2-way rsync with delete; SUMMARY

[Kovalev, Ivan]

Yes, Michael, indeed it is. Thank you. Just one unfortunate draw back:
"Unison does not currently understand hard links." (from its shortcoming
section). The ftp site to be clustered has about 250mb * 100 hard links.
That means a waste of about 25GB if implement it with unison.

SUMMARY.

Rsync is currently capable of doing 2-way synchronization of simultaneously
updated systems except for deletion of files. Therefore, it is possible to
use rsync coupled with load balancing for clustering servers with
independent storage. This even can be done for free using Resonate's SLB
NOW! balancer (free for 2 servers). The other downside of rsync I noticed -
it fails on files being deleted on source system when they are already on
the list to be replicated ( I would expect it to skip these).

For some cases it is feasible to do deletion bypassing rsync ( on all
servers at once ). For example, ftp site for user's downloads only is a good
candidate for this. 

Unison seem to be very promising advance of rsync, which would not have
problem with deletion of files and will skip files deleted during process of
replication, but it has its own shortcomings, one of the big ones - it
cannot handle hardlinks.

Ivan

-Original Message-
From: Michael Salmon [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, June 26, 2001 8:38 AM
To: Kovalev, Ivan; '[EMAIL PROTECTED]'
Subject: Re: 2-way rsync with delete


On Monday, June 25, 2001 03:17:18 PM -0400 "Kovalev, Ivan"
<[EMAIL PROTECTED]> wrote:
+--
| I am doing a "poor man cluster" using rsync to synchronize content of 2
| servers each of which has its own directly attached storage. Since it is a
| cluster (load balancer on top of these 2 servers), new additions as well
| as deletions might appear on any of the 2 servers.
[...]
| The problem would be solved if rsync had a way to do updates compare to
| the known system state. I mean if I could direct it to replicate only the
| changes that were done after time T.
+-X8

What you are describing is unison
<http://www.cis.upenn.edu/~bcpierce/unison/>.


/Michael
--
This space intentionally left non-blank.

Re: 2-way rsync with delete

[Michael Salmon]

On Monday, June 25, 2001 03:17:18 PM -0400 "Kovalev, Ivan"
<[EMAIL PROTECTED]> wrote:
+--
| I am doing a "poor man cluster" using rsync to synchronize content of 2
| servers each of which has its own directly attached storage. Since it is a
| cluster (load balancer on top of these 2 servers), new additions as well
| as deletions might appear on any of the 2 servers.
[...]
| The problem would be solved if rsync had a way to do updates compare to
| the known system state. I mean if I could direct it to replicate only the
| changes that were done after time T.
+-X8

What you are describing is unison
.


/Michael
--
This space intentionally left non-blank.

RE: 2-way rsync with delete

[Kovalev, Ivan]

Thank you, Britton and Dave.

I do use -qaHzu options, but that does not seem to be enough because file
needs to be present on both servers for this to work. If I went through
trouble of catching up any activity on the file system and trigger update on
remote host, I would be as happy as Britton is now with his home - office
setup.

Ivan

-Original Message-
From: Dave Dykstra [mailto:[EMAIL PROTECTED]]
Sent: Monday, June 25, 2001 4:21 PM
To: Britton
Cc: Kovalev, Ivan; '[EMAIL PROTECTED]'
Subject: Re: 2-way rsync with delete


In general Britton is correct.  The only thing that might help Ivan is the
-u option, which works strictly on file modification time.  He didn't
mention having discovered it.

- Dave Dykstra

On Mon, Jun 25, 2001 at 12:01:00PM -0800, Britton wrote:
> 
> I do this sort of thing between my home and work machines, but only one
> system changes at a time, and I do a sync in the appropriate direction at
> the end of each session.
> 
> I think what you want may not be possible, since rsync doesn't maintain
> any database about the files it handles and deleted files have no date
> stamps associated with them.  I think you need full blown distributed file
> system.
> 
> Britton Kerin
> __
> GNU GPL: "The Source will be with you... always."
> 
> On Mon, 25 Jun 2001, Kovalev, Ivan wrote:
> 
> > I am doing a "poor man cluster" using rsync to synchronize content of 2
> > servers each of which has its own directly attached storage. Since it is
a
> > cluster (load balancer on top of these 2 servers), new additions as well
as
> > deletions might appear on any of the 2 servers.
> >
> > Newly added files are replicated just fine, but I need list's wisdom on
how
> > to replicate deletions.
> >
> > Suppose I have a new file on server1 and it is pooling from server2 with
> > --delete option. Then my new file will be deleted because it is not
present
> > on server2. If I do push from server1 to server2 without --delete as a
first
> > step (to make sure new file is there by the time I will pool), then I
will
> > also push old files from server1 which were deleted on server2 since
last
> > synchronization. This will effectively kill replication of deletions.
> >
> > The problem would be solved if rsync had a way to do updates compare to
the
> > known system state. I mean if I could direct it to replicate only the
> > changes that were done after time T.
> >
> > I did not find anything like this in man pages or list's archives for
last
> > few months, but the question seem to be so obvious, that I am probably
> > missing something.
> >
> > Ivan Kovalev
> > Thomson Financial, BFMG / IBES
> > 195 Broadway, 6-th floor
> > New York, NY 10007-3100
> > [EMAIL PROTECTED]
> > tel. (646)822-2939
> > fax.(646)822-2800
> >
> >
> >
> 

Re: 2-way rsync with delete

[Dave Dykstra]

In general Britton is correct.  The only thing that might help Ivan is the
-u option, which works strictly on file modification time.  He didn't
mention having discovered it.

- Dave Dykstra

On Mon, Jun 25, 2001 at 12:01:00PM -0800, Britton wrote:
> 
> I do this sort of thing between my home and work machines, but only one
> system changes at a time, and I do a sync in the appropriate direction at
> the end of each session.
> 
> I think what you want may not be possible, since rsync doesn't maintain
> any database about the files it handles and deleted files have no date
> stamps associated with them.  I think you need full blown distributed file
> system.
> 
> Britton Kerin
> __
> GNU GPL: "The Source will be with you... always."
> 
> On Mon, 25 Jun 2001, Kovalev, Ivan wrote:
> 
> > I am doing a "poor man cluster" using rsync to synchronize content of 2
> > servers each of which has its own directly attached storage. Since it is a
> > cluster (load balancer on top of these 2 servers), new additions as well as
> > deletions might appear on any of the 2 servers.
> >
> > Newly added files are replicated just fine, but I need list's wisdom on how
> > to replicate deletions.
> >
> > Suppose I have a new file on server1 and it is pooling from server2 with
> > --delete option. Then my new file will be deleted because it is not present
> > on server2. If I do push from server1 to server2 without --delete as a first
> > step (to make sure new file is there by the time I will pool), then I will
> > also push old files from server1 which were deleted on server2 since last
> > synchronization. This will effectively kill replication of deletions.
> >
> > The problem would be solved if rsync had a way to do updates compare to the
> > known system state. I mean if I could direct it to replicate only the
> > changes that were done after time T.
> >
> > I did not find anything like this in man pages or list's archives for last
> > few months, but the question seem to be so obvious, that I am probably
> > missing something.
> >
> > Ivan Kovalev
> > Thomson Financial, BFMG / IBES
> > 195 Broadway, 6-th floor
> > New York, NY 10007-3100
> > [EMAIL PROTECTED]
> > tel. (646)822-2939
> > fax.(646)822-2800
> >
> >
> >
> 

Re: 2-way rsync with delete

[Britton]

I do this sort of thing between my home and work machines, but only one
system changes at a time, and I do a sync in the appropriate direction at
the end of each session.

I think what you want may not be possible, since rsync doesn't maintain
any database about the files it handles and deleted files have no date
stamps associated with them.  I think you need full blown distributed file
system.

Britton Kerin
__
GNU GPL: "The Source will be with you... always."

On Mon, 25 Jun 2001, Kovalev, Ivan wrote:

> I am doing a "poor man cluster" using rsync to synchronize content of 2
> servers each of which has its own directly attached storage. Since it is a
> cluster (load balancer on top of these 2 servers), new additions as well as
> deletions might appear on any of the 2 servers.
>
> Newly added files are replicated just fine, but I need list's wisdom on how
> to replicate deletions.
>
> Suppose I have a new file on server1 and it is pooling from server2 with
> --delete option. Then my new file will be deleted because it is not present
> on server2. If I do push from server1 to server2 without --delete as a first
> step (to make sure new file is there by the time I will pool), then I will
> also push old files from server1 which were deleted on server2 since last
> synchronization. This will effectively kill replication of deletions.
>
> The problem would be solved if rsync had a way to do updates compare to the
> known system state. I mean if I could direct it to replicate only the
> changes that were done after time T.
>
> I did not find anything like this in man pages or list's archives for last
> few months, but the question seem to be so obvious, that I am probably
> missing something.
>
> Ivan Kovalev
> Thomson Financial, BFMG / IBES
> 195 Broadway, 6-th floor
> New York, NY 10007-3100
> [EMAIL PROTECTED]
> tel. (646)822-2939
> fax.(646)822-2800
>
>
>

2-way rsync with delete

[Kovalev, Ivan]

I am doing a "poor man cluster" using rsync to synchronize content of 2
servers each of which has its own directly attached storage. Since it is a
cluster (load balancer on top of these 2 servers), new additions as well as
deletions might appear on any of the 2 servers.
 
Newly added files are replicated just fine, but I need list's wisdom on how
to replicate deletions.
 
Suppose I have a new file on server1 and it is pooling from server2 with
--delete option. Then my new file will be deleted because it is not present
on server2. If I do push from server1 to server2 without --delete as a first
step (to make sure new file is there by the time I will pool), then I will
also push old files from server1 which were deleted on server2 since last
synchronization. This will effectively kill replication of deletions.
 
The problem would be solved if rsync had a way to do updates compare to the
known system state. I mean if I could direct it to replicate only the
changes that were done after time T.
 
I did not find anything like this in man pages or list's archives for last
few months, but the question seem to be so obvious, that I am probably
missing something.
 
Ivan Kovalev 
Thomson Financial, BFMG / IBES 
195 Broadway, 6-th floor
New York, NY 10007-3100
[EMAIL PROTECTED] 
tel. (646)822-2939 
fax.(646)822-2800