Re: rsync replacement

2007-07-18 Thread Jamie Lokier
Jason Haar wrote:
 Jamie Lokier wrote:
  Check out the TCP: advanced congestion control option in a 2.6 Linux
  kernel, and there is plenty of research on the topic.  See SCTP and
  DSCP (among others) for the more transaction oriented side.

 Hi there Jamie
 
 Like yourself, our WAN (VPN over Internet) suffers majorly from large
 pipe, but large latency issues.

The jargon is large bandwidth-delay product.

 e.g. we have a 10Mbs Internet link, but
 can only get ~2Mbs for a single TCP stream (e.g. HTTP, Rsync).

If you aren't suffering from congestion (in the form of significant
latency jitter, or dropped packets), then the limit is due to the TCP
window size, which you can increase.

It is easy to tell if window size is the problem by looking at the
packet stream using Ethereal: every transmit will keep the window full.

Individual applications can increase it using the SO_SNDBUF and
SO_RCVBUF options.  (Use the appropriate option at the sending and
receiving sides :-)

The default (for applications which don't use those options) is
increased in an OS-specific way.  On Linux, you'd write different
values to /proc/sys/net/ipv4/{tcp_wmem,tcp_rmem,tcp_window_scaling}
and possibly other window-related settings.  Googling for TCP window
size and configuring Apache, Samba etc. will probably yield plenty of
suggestions.

If the slowdown is due to congestion or latency jitter, there's a lot
of tuning knobs on Linux and BSD, and the best choice depends on the
types of link you are running over.

You may also be limited by your router _intentionally_ limiting a
single TCP stream.  Or, if your 10Mbs link is implemented by an
aggregation of slower links, the aggregating routers may be mapping
individual TCP streams on to only one of those links (I've seen this,
I think), limiting the speed to one link.

If you have those router limits, all you can do is used multiple
streams.  If you switched to another protocol over UDP, the routers
would probably have the same limitation.  Assuming so, you can work
around it by bonding multiple UDP-based VPNs and run a single TCP stream
over that.  Don't try that until you're sure it's the problem :-)

 Are protocols like SCTP and DSCP really capable of helping in such
 situations? i.e would they improve the throughput of single streamed
 transactions?

No, SCTP and DSCP are not designed for that; they are more aimed at
having multiple concurrent transactions or streams multiplexed into a
single stream of larger packets, without individual delays affecting
every transaction, and sharing congestion control.  (However, you
could implement any of the things which run over UDP, using DSCP
instead and get OS-level congestion management, I think).

-- Jamie
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Andreas Kotes wrote:
 seems like they've implemented something similiar TCP on top of UDP
 which does a seriously better job (the information they provide points
 in that direction). Shame they don't give it to the public for free,
 like they got TCP, UDP, IP, DNS, SMTP, HTTP, ... ... ...
 
Andreas
 
 P.S: I think it's certainly legitimate to make money from your
 invention, but holding back the evolution of the internet like this
 (i.e. when you know we all could do better by what you did if you just
 made an RFC and allowed public use) is just a shame. see you in 70 years
 :(

It's not 70 years, it's 20-something years for a patent, in the US.
(Their patent is only in the US).

It seems unlikely to hold back the internet much, as there are free
alternatives which do a similar job.  If their implementation is
particularly good it can serve as a benchmark for other methods to be
compared with and aim to outperform; no matter how good, it's highly
unlikely that they stumbled across a uniquely magical heuristic.

Check out the TCP: advanced congestion control option in a 2.6 Linux
kernel, and there is plenty of research on the topic.  See SCTP and
DSCP (among others) for the more transaction oriented side.

-- Jamie
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Mike Jackson

Jamie,

We are not claiming superiority, just that we provide performance  
gains over TCP when going over wan or congested networks.  In-fact,  
we have a ftp server set up in Singapore if you would like to compare  
our technology to your ftp solution.  you can find the client  
download on our site.


Mike



On Jul 17, 2007, at 5:03 AM, Jamie Lokier wrote:


Matt McCutchen wrote:

  Does anyone have any experience with 'syncdat' from Data
Expedition?  How does it compare to rsync?


I looked at the syncdat feature list (
http://www.dataexpedition.com/syncdat/features.html ).  Aside from  
the

claim of much better performance, syncdat appears to be equivalent to
a combination of rsync, unison, and ssh.  Mike, am I missing  
something

that syncdat can do but rsync and unison can't?  And is more
information available about the benchmark that found that syncdat is
Up to ten times faster than rsync (
http://www.dataexpedition.com/syncdat/ )?


I looked at their web site, and they're claiming that their patented
MTP/IP protocol is so much superior to TCP/IP over wide area networks
that it makes data transfer faster - due to handling congestion
better.  If you're transferring over a LAN, or uncongested network, it
offers no advantage.

I read their patent; it's written in impenetrable bullshitese, and
seems to resemble a sort of transaction-oriented TCP SACK.  If I had
to trial their program, I'd compare with HTTP, FTP or rsync/similar
over a modern TCP or SCTP implementation with a few of the esoteric
TCP congestion control modules in 2.6 Linux, and ensure I wasn't CPU
bound.

Their FAQ indicates that their program does not compress
automatically, and I saw no indication of a delta-transmission
protocol equivalent to the rsync algorithm.

They have a product which tunnels any TCP/IP stream over their MTP/IP.
I'm guessing running rsync over that would give the same gains as
using their file transfer utility, assuming there are gains, and the
added benefit of rsync delta-transmission.

-- Jamie


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Mike Jackson wrote:
We  are  not  claiming  superiority,  just that we provide performance
gains over TCP when going over wan or congested networks.  In-fact, we
have a ftp server set up in Singapore if you would like to compare our
technology  to your ftp solution.  you can find the client download on
our site.

I'm not really interested enough to do that test, but thank you for
making it possible.  I do in fact have to transfer a lot of data over
constrained WAN links, so I might come back to it but _only_ if the
source is available: all my file transfers involve embedded uClinux
devices, and cannot run standard Linux executables.

Conversely, you might like to test the variety of TCP congestion
control algorithms which you can select in the latest Linux 2.6
kernels, just to get a sense of how the current public offerings
compare with yours.

I saw no mention of delta-transmission as used by rsync in your
program, but I did see your TCP-over-MTP tunnelling product.

So am I right in thinking that using rsync in conjunction with your
tunnelling product as the underlying transport might give better
performance for incremental file transfers than your current client?

(That at least would be on-topic!)

-- Jamie
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Matt McCutchen

On 7/17/07, Jamie Lokier [EMAIL PROTECTED] wrote:

So am I right in thinking that using rsync in conjunction with your
tunnelling product as the underlying transport might give better
performance for incremental file transfers than your current client?


As I understand it, there are three performance-increasing techniques:

1. Delta transmission
2. Sustained full utilization of the connection via better congestion control
3. The ability to act immediately on any packet that reaches the
destination rather than waiting for the next one in sequence (helpful
when congestion is causing significant packet loss)

Rsync gets #1.  Any program running on MTP gets #2.  However, #3 can
only be achieved by a parallelized application protocol running on a
parallelized network protocol (MTP, UDP, etc.).  I infer that syncdat
uses a parallelized application protocol; Mike, please correct me if
this is not the case.

Thus, syncdat gets #2 and #3 but (it seems) not #1.  Rsync running on
a TCP-over-MTP tunnel would get #1 and #2 but not #3.  To get all
three benefits, we would need to make a program that has both delta
transmission like rsync and a parallelized protocol like syncdat and
run the result on a parallelized, congestion-controlling protocol like
MTP.  (Encryption, if any, would have to be done on individual packets
instead of as a stream filter; I think MTP does this already.)  I
would be very interested to see such a program and even more
interested if it had a BitTorrent-like capability to pull pieces of
the data from multiple sources.

Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Matt McCutchen wrote:
 Thus, syncdat gets #2 and #3 but (it seems) not #1.  Rsync running on
 a TCP-over-MTP tunnel would get #1 and #2 but not #3.  To get all
 three benefits, we would need to make a program that has both delta
 transmission like rsync and a parallelized protocol like syncdat and
 run the result on a parallelized, congestion-controlling protocol like
 MTP.  (Encryption, if any, would have to be done on individual packets
 instead of as a stream filter; I think MTP does this already.)  I
 would be very interested to see such a program and even more
 interested if it had a BitTorrent-like capability to pull pieces of
 the data from multiple sources.

I am in fact working towards such a program, though not with
proprietary congestion control. :-)  It's currently in the form of a
distributed database of 650 devices, which I aim to scale up to
internet numbers, and I'm very interested in distributed delta techniques.

-- Jamie
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Matt McCutchen

On 7/17/07, Jamie Lokier [EMAIL PROTECTED] wrote:

I am in fact working towards such a program, though not with
proprietary congestion control. :-)  It's currently in the form of a
distributed database of 650 devices, which I aim to scale up to
internet numbers, and I'm very interested in distributed delta techniques.


I would be interested in more information about this program.  What
exactly do you mean by distributed delta techniques?  Do you mean
techniques for efficiently reconstructing a set of files given access
to multiple servers holding files similar to the ones you want?  That
would be something like BitTorrent plus fancier delta transmission.

Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Charles Marcus

Matt McCutchen, on 7/17/2007 2:50 PM, said the following:

On 7/17/07, Jamie Lokier [EMAIL PROTECTED] wrote:

I am in fact working towards such a program, though not with
proprietary congestion control. :-)  It's currently in the form of a
distributed database of 650 devices, which I aim to scale up to
internet numbers, and I'm very interested in distributed delta 
techniques.



I would be interested in more information about this program.  What
exactly do you mean by distributed delta techniques?  Do you mean
techniques for efficiently reconstructing a set of files given access
to multiple servers holding files similar to the ones you want?  That
would be something like BitTorrent plus fancier delta transmission.


Sounds like something I read about some time ago - let me see... yes, 
here it is... this is *very* cool looking...


http://www.cleversafe.org/wiki/PROJECT_OVERVIEW

--

Best regards,

Charles
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-17 Thread Jason Haar
Jamie Lokier wrote:
 Check out the TCP: advanced congestion control option in a 2.6 Linux
 kernel, and there is plenty of research on the topic.  See SCTP and
 DSCP (among others) for the more transaction oriented side.
   
Hi there Jamie

Like yourself, our WAN (VPN over Internet) suffers majorly from large
pipe, but large latency issues. e.g. we have a 10Mbs Internet link, but
can only get ~2Mbs for a single TCP stream (e.g. HTTP, Rsync).

Are protocols like SCTP and DSCP really capable of helping in such
situations? i.e would they improve the throughput of single streamed
transactions? If so, how would you manhandle rsync (a TCP app) to be
able to use them? Any such thing as a TCP-to-SCTP proxy?

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-16 Thread Matt McCutchen

On 7/13/07, Chris Shoemaker [EMAIL PROTECTED] wrote:

On Fri, Jul 13, 2007 at 03:21:43PM -0400, Mike Jackson wrote:
 Looking for more efficient replication or synchronization solution
 than rsync, take a look at syncdat by www.dataexpedition.com



   Does anyone have any experience with 'syncdat' from Data
Expedition?  How does it compare to rsync?


I looked at the syncdat feature list (
http://www.dataexpedition.com/syncdat/features.html ).  Aside from the
claim of much better performance, syncdat appears to be equivalent to
a combination of rsync, unison, and ssh.  Mike, am I missing something
that syncdat can do but rsync and unison can't?  And is more
information available about the benchmark that found that syncdat is
Up to ten times faster than rsync (
http://www.dataexpedition.com/syncdat/ )?

Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-16 Thread Andreas Kotes
Hello Matt,

* Matt McCutchen [EMAIL PROTECTED] [20070716 21:07]:
 I looked at the syncdat feature list (
 http://www.dataexpedition.com/syncdat/features.html ).  Aside from the
 claim of much better performance, syncdat appears to be equivalent to
 a combination of rsync, unison, and ssh.  Mike, am I missing something
 that syncdat can do but rsync and unison can't?

seems like they've implemented something similiar TCP on top of UDP
which does a seriously better job (the information they provide points
in that direction). Shame they don't give it to the public for free,
like they got TCP, UDP, IP, DNS, SMTP, HTTP, ... ... ...

   Andreas

P.S: I think it's certainly legitimate to make money from your
invention, but holding back the evolution of the internet like this
(i.e. when you know we all could do better by what you did if you just
made an RFC and allowed public use) is just a shame. see you in 70 years
:(

-- 
At the end of knowledge, wisdom begins, and at the end of wisdom, there is no
grief ... but hope.
-- Lloyd Alexander, 30.1.1924-17.5.2007
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-16 Thread Mike Jackson

Hello Matt,

You are correct when you say that syncdat is a rsync or unison like  
product but what sets them apart is the enhanced performance over a  
wide area network that syncdat provides.  By utilizing our MTP  
technology, sycndat is able to scan and send changed files up to ten  
times faster and in some cases even faster depending on the  
configuration and network path.  We use live customer scenarios for  
our benchmarks, which I could forward onto you but what I recommend  
that you do is give the software a try in your environment.  This  
will give you a true benchmark for your situation based on your live  
network environment.


Mike





On Jul 16, 2007, at 3:06 PM, Matt McCutchen wrote:


On 7/13/07, Chris Shoemaker [EMAIL PROTECTED] wrote:

On Fri, Jul 13, 2007 at 03:21:43PM -0400, Mike Jackson wrote:
 Looking for more efficient replication or synchronization solution
 than rsync, take a look at syncdat by www.dataexpedition.com



   Does anyone have any experience with 'syncdat' from Data
Expedition?  How does it compare to rsync?


I looked at the syncdat feature list (
http://www.dataexpedition.com/syncdat/features.html ).  Aside from the
claim of much better performance, syncdat appears to be equivalent to
a combination of rsync, unison, and ssh.  Mike, am I missing something
that syncdat can do but rsync and unison can't?  And is more
information available about the benchmark that found that syncdat is
Up to ten times faster than rsync (
http://www.dataexpedition.com/syncdat/ )?

Matt


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacement

2007-07-16 Thread Matt McCutchen

On 7/16/07, Andreas Kotes [EMAIL PROTECTED] wrote:

seems like they've implemented something similiar TCP on top of UDP
which does a seriously better job (the information they provide points
in that direction).


I'm wondering whether MTP is simply a better stream protocol (so that
rsync, could be run over an MTP connection and probably perform
similarly to syncdat) or to what extent it relies on the
parallelizability of a large file transfer to contain the effects of
packet loss.  There's some information at
http://www.dataexpedition.com/downloads/DEI-MTP-Info.pdf and
http://www.patentstorm.us/patents/7158479-claims.html .

Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-13 Thread Warren Oates

On 7/13/07, Mike Jackson [EMAIL PROTECTED] wrote:

Looking for more efficient replication or synchronization solution than
rsync, take a look at syncdat by www.dataexpedition.com


Mike


--
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html



Open source is it, Mike?
--
W. Oates
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync replacement

2007-07-13 Thread Chris Shoemaker
On Fri, Jul 13, 2007 at 03:21:43PM -0400, Mike Jackson wrote:
 Looking for more efficient replication or synchronization solution  
 than rsync, take a look at syncdat by www.dataexpedition.com

Hi Mike, 

   You seem to have a misunderstanding about what qualifies as
on-topic for the rsync mailing list.  (Hint: It's not for advertising
your product.)

   Let me attempt to help out by bringing this thread on topic:

Hello List,
   
   Does anyone have any experience with 'syncdat' from Data
Expedition?  How does it compare to rsync?  Are there any ways that rsync
could be improved to be a better replacement for syncdat?  Thanks!

-chris
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html