Re: Telnet / SSH connection timeout on LAN
On Jul 13, 2015, at 10:34 AM, Andrey Repin wrote:
>
> In my environment, a small touch to the original file cause changes throughout
> the entirety of its stored image. ('cause storage format is actually an
> archive, and a small change here and there in the source file cause massive
> shifts in the resulting image.)
Unless those files are written using either whole-archive compression or
whole-archive encryption, rsync should still be able to find substantial
savings in the transfer with its rolling checksums. rsync won’t be confused by
simple changes like a new byte added to the middle of a file, shifting all
subsequent bytes down by one.
Some “archive” formats do use compression, but in a piecewise fashion, so that
changing one byte of one piece of the archive may cause that entire chunk to
change, but it might not affect any of the others. An example of this is the
Fossil database format.
You can figure out if your archive files work this way by adding -v to your
rsync command. It reports a ratio of the on-disk data size to the transfer
size as “speedup is N”, where N > 1.0 means it is not re-sending the entire
file. The output of --stats gives similar info, more verbosely.
The point I made in the original post, however, is that all this work to save
network bandwidth comes at a disk I/O and CPU cost in the case of rsync,
because it doesn’t have a daemon that can sit around watching for filesystem
change events. The larger the files are with respect to the change sizes, the
greater the waste.
Always-running software like Dropbox avoids much of this cost because it can
watch for those events, and thus only do work when the OS tells it that a
particular file has changed.
I have also left out another disadvantage of rsync: it’s basically a one-way
operation. If you ever need two-way (or N-way) syncing, you’re better off
moving to one of the many alternatives that know how to do this correctly.
Multilateral syncing is surprisingly hard to get right.
I don’t mean to advertise for Dropbox, just to give it as an example that
everyone can relate to.
An alternative that’s open source, more secure, and definitely does pay
attention to the OS’s filesystem event API is SpiderOak. You can see from
their Github contents that they’ve got OS-specific file change notifiers:
https://github.com/SpiderOak
Now contrast Syncthing, which has many of the same virtues, but currently
doesn’t have file change notification built in, causing some third party to
write a helper for Syncthing to fill the gap:
https://syncthing.net/
https://github.com/syncthing/syncthing-inotify/
These tables may be helpful:
https://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
Greetings, Warren Young!
>>> Consider all the disk I/O required. In its default mode, rsync must do a
>>> full directory tree scan on the directory to be transferred, on *both* ends.
>>> For each file with a different mtime or size, it must then recompute all the
>>> hashes in that file, again on both sides.
>>
>> Wrong. In default mode, rsync only care about timestamp and size.
>> It will not go on hashing crusade unless explicitly told to.
> You misunderstood what I wrote. Given a changed mtime or file size, rsync
> then must re-read the file on both sides in order to generate a series of
> checksums on chunks of the file in order to determine which parts of the
> file have changed, so that it transfers only the changes. See:
>
> https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_of_a_file_have_changed
> If you skip this step, you must transfer the entire file contents to the
> other side, since a difference only in the mtime and/or file size doesn’t
> tell you which parts of the file have changed.
In my environment, a small touch to the original file cause changes throughout
the entirety of its stored image. ('cause storage format is actually an
archive, and a small change here and there in the source file cause massive
shifts in the resulting image.)
You see, it is a matter of application and environment.
--
With best regards,
Andrey Repin
Monday, July 13, 2015 19:31:58
Sorry for my terrible english...
Re: Telnet / SSH connection timeout on LAN
On Jul 10, 2015, at 8:38 PM, Andrey Repin wrote: > >> Consider all the disk I/O required. In its default mode, rsync must do a >> full directory tree scan on the directory to be transferred, on *both* ends. >> For each file with a different mtime or size, it must then recompute all the >> hashes in that file, again on both sides. > > Wrong. In default mode, rsync only care about timestamp and size. > It will not go on hashing crusade unless explicitly told to. You misunderstood what I wrote. Given a changed mtime or file size, rsync then must re-read the file on both sides in order to generate a series of checksums on chunks of the file in order to determine which parts of the file have changed, so that it transfers only the changes. See: https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_of_a_file_have_changed If you skip this step, you must transfer the entire file contents to the other side, since a difference only in the mtime and/or file size doesn’t tell you which parts of the file have changed. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
Greetings, Warren Young! > On Jul 9, 2015, at 12:04 AM, Andrey Repin wrote: >> >>> rsync requires a pretty heavy network transaction to figure >>> out if files have changed. >> >> I'm rsync'ing about 15 gigabytes of my home directory with just a few megs of >> network exchange. > “Just?” > That was my definition of “heavy”. Sorry, but you can't sync data without sending data. And these few megs is the actual data I generate daily. > Consider all the disk I/O required. In its default mode, rsync must do a > full directory tree scan on the directory to be transferred, on *both* ends. > For each file with a different mtime or size, it must then recompute all the > hashes in that file, again on both sides. Wrong. In default mode, rsync only care about timestamp and size. It will not go on hashing crusade unless explicitly told to. > Can you really handwave away megs of network I/O and potentially gigs of > disk I/O? Do you never use locate(1) instead of find(1)? Same issue. I normally know where my stuff is, so I don't need to `find` or waste space on `locate` hash tables. > On top of that, the OP wants to do this every time the machine becomes > idle. Even if it was idle a few seconds ago, did some work, and is idle > again, the OP wants all this work to be done all over again. > Horribly wasteful. > I believe Dropbox and its major competitors avoid the need for this tree > scan by hooking into the OS’s filesystem change notification API, so that > they don’t do any network or disk I/O until one of the files it is > responsible for changes. > That’s the right way. VSS would be the right way, but I still did not find time to research its extents. -- With best regards, Andrey Repin Saturday, July 11, 2015 05:35:21 Sorry for my terrible english...
Re: Telnet / SSH connection timeout on LAN
On 10 July 2015 at 07:19, Warren Young wrote: > On Jul 9, 2015, at 12:04 AM, Andrey Repin wrote: >> >>> rsync requires a pretty heavy network transaction to figure >>> out if files have changed. >> >> I'm rsync'ing about 15 gigabytes of my home directory with just a few megs of >> network exchange. > > “Just?” > > That was my definition of “heavy”. > > Consider all the disk I/O required. In its default mode, rsync must do a > full directory tree scan on the directory to be transferred, on *both* ends. > For each file with a different mtime or size, it must then recompute all the > hashes in that file, again on both sides. > > Can you really handwave away megs of network I/O and potentially gigs of disk > I/O? Do you never use locate(1) instead of find(1)? Same issue. > > On top of that, the OP wants to do this every time the machine becomes idle. > Even if it was idle a few seconds ago, did some work, and is idle again, the > OP wants all this work to be done all over again. > > Horribly wasteful. > > I believe Dropbox and its major competitors avoid the need for this tree scan > by hooking into the OS’s filesystem change notification API, so that they > don’t do any network or disk I/O until one of the files it is responsible for > changes. > > That’s the right way. Dropbox etc work well if you have a fast and low latency upstream channel. Trying to backup even small changes over a 1.5 MB DSL (still the most common in the United States) can seriously affect any and all network activity. Heck even with a 20 MB uplink speed you are looking at serious delays to do backups of just general changes to a system. In situations where bandwidth is limited then the local rsync is the way to go. [If you can tie in a dump like behaviour via the OS filesystem change api then even better but local services would need to be what is looked at for a majority of places (at least in the US).] > -- > Problem reports: http://cygwin.com/problems.html > FAQ: http://cygwin.com/faq/ > Documentation: http://cygwin.com/docs.html > Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple > -- Stephen J Smoogen. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
On Jul 9, 2015, at 12:04 AM, Andrey Repin wrote: > >> rsync requires a pretty heavy network transaction to figure >> out if files have changed. > > I'm rsync'ing about 15 gigabytes of my home directory with just a few megs of > network exchange. “Just?” That was my definition of “heavy”. Consider all the disk I/O required. In its default mode, rsync must do a full directory tree scan on the directory to be transferred, on *both* ends. For each file with a different mtime or size, it must then recompute all the hashes in that file, again on both sides. Can you really handwave away megs of network I/O and potentially gigs of disk I/O? Do you never use locate(1) instead of find(1)? Same issue. On top of that, the OP wants to do this every time the machine becomes idle. Even if it was idle a few seconds ago, did some work, and is idle again, the OP wants all this work to be done all over again. Horribly wasteful. I believe Dropbox and its major competitors avoid the need for this tree scan by hooking into the OS’s filesystem change notification API, so that they don’t do any network or disk I/O until one of the files it is responsible for changes. That’s the right way. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
Greetings, Warren Young! > Why are you reinventing these perfectly good wheels, poorly? > Yes, poorly. rsync requires a pretty heavy network transaction to figure > out if files have changed. O.o I'm rsync'ing about 15 gigabytes of my home directory with just a few megs of network exchange. What I'm doing wrong? Either way, the issue has been resolved and was a simple user error. -- With best regards, Andrey Repin Thursday, July 9, 2015 09:02:57 Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
On Jul 6, 2015, at 1:03 PM, Joseph B wrote: > > LAN Adapter > IP: 192.168.0.2 (static) > Subnet Mask: 255.255.255.0 > no DNS server > > Wifi Adapter > 192.168.254.18 (dynamic) > 255.255.255.0 > gateway: 192.168.254.254 > Mask 255.255.255.0 means the first three octets are the network number, and the last is the station number. Thus, your two machines are on *completely different networks*, at a logical level. Your OS is sending the packets from one to your home router, and your router doubtless has no idea that it is supposed to be routing between two different 192.168 networks. Solution: Put the static machine on 192.168.254.x. Look at your DHCP server’s configuration to find a safe value for x. And since you said you have no idea what you’re doing, I will tell you that your LAN’s DHCP server is almost certainly a feature on your home router. DHCP configuration is different for every router. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
On Jul 5, 2015, at 6:35 PM, Spinfusion wrote: > > To set up rsync over SSH, to sync my org-mode GTD files, activated every > time my laptop or pc goes idle, as described by Gina Trapani. If you’re referring to the following article, it was written about a month before Dropbox went live. http://lifehacker.com/196122/ Since then, Google, Apple, Microsoft and others have jumped onto the free cloud storage bandwagon. All of those services address every one of the advantages Gina gives for her solution, except open source, and for that, you’ve got the likes of tarsnap: http://www.tarsnap.com/ Why are you reinventing these perfectly good wheels, poorly? Yes, poorly. rsync requires a pretty heavy network transaction to figure out if files have changed. These other services monitor local files and initiate the delta transfer only when they see a local change. That cost is best paid as rarely as possible; “on every idle” is too often. Anyway, how is futzing around with DIY backup schemes Getting Things Done? :) > ssh error: connection timed out The remote machine either: a) isn’t running ssh; verify with “netstat -na | grep 22.*LISTEN" b) is running it, but it’s firewalled; just because you think the firewall is down doesn’t mean it is; there could be two or more firewalls, for example c) it’s on another port; check the /etc/sshd_config file, unless you know for a fact that you never touched it Failing all that, what does ssh -vvv say? > I don't know what I'm doing and would appreciate some guidance. Did you set Cygwin sshd up using ssh-host-config? That should be all the guidance you require. Maybe run it again and pay more attention to the things it tells you this time? :) -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
Hi Andrew, Desktop LAN Adapter IP: 192.168.0.2 (static) Subnet Mask: 255.255.255.0 no DNS server Wifi Adapter 192.168.254.18 (dynamic) 255.255.255.0 gateway: 192.168.254.254 Laptop LAN 192.168.0.1 (static) 255.255.255.0 no DNS Wifi 192.168.254.19 (dynamic) 255.255.255.0 192.168.254.254 I don't think these namespaces overlap? Note that my ethernet cable is standard 5e, not crossover. I am directly wiring the computers without a switch or hub. This is not supposed to work, and indeed it doesn't work very well. I am content at this point to use wifi's slower transfer speed, since the big 250 GB transfer is done. So I would rather stop using the ethernet cable, as it is a likely failure point. Testing with ethernet cable only, wifi disabled on both: I re-attached the ethernet cable. After manually changing the network type from "public" to "private (work)" on both computers, I was able to access Windows shared folders, as before. Network is still labeled "unidentified" on both machines, as before. Synctoy runs about the same. It finishes the 250 GB sync with about 1000 errors. Not sure whether it's faster; maybe. Cygwin ping works from laptop to desktop. It sends 4 packets, time 0 ms. However, from desktop to laptop, it sends 13 successful pings averaging .5 ms and then freezes. It never presents the statistical summary. Ctrl-c does not break the loop. Had to restart the Cygwin window. I did not observe this behavior before. Ping from desktop windows CLI succeeds normally. Cygwin telnet fails to cross the cable both ways, with error "unable to connect to remote host: Connection timed out". Telnetting localhost gives "connection refused". I don't know how to set up a telnet server. Cygwin ssh to localhost / ip address succeeds. E.g. "ssh localhost" Connection to other machine fails with error "connect to host 192.168.0.x port 22: Connection timed out". Command was e.g. "ssh 192.168.0.1" Thanks for the advice. Please let me know what to do next. Adieu. On Mon, Jul 6, 2015 at 10:26 AM, Andrey Repin wrote: > Greetings, Spinfusion! > >> My goal: >> To set up rsync over SSH, to sync my org-mode GTD files, activated every >> time my laptop or pc goes idle, as described by Gina Trapani. > >> Network hardware architecture: >> Windows 7 laptop and desktop, connected directly by cat 5e ethernet cable. >> Also both connected via wifi to the same router. > > What are assigned IP addresses on both links? > Your further hints indicate that you have name resolution conflict due to > overlapping network address space. > Disconnect from WiFi on both hosts and try only cable. > >> Network functionality: >> Windows firewalls currently disabled >> Windows shared folders works for directory browsing and editing >> Synctoy worked to sync drives, although it spewed a lot of errors and took a >> long time and many retries to sync ~250 gb. Did it over the ethernet cable >> because wifi was too slow. > >> Cygwin functionality: >> ping works. >> telnet error: connection timed out >> ssh error: connection timed out >> ssh connection to localhost - works > >> Background and guesses: > >> I've read and tried many fixes with no luck, to the best of my limited >> understanding. > >> I doubt network names are configured correctly; I'm out of my depth there. >> Pinging works with fixed IP addresses assigned for the ethernet connection. > >> I don't know what I'm doing and would appreciate some guidance. Cheers. > > > -- > With best regards, > Andrey Repin > Monday, July 6, 2015 18:24:38 > > Sorry for my terrible english... > -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
Greetings, Spinfusion! > My goal: > To set up rsync over SSH, to sync my org-mode GTD files, activated every > time my laptop or pc goes idle, as described by Gina Trapani. > Network hardware architecture: > Windows 7 laptop and desktop, connected directly by cat 5e ethernet cable. > Also both connected via wifi to the same router. What are assigned IP addresses on both links? Your further hints indicate that you have name resolution conflict due to overlapping network address space. Disconnect from WiFi on both hosts and try only cable. > Network functionality: > Windows firewalls currently disabled > Windows shared folders works for directory browsing and editing > Synctoy worked to sync drives, although it spewed a lot of errors and took a > long time and many retries to sync ~250 gb. Did it over the ethernet cable > because wifi was too slow. > Cygwin functionality: > ping works. > telnet error: connection timed out > ssh error: connection timed out > ssh connection to localhost - works > Background and guesses: > I've read and tried many fixes with no luck, to the best of my limited > understanding. > I doubt network names are configured correctly; I'm out of my depth there. > Pinging works with fixed IP addresses assigned for the ethernet connection. > I don't know what I'm doing and would appreciate some guidance. Cheers. -- With best regards, Andrey Repin Monday, July 6, 2015 18:24:38 Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Telnet / SSH connection timeout on LAN
On 7/6/2015 2:35 AM, Spinfusion wrote: My goal: To set up rsync over SSH, to sync my org-mode GTD files, activated every time my laptop or pc goes idle, as described by Gina Trapani. Network hardware architecture: Windows 7 laptop and desktop, connected directly by cat 5e ethernet cable. Also both connected via wifi to the same router. Network functionality: Windows firewalls currently disabled Windows shared folders works for directory browsing and editing Synctoy worked to sync drives, although it spewed a lot of errors and took a long time and many retries to sync ~250 gb. Did it over the ethernet cable because wifi was too slow. Are you sure it was over the cable ? A lot of errors means a broken cable or a weak wifi. Cygwin functionality: ping works. telnet error: connection timed out ssh error: connection timed out ssh connection to localhost - works It seems a routing table problem, not a cygwin issue. see windows command route /help Background and guesses: I've read and tried many fixes with no luck, to the best of my limited understanding. I doubt network names are configured correctly; I'm out of my depth there. Pinging works with fixed IP addresses assigned for the ethernet connection. I don't know what I'm doing and would appreciate some guidance. Cheers. This can give you some understanding http://www.sevenforums.com/tutorials/320978-static-routes-create-remove.html Regards Marco -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Telnet / SSH connection timeout on LAN
My goal: To set up rsync over SSH, to sync my org-mode GTD files, activated every time my laptop or pc goes idle, as described by Gina Trapani. Network hardware architecture: Windows 7 laptop and desktop, connected directly by cat 5e ethernet cable. Also both connected via wifi to the same router. Network functionality: Windows firewalls currently disabled Windows shared folders works for directory browsing and editing Synctoy worked to sync drives, although it spewed a lot of errors and took a long time and many retries to sync ~250 gb. Did it over the ethernet cable because wifi was too slow. Cygwin functionality: ping works. telnet error: connection timed out ssh error: connection timed out ssh connection to localhost - works Background and guesses: I've read and tried many fixes with no luck, to the best of my limited understanding. I doubt network names are configured correctly; I'm out of my depth there. Pinging works with fixed IP addresses assigned for the ethernet connection. I don't know what I'm doing and would appreciate some guidance. Cheers. -- View this message in context: http://cygwin.1069669.n5.nabble.com/Telnet-SSH-connection-timeout-on-LAN-tp119480.html Sent from the Cygwin list mailing list archive at Nabble.com. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
