Re: Intermittent rsync Issues
On 17/02/2011 10:04, Shai wrote: What was the resulting configuration change (from default) that you had made to fix this? I have reported the same issue today, and this configuration change might be what I'm looking for. In /etc/rsyncd.conf, try changing : max connections = Regards Brian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
This isn't exactly what I'm using.. I'm using rsync over SSH. So I set these in sshd_config MaxSessions 30 MaxStartups 20:60:120 And I wanted to know what he had set to make his rsync finish without further errors. Shai On Thu, Feb 17, 2011 at 12:29 PM, Brian Hickey bhic...@italk.ie wrote: On 17/02/2011 10:04, Shai wrote: What was the resulting configuration change (from default) that you had made to fix this? I have reported the same issue today, and this configuration change might be what I'm looking for. In /etc/rsyncd.conf, try changing : max connections = Regards Brian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
Hi, this might be a problem with maximal number of concurrent ssh connections or connection requests. Ssh daemon has two configuration settings where you can define what is the maximal number of clients which can connect concurrently. This number is by default not very high, so you are probably bumping against that limit. MaxSessions Specifies the maximum number of open sessions permitted per net- work connection. The default is 10. MaxStartups Specifies the maximum number of concurrent unauthenticated con- nections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10. Alternatively, random early drop can be enabled by specifying the three colon separated values ``start:rate:full'' (e.g. 10:30:60). sshd(8) will refuse connection attempts with a probability of ``rate/100'' (30%) if there are currently ``start'' (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches ``full'' (60). izidor On 20.9.2010, at 19:19, J. T. Gray wrote: Greetings, I posted this at serverfault, but in hopes that the people on this list are different from the people there, here it is: Update: 9/20/10: Updated the EC2 AMI on both the client and the server and ran a 3-box test with 2 clients downloading from 1 server over 24 hours. Upon test completion, the logs had zero errors so I began replacing other instances with the updated AMI instances. After a weekend of running the 35-40ish clients, I have logs once again filled with: 2010/09/20 16:27:01 [18581] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] 2010/09/20 16:30:01 [18627] rsync error: unexplained error (code 255) at io.c(601) Is it unreasonable to have 35-40 clients connect to an rsync server simultaneously? Is this possibly a load issue? On Thu, Sep 16, 2010 at 11:04 AM, J. T. Gray j...@intarcorp.com wrote: Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00, and success at 11:01. It takes about 1.7s to execute the rsync command, for what that's worth. That it seems to fail immediately suggests it's something connection-related, but that it fails on rsh and ssh both suggests it's something specific to the client app. I'm currently updating my EC2 image to see if it's specific to the build of Ubuntu I'm using. On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote: Already tried --bwlimit. In my case it didn't change the error. It just slowed down the identical file recognition. But when it came to a file modified from what exist on the backup server, it dropped the connection. Thanks anyway for the help! Vitorio Le 16 sept. 10 à 15:56, Paul Slootman a écrit : On Thu 16 Sep 2010, Mac User FR wrote: I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. I wouldn't be surprised that the extra overhead related to ssh slows the transfer sufficiently to make a flakey network connection not give up. Perhaps try rate-limiting the rsync transfer. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
Izidor, this appears to resolve my problem. Thank you for your input, I will be sure to attribute you on serverfault.com when I add an answer to my question post. On Tue, Sep 21, 2010 at 2:49 AM, Izidor Jerebic ij.l...@gmail.com wrote: Hi, this might be a problem with maximal number of concurrent ssh connections or connection requests. Ssh daemon has two configuration settings where you can define what is the maximal number of clients which can connect concurrently. This number is by default not very high, so you are probably bumping against that limit. MaxSessions Specifies the maximum number of open sessions permitted per net- work connection. The default is 10. MaxStartups Specifies the maximum number of concurrent unauthenticated con- nections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10. Alternatively, random early drop can be enabled by specifying the three colon separated values ``start:rate:full'' (e.g. 10:30:60). sshd(8) will refuse connection attempts with a probability of ``rate/100'' (30%) if there are currently ``start'' (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches ``full'' (60). izidor On 20.9.2010, at 19:19, J. T. Gray wrote: Greetings, I posted this at serverfault, but in hopes that the people on this list are different from the people there, here it is: Update: 9/20/10: Updated the EC2 AMI on both the client and the server and ran a 3-box test with 2 clients downloading from 1 server over 24 hours. Upon test completion, the logs had zero errors so I began replacing other instances with the updated AMI instances. After a weekend of running the 35-40ish clients, I have logs once again filled with: 2010/09/20 16:27:01 [18581] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] 2010/09/20 16:30:01 [18627] rsync error: unexplained error (code 255) at io.c(601) Is it unreasonable to have 35-40 clients connect to an rsync server simultaneously? Is this possibly a load issue? On Thu, Sep 16, 2010 at 11:04 AM, J. T. Gray j...@intarcorp.com wrote: Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00, and success at 11:01. It takes about 1.7s to execute the rsync command, for what that's worth. That it seems to fail immediately suggests it's something connection-related, but that it fails on rsh and ssh both suggests it's something specific to the client app. I'm currently updating my EC2 image to see if it's specific to the build of Ubuntu I'm using. On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote: Already tried --bwlimit. In my case it didn't change the error. It just slowed down the identical file recognition. But when it came to a file modified from what exist on the backup server, it dropped the connection. Thanks anyway for the help! Vitorio Le 16 sept. 10 à 15:56, Paul Slootman a écrit : On Thu 16 Sep 2010, Mac User FR wrote: I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. I wouldn't be surprised that the extra overhead related to ssh slows the transfer sufficiently to make a flakey network connection not give up. Perhaps try rate-limiting the rsync transfer. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
Greetings, I posted this at serverfault, but in hopes that the people on this list are different from the people there, here it is: Update: 9/20/10: Updated the EC2 AMI on both the client and the server and ran a 3-box test with 2 clients downloading from 1 server over 24 hours. Upon test completion, the logs had zero errors so I began replacing other instances with the updated AMI instances. After a weekend of running the 35-40ish clients, I have logs once again filled with: 2010/09/20 16:27:01 [18581] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] 2010/09/20 16:30:01 [18627] rsync error: unexplained error (code 255) at io.c(601) Is it unreasonable to have 35-40 clients connect to an rsync server simultaneously? Is this possibly a load issue? On Thu, Sep 16, 2010 at 11:04 AM, J. T. Gray j...@intarcorp.com wrote: Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00, and success at 11:01. It takes about 1.7s to execute the rsync command, for what that's worth. That it seems to fail immediately suggests it's something connection-related, but that it fails on rsh and ssh both suggests it's something specific to the client app. I'm currently updating my EC2 image to see if it's specific to the build of Ubuntu I'm using. On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote: Already tried --bwlimit. In my case it didn't change the error. It just slowed down the identical file recognition. But when it came to a file modified from what exist on the backup server, it dropped the connection. Thanks anyway for the help! Vitorio Le 16 sept. 10 à 15:56, Paul Slootman a écrit : On Thu 16 Sep 2010, Mac User FR wrote: I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. I wouldn't be surprised that the extra overhead related to ssh slows the transfer sufficiently to make a flakey network connection not give up. Perhaps try rate-limiting the rsync transfer. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
Hi! Interesting subject. Since a week now I'm trying to solve the same problem on a very precise configuration. In my company we have some wired computers and some wireless ones. All of them under Windows XP SP3 with all security updates. They are backed up in a FreeNAS box under FreeNAS 0.7.1, so with rsync server 3.0.7. Clients run under the last cwRsync 4.0.5, so with cygwin 1.7.6 and rsync 3.0.7. The model rsync command used is: rsync -aR --link-dest=/lastest --link-dest=/lastbutone --link- dest=/lastbuttwo --chmod=ugo=rwX directories_to_backup 172.20.30.194::module/backup All computers backup fine, except one. I suspect a Windows update to break some compatibility with it's Netgear WG311v2 PCI wireless card (that have crap drivers). In this way, this computer can do anything on the network (browse, ftp with large files, imap mail, ...) BUT rsync. Well, rsync connection is OK. At the start of the command with -vvv option I can see it's browsing the hierarchy of files and making hard- links of existing files without problem. But, at the first different file found, it sends the file (or at least partially, I can see there is network activity) and then the network connection drops with error 12 as Mr Gray. Before error 12 I can see: rsync: writefd_unbuffered failed to write x bytes and then Connection reset by peer (104) I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. So, I'm now fighting with key pairing between the wireless host and the server to do the rsync backup in batch mode without the need of password. I would like to know, if anybody knows, why the ssh connection is more robust to network problems than rsync protocol and if it would be possible to make rsync more network problems proof. I have nothing against ssh, but I don't need a secure tunneling in my lan. Plain rsync would be far simpler for me. Hope J. T. Gray can profit from my experience that isn't far from his. Best regards, Vitorio Le 15 sept. 10 à 21:10, J. T. Gray a écrit : Greetings, I posted this to a couple online forums already am am already doubting anyone will be able to directly help me solve my problems. So I am here to query the experts directly: I have a number of rsync clients trying to connect to an rsync server routinely, and they're intermittently failing with one of a couple error messages. Either: 2010/09/15 13:45:23 [32143] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] 2010/09/15 13:45:23 [32143] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] Or: 2010/09/15 13:40:01 [7617] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] 2010/09/15 13:40:01 [7617] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] The current version of the rsync command I'm using is: rsync --rsync-path=/usr/bin/rsync --stats --compress --times -- links --log-file=/home/ubuntu/rsynclog.txt --exclude thatfile -- recursive xxx.xx.xxx.xx:/home/ubuntu/utility_scripts/ /home/ubuntu/ utility_scripts I previously had --verbose and --progress but removed them after reading on another forum that someone had resolved some latency issues by removing those options. I've also tried this command in the form of a shell script, thinking perhaps the issue was that my rsync client was attempting to reuse an expired ssh connection. To that end, it fails seemingly at random whether using rsh or ssh. It periodically fails whether or not I do --del or --delete, --compress or not, --rsync-path or not. I cannot get the command to fail from the command line, but when it runs every minute, it fails 5-15 times an hour, depending on the directory being rsync'ed. The permissions and ownership appear to all be correct, and I'm not relying on any sort of environmental variables that would be causing the cron to fail. All of the relevant software packages (bash, rsync, ssh, Linux) are up to date, all key ports are open, and all clients do not fail simultaneously, suggesting this is not a server-side problem. If anyone has any constructive feedback for resolving or troubleshooting this/these issue/s, I am very interested in hearing it. Thanks, JTG -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
On Thu 16 Sep 2010, Mac User FR wrote: I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. I wouldn't be surprised that the extra overhead related to ssh slows the transfer sufficiently to make a flakey network connection not give up. Perhaps try rate-limiting the rsync transfer. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
Already tried --bwlimit. In my case it didn't change the error. It just slowed down the identical file recognition. But when it came to a file modified from what exist on the backup server, it dropped the connection. Thanks anyway for the help! Vitorio Le 16 sept. 10 à 15:56, Paul Slootman a écrit : On Thu 16 Sep 2010, Mac User FR wrote: I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. I wouldn't be surprised that the extra overhead related to ssh slows the transfer sufficiently to make a flakey network connection not give up. Perhaps try rate-limiting the rsync transfer. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00, and success at 11:01. It takes about 1.7s to execute the rsync command, for what that's worth. That it seems to fail immediately suggests it's something connection-related, but that it fails on rsh and ssh both suggests it's something specific to the client app. I'm currently updating my EC2 image to see if it's specific to the build of Ubuntu I'm using. On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote: Already tried --bwlimit. In my case it didn't change the error. It just slowed down the identical file recognition. But when it came to a file modified from what exist on the backup server, it dropped the connection. Thanks anyway for the help! Vitorio Le 16 sept. 10 à 15:56, Paul Slootman a écrit : On Thu 16 Sep 2010, Mac User FR wrote: I read somewhere that the ssh connection was stronger than rsync one in unstable networks. And effectively, if I run rsync via ssh ( -e option), the transfer happen without errors, on the same computer using the same networking setup. I wouldn't be surprised that the extra overhead related to ssh slows the transfer sufficiently to make a flakey network connection not give up. Perhaps try rate-limiting the rsync transfer. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Intermittent rsync Issues
Greetings, I posted this to a couple online forums already am am already doubting anyone will be able to directly help me solve my problems. So I am here to query the experts directly: I have a number of rsync clients trying to connect to an rsync server routinely, and they're intermittently failing with one of a couple error messages. Either: 2010/09/15 13:45:23 [32143] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] 2010/09/15 13:45:23 [32143] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] Or: 2010/09/15 13:40:01 [7617] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] 2010/09/15 13:40:01 [7617] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] The current version of the rsync command I'm using is: rsync --rsync-path=/usr/bin/rsync --stats --compress --times --links --log-file=/home/ubuntu/rsynclog.txt --exclude thatfile --recursive xxx.xx.xxx.xx:/home/ubuntu/utility_scripts/ /home/ubuntu/utility_scripts I previously had --verbose and --progress but removed them after reading on another forum that someone had resolved some latency issues by removing those options. I've also tried this command in the form of a shell script, thinking perhaps the issue was that my rsync client was attempting to reuse an expired ssh connection. To that end, it fails seemingly at random whether using rsh or ssh. It periodically fails whether or not I do --del or --delete, --compress or not, --rsync-path or not. I cannot get the command to fail from the command line, but when it runs every minute, it fails 5-15 times an hour, depending on the directory being rsync'ed. The permissions and ownership appear to all be correct, and I'm not relying on any sort of environmental variables that would be causing the cron to fail. All of the relevant software packages (bash, rsync, ssh, Linux) are up to date, all key ports are open, and all clients do not fail simultaneously, suggesting this is not a server-side problem. If anyone has any constructive feedback for resolving or troubleshooting this/these issue/s, I am very interested in hearing it. Thanks, JTG -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Intermittent rsync Issues
I posted this to a couple online forums already am am already doubting anyone will be able to directly help me solve my problems. So I am here to query the experts directly: I have a number of rsync clients trying to connect to an rsync server routinely, and they're intermittently failing with one of a couple error messages. Either: 2010/09/15 13:45:23 [32143] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] 2010/09/15 13:45:23 [32143] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] Or: 2010/09/15 13:40:01 [7617] rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] 2010/09/15 13:40:01 [7617] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] The current version of the rsync command I'm using is: rsync --rsync-path=/usr/bin/rsync --stats --compress --times --links --log-file=/home/ubuntu/rsynclog.txt --exclude thatfile --recursive xxx.xx.xxx.xx:/home/ubuntu/utility_scripts/ /home/ubuntu/utility_scripts I previously had --verbose and --progress but removed them after reading on another forum that someone had resolved some latency issues by removing those options. I've also tried this command in the form of a shell script, thinking perhaps the issue was that my rsync client was attempting to reuse an expired ssh connection. To that end, it fails seemingly at random whether using rsh or ssh. It periodically fails whether or not I do --del or --delete, --compress or not, --rsync-path or not. I cannot get the command to fail from the command line, but when it runs every minute, it fails 5-15 times an hour, depending on the directory being rsync'ed. The permissions and ownership appear to all be correct, and I'm not relying on any sort of environmental variables that would be causing the cron to fail. All of the relevant software packages (bash, rsync, ssh, Linux) are up to date, all key ports are open, and all clients do not fail simultaneously, suggesting this is not a server-side problem. If anyone has any constructive feedback for resolving or troubleshooting this/these issue/s, I am very interested in hearing it. Typically how long will the command take to execute? You could add the 'time' command before rsync to find out. Hope this helps. - This email is protected by LBackup, an open source backup solution. http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
