I just measured the time it takes for me to do the full rsync of
/var/www/html/weewx .
Rersults: real 0m0.438s / user 0m0.050s / sys 0m0.107s
Source was my main computer (5y old mother board) to a RPi400 (Rpi4B witj 4
GB mem, inside a keyboard)
So your connection must be very slow to make it around 5 minutes.
In this case, a possible "trick" could be to use the "compression" option
of rsync. Add -z or --compression in the options of rsync.
The size of all transmitted files is reduced by ~30 %. In facts, the image
files are almost unchanged, but they are all rather small, while the
Index.html and other text files are strongly compressed.
30 % reductionis not much, but is still 3.5 min instead of 5 which would
solve you problem.
Before changing the configurations, you could test just rsync with the
following command on the machine running weewx (not the www server):
time rsync -av /var/www/html/weewx/index.html <www user>@<your
www-server>:
Then remoce index.html on the <www server> and try the same rsync on the
weewx machine, but with compression:
time rsync -avz /var/www/html/weewx/index.html <www user>@<your
www-server>:
Do you get some/sufficient improvement ? I would be interested to see
your results and possibly make some (helpfull ?) comments.
Good luck ! Yours, Paul
Le Sunday, October 29, 2023 à 6:11:05 PM UTC+1, Tom -KQ5S a écrit :
> I appreciate the help but I am getting in over my head. The fact that it
> is intermittent makes me think the setup is OK or else it would fail all
> the time. This is one reason I posted on this group thinking it might be
> something in the weewx code or that my internet provider was blowing smoke.
>
> I did set debug=2 and got the below results. I have not seen this
> particular error message:
>
> Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: ****
> rsync error: error in rsync protocol data stream (code 12) at io.c(228)
> [sender=3.2.3]
>
> ====================================================
> Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: Running
> report 'RSYNC'
> Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: Found
> configuration file /home/weewx/skins/Rsync/skin.conf for report 'RSYNC'
> Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: Cannot
> read localization file /home/weewx/skins/Rsync/lang/en.conf for report
> 'RSYNC': Config file not found: "/home/weewx/skins/Rsync/lang/en.conf".
> Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weewx.reportengine: ****
> Using defaults instead.
> Oct 29 11:25:20 raspberrypi weewx[9966] DEBUG weeutil.rsyncupload:
> rsyncupload: cmd: [['rsync', '--archive', '--stats', '-e', 'ssh -p 2222',
> '/home/weewx/public_html/', ‘xx...@kq5s.com:public_html']]
> Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: rsync
> reported errors. Original command: ['rsync', '--archive', '--stats', '-e',
> 'ssh -p 2222', '/home/weewx/public_html/', ‘xx...@kq5s.com:public_html']
> Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: ****
> kex_exchange_identification: Connection closed by remote host
> Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: ****
> Connection closed by 199.xxx.xxx.xx port 2222
> Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: ****
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> Oct 29 11:25:20 raspberrypi weewx[9966] ERROR weeutil.rsyncupload: ****
> rsync error: error in rsync protocol data stream (code 12) at io.c(228)
> [sender=3.2.3)
> ======================================================
>
> Original from my internet provider.
>
> Regarding SSH, a review of the server's logs shows two repeated
> occurrences for your user ID "weewx". I've copied an example of each error
> below:
>
> *Oct 11 15:37:17 ecbiz246 sshd[55604]: Invalid user weewx from
> xxx.xx.xxx.xx port 60374*
> *Oct 11 15:37:17 ecbiz246 sshd[55604]: Received disconnect from *
> *xxx.xx.xxx.xx** port 60374:11: Bye Bye [preauth]*
> *Oct 11 15:37:17 ecbiz246 sshd[55604]: Disconnected from **xxx.xx.xxx.xx**
> port
> 60374 [preauth]*
>
>
> *Oct 11 06:46:23 ecbiz246 sshd[3762]: drop connection #28
> <https://help.inmotionhosting.com/hc/requests/28> from
> [**xxx.xx.xxx.xx**]:42380
> on [**xxx.xx.xxx.xx**]:2222 past MaxStartups*
> *...*
> *Oct 11 09:30:20 ecbiz246 sshd[3762]: drop connection #27
> <https://help.inmotionhosting.com/hc/requests/27> from
> [x**xxx.xx.xxx.xx**]:47222
> on [**xxx.xx.xxx.xx**]:2222 past MaxStartups*
>
> It appears your Pi is intermittently trying to connect with an incorrect
> SSH user string, and subsequent connections are maxing the limit for
> concurrent SSH connections (MaxStartups), so the excess connections are
> simply dropped by the server. I suggest reviewing the SSH connection string
> for the PI, as well as ensuring it is closing connections correctly.
> Additionally, you will want to ensure you are not opening too many
> concurrent connections for transfer. If the issue persists you may want to
> increase the amount of time between uploads, as we are unable to increase
> the MaxStartups setting on shared server.
>
> On Sunday, October 29, 2023 at 10:39:05 AM UTC-5 cric...@pobox.com wrote:
>
>> Yeah, I need to read slower. :^)
>> You are running weewx as user pi? Default is to run the process as root.
>> So, I'm assuming since you running as pi, you've rync'd your those ssh
>> keys to the remote host?
>>
>> BTW, do you have debug=1 (or 2) in your weewx.conf? That might help shed
>> some light.
>>
>> Please post your [[RSYNC]] config from the [StdReport].
>>
>> The one other thing to try is running the rsync by hand from the command
>> line several times and see
>> if you can reproduce the error and get more messaging.
>>
>> I'm pretty sure weewx only starts one rsync at a time. Looking through
>> the code, I can only find one place
>> where those error messages are generated, and it always prints the
>> command, and should print any of the cmd's output.
>> bin/weeutil/rsyncupload.py: log.error("rsync reported errors. Original
>> command: %s", cmd)
>> I'm going to have to defer to the experts on this one.
>>
>> Chris
>>
>> On Saturday, October 28, 2023 at 9:35:04 AM UTC-6 kq5...@gmail.com wrote:
>>
>>> I’m using the standard 5 minute interval. The files being uploaded are
>>> the files generated by the standard and seasons report and the Saratoga
>>> extension. As you can see the upload takes less than 2 seconds. When I
>>> was using ftp to upload I was getting errors due to the time it took.
>>> Rsync was suggested and it has been working well. I don’t know much about
>>> how weewx and rsync work but my internet host seems to think it is a
>>> software problem.
>>>
>>> It appears your Pi is intermittently trying to connect with an incorrect
>>>>>> SSH user string, and subsequent connections are maxing the limit for
>>>>>> concurrent SSH connections (MaxStartups), so the excess connections are
>>>>>> simply dropped by the server. I suggest reviewing the SSH connection
>>>>>> string
>>>>>> for the PI, as well as ensuring it is closing connections correctly.
>>>>>> Additionally, you will want to ensure you are not opening too many
>>>>>> concurrent connections for transfer. If the issue persists you may want
>>>>>> to
>>>>>> increase the amount of time between uploads, as we are unable to
>>>>>> increase
>>>>>> the MaxStartups setting on shared server.
>>>>>>
>>>>>> On Oct 28, 2023, at 8:54 AM, cric...@pobox.com <cric...@pobox.com>
>>> wrote:
>>>
>>> Just an observation, but that's a LOT of files and data getting
>>> uploaded, and the total time is
>>> close to 5 minutes, which is the default archive interval. That
>>> MaxStartups limit may be set to 1,
>>> so is it possible weewx is starting another upload cycle while the last
>>> one is in progress? You could
>>> try changing this to 600 and see if it helps (in your weewx.conf):
>>>
>>> [StdArchive]
>>> archive_interval = 300
>>>
>>> Chris
>>> On Friday, October 27, 2023 at 3:56:51 PM UTC-6 Tom -KQ5S wrote:
>>>
>>>> It has worked in the past. I am unsure when it started the working/not
>>>> working behavior since I don’t look at the logs that often and the error
>>>> was not showing up in the logwatch file.
>>>>
>>>> I am running as user pi.
>>>>
>>>> version = 4.10.2
>>>>
>>>> Pi OS Bullseye
>>>>
>>>> SKIN_NAME = Rsync
>>>> SKIN_VERSION = 4.10.2
>>>>
>>>> [Generators]
>>>> generator_list = weewx.reportengine.RsyncGenerator
>>>>
>>>>
>>>> The fact that it is intermittent makes me think the authorizations are
>>>> OK. Below is the last couple of iterations where you can see it failed
>>>> and
>>>> then worked
>>>>
>>>> Oct 27 13:55:21 raspberrypi weewx[15431] ERROR weeutil.rsyncupload:
>>>> rsync reported errors.:
>>>>
>>>> Oct 27 14:00:26 raspberrypi weewx[15431] INFO weeutil.rsyncupload:
>>>> rsync'd 149 files (919,702 bytes) in 1.94 seconds
>>>>
>>>> Oct 27 14:05:20 raspberrypi weewx[15431] INFO weeutil.rsyncupload:
>>>> rsync'd 29 files (448,591 bytes) in 1.34 seconds
>>>>
>>>> Oct 27 14:10:22 raspberrypi weewx[15431] INFO weeutil.rsyncupload:
>>>> rsync'd 32 files (453,999 bytes) in 1.28 seconds
>>>>
>>>> Oct 27 14:15:19 raspberrypi weewx[15431] ERROR weeutil.rsyncupload:
>>>> rsync reported errors.
>>>>
>>>> Oct 27 14:20:22 raspberrypi weewx[15431] INFO weeutil.rsyncupload:
>>>> rsync'd 32 files (454,465 bytes) in 1.29 seconds
>>>>
>>>> Oct 27 14:25:19 raspberrypi weewx[15431] ERROR weeutil.rsyncupload:
>>>> rsync reported errors. Original command: ['rsync', '--archive', '--stats',
>>>> '-e', 'ssh -p 2222', '/home/weewx/public_html/', 'kq5...@kq5s.com
>>>> :public_html']
>>>>
>>>> Oct 27 14:30:21 raspberrypi weewx[15431] ERROR weeutil.rsyncupload:
>>>> rsync reported errors
>>>>
>>>> Oct 27 14:35:20 raspberrypi weewx[15431] INFO weeutil.rsyncupload:
>>>> rsync'd 32 files (457,673 bytes) in 1.39 seconds
>>>>
>>>> Oct 27 14:40:22 raspberrypi weewx[15431] INFO weeutil.rsyncupload:
>>>> rsync'd 32 files (457,727 bytes) in 1.28 seconds
>>>>
>>>> On Friday, October 27, 2023 at 8:30:37 AM UTC-5 cric...@pobox.com
>>>> wrote:
>>>>
>>>>> Is this a new setup, or has it worked in the past?
>>>>> What userID is running weewx?
>>>>> What version of weewx are you running?
>>>>> The contents of my rsync/skin.conf is:
>>>>> SKIN_NAME = Rsync
>>>>> SKIN_VERSION = 4.10.2
>>>>>
>>>>> [Generators]
>>>>> generator_list = weewx.reportengine.RsyncGenerator
>>>>>
>>>>> Does rsync of your /home/weewx/public_html to your web hosting
>>>>> service work from the command line?
>>>>> If, as which user?
>>>>>
>>>>> On Friday, October 27, 2023 at 4:20:53 AM UTC-6 kq5...@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>>
>>>>>> On Oct 26, 2023, at 8:24 AM, Tom -KQ5S <kq5...@gmail.com> wrote:
>>>>>>
>>>>>> I am getting the following error from time to time. Rsync will work
>>>>>> fine for a while and then I get the error. I might get the error for a
>>>>>> couple of cycles then it will work again, then the error again. I have
>>>>>> included the reply from my internet host as well. Ideas?
>>>>>>
>>>>>> Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine:
>>>>>> Running report 'RSYNC'
>>>>>> Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine:
>>>>>> Found configuration file /home/weewx/skins/Rsync/skin.conf for report
>>>>>> 'RSYNC'
>>>>>> Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine:
>>>>>> Cannot read localization file /home/weewx/skins/Rsync/lang/en.conf for
>>>>>> report 'RSYNC': Config file not found:
>>>>>> "/home/weewx/skins/Rsync/lang/en.conf".
>>>>>> Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weewx.reportengine:
>>>>>> **** Using defaults instead.
>>>>>> Oct 26 07:50:26 raspberrypi weewx[30090] DEBUG weeutil.rsyncupload:
>>>>>> rsyncupload: cmd: [['rsync', '--archive', '--stats', '-e', 'ssh -p
>>>>>> 2222',
>>>>>> '/home/weewx/public_html/', ‘xx...@kq5s.com]]
>>>>>> Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload:
>>>>>> rsync reported errors. Original command: ['rsync', '--archive',
>>>>>> '--stats',
>>>>>> '-e', 'ssh -p 2222', '/home/weewx/public_html/', ‘xxx...@kq5s.com:xxxx’]
>>>>>> Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload:
>>>>>> **** kex_exchange_identification: Connection closed by remote host
>>>>>> Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload:
>>>>>> **** Connection closed by xxx.xx.xxx.xx port 2222
>>>>>> Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload:
>>>>>> **** rsync: connection unexpectedly closed (0 bytes received so far)
>>>>>> [sender]
>>>>>> Oct 26 07:50:27 raspberrypi weewx[30090] ERROR weeutil.rsyncupload:
>>>>>> **** rsync error: unexplained error (code 255) at io.c(228)
>>>>>> [sender=3.2.3]
>>>>>>
>>>>>> ==========================================================
>>>>>> From Internet host. I'm not sure where the user weewx is coming
>>>>>> from. It is not the user set in my config file for rsync.
>>>>>>
>>>>>> Regarding SSH, a review of the server's logs shows two repeated
>>>>>> occurrences for your user ID "weewx". I've copied an example of each
>>>>>> error
>>>>>> below:
>>>>>>
>>>>>> *Oct 11 15:37:17 ecbiz246 sshd[55604]: Invalid user weewx from
>>>>>> xxx.xx.xxx.xx port 60374*
>>>>>> *Oct 11 15:37:17 ecbiz246 sshd[55604]: Received disconnect from *
>>>>>> *xxx.xx.xxx.xx** port 60374:11: Bye Bye [preauth]*
>>>>>> *Oct 11 15:37:17 ecbiz246 sshd[55604]: Disconnected from *
>>>>>> *xxx.xx.xxx.xx** port 60374 [preauth]*
>>>>>>
>>>>>>
>>>>>> *Oct 11 06:46:23 ecbiz246 sshd[3762]: drop connection #28
>>>>>> <https://help.inmotionhosting.com/hc/requests/28> from [*
>>>>>> *xxx.xx.xxx.xx**]:42380 on [**xxx.xx.xxx.xx**]:2222 past MaxStartups*
>>>>>> *...*
>>>>>> *Oct 11 09:30:20 ecbiz246 sshd[3762]: drop connection #27
>>>>>> <https://help.inmotionhosting.com/hc/requests/27> from [x*
>>>>>> *xxx.xx.xxx.xx**]:47222 on [**xxx.xx.xxx.xx**]:2222 past MaxStartups*
>>>>>>
>>>>>> It appears your Pi is intermittently trying to connect with an
>>>>>> incorrect SSH user string, and subsequent connections are maxing the
>>>>>> limit
>>>>>> for concurrent SSH connections (MaxStartups), so the excess connections
>>>>>> are
>>>>>> simply dropped by the server. I suggest reviewing the SSH connection
>>>>>> string
>>>>>> for the PI, as well as ensuring it is closing connections correctly.
>>>>>> Additionally, you will want to ensure you are not opening too many
>>>>>> concurrent connections for transfer. If the issue persists you may want
>>>>>> to
>>>>>> increase the amount of time between uploads, as we are unable to
>>>>>> increase
>>>>>> the MaxStartups setting on shared server.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx-user+...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-user/038c234b-113e-4ace-9a62-24c896e5bac2n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/weewx-user/038c234b-113e-4ace-9a62-24c896e5bac2n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>>
>>>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-user+...@googlegroups.com.
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/44ac6f5d-f47d-4456-be6f-ddcf1fc1391an%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/44ac6f5d-f47d-4456-be6f-ddcf1fc1391an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/0e340c90-536c-46fe-b1b2-edbbce7337efn%40googlegroups.com.
