Thanks Cameron - I see that now in the log, any pointers on how I could try and find that mem leak?
On Tue, Sep 20, 2022 at 9:33 AM 'Cameron D' via weewx-user < [email protected]> wrote: > looking at the last line of the log, weewx had been allocated VM totalling > 21GB, RSS 14GB. Killing everything else is not going to make much > difference. > > On Tuesday, 20 September 2022 at 11:28:43 pm UTC+10 [email protected] > wrote: > >> You still have the problem that something is eating all of your memory. >> >> Assuming you got the process ID correct, something(s) else will be >> killed. >> >> On Sep 20, 2022, at 3:40 AM, Johnnie Walker <[email protected]> wrote: >> >> >> >> Hi John/Tom - many thanks to you both. >> >> On the server instance: running on normal PC hardware, 16GB RAM, Ubuntu >> Server (think 20.04). I wrote "-1000" to /proc/PROC_ID/oom_score_adj - and >> we'll see what happens from here. Server mainly used for internal websites, >> also runs Ubiquiti Unifi Controller. >> >> Tom - I had thought I needed the WLL to serve data to weewx over the LAN, >> so figured [WeatherLinkLive] in weewx.conf. Have just now checked the file >> and I have both [Vantage] and [WeatherLinkLive] sections configured. It >> sounds as if I could remove the WeatherLink one? Am remembering I used a >> plugin initially to link to the WLL, maybe that's obsolete now. >> >> Both sections copied in below from conf file. >> >> Thanks - JW >> >> >> [Vantage] >> # This section is for the Davis Vantage series of weather stations. >> >> # Connection type: serial or ethernet >> # serial (the classic VantagePro) >> # ethernet (the WeatherLinkIP or Serial-Ethernet bridge) >> type = ethernet >> >> # If the connection type is serial, a port must be specified: >> # Debian, Ubuntu, Redhat, Fedora, and SuSE: >> # /dev/ttyUSB0 is a common USB port name >> # /dev/ttyS0 is a common serial port name >> # BSD: >> # /dev/cuaU0 is a common serial port name >> port = /dev/ttyUSB0 >> >> # If the connection type is ethernet, an IP Address/hostname is >> required: >> host = 192.168.1.210 >> >> ###################################################### >> # The rest of this section rarely needs any attention. >> # You can safely leave it "as is." >> ###################################################### >> >> # Serial baud rate (usually 19200) >> baudrate = 19200 >> >> # TCP port (when using the WeatherLinkIP) >> tcp_port = 22222 >> >> # TCP send delay (when using the WeatherLinkIP): >> tcp_send_delay = 0.5 >> >> # The type of LOOP packet to request: 1 = LOOP1; 2 = LOOP2; 3 = both >> loop_request = 1 >> >> # The id of your ISS station (usually 1). If you use a wind meter >> connected >> # to a anemometer transmitter kit, use its id >> iss_id = 1 >> >> # How long to wait for a response from the station before giving up >> (in >> # seconds; must be greater than 2) >> timeout = 4 >> >> # How long to wait before trying again (in seconds) >> wait_before_retry = 1.2 >> >> # How many times to try before giving up: >> max_tries = 4 >> >> # Vantage model Type: 1 = Vantage Pro; 2 = Vantage Pro2 >> model_type = 2 >> >> # The driver to use: >> driver = weewx.drivers.vantage >> >> ...... >> >> [WeatherLinkLive] >> driver = user.weatherlink_live >> host = 192.168.1.210 >> polling_interval = 10 >> mapping = th:1, th_indoor, baro, rain:1, wind:1, thw:1, thsw:1, >> windchill:1 >> >> >> On Mon, Sep 19, 2022 at 9:38 PM Tom Keffer <[email protected]> wrote: >> >>> I agree with John that this is an out-of-memory problem, but I wonder >>> why you are getting the offset in humidity during the down time. Is this an >>> artifact of using WeatherLinkLive? Indeed, why are you using WLL? Why not >>> just go to the VP2 directly? >>> >>> On Mon, Sep 19, 2022 at 6:25 PM 'John Kline' via weewx-user < >>> [email protected]> wrote: >>> >>>> Correction, the file is /proc/<pid-of-weewx-process>/oom-score-adj >>>> >>>> On Sep 19, 2022, at 6:23 PM, John Kline <[email protected]> wrote: >>>> >>>> >>>> It doesn’t look like you are losing the connection to your Vantage >>>> Pro2. Rather, the kernel is killing WeeWX because you’re running of of >>>> memory. >>>> >>>> You could write -1000 to the /proc/<pid-of-weewx-process>/oom-kill-adj >>>> file to keep it from killing WeeWX, but then something else will be killed >>>> to free up memory. >>>> >>>> What are you running WeeWX on? How much memory? What else is running? >>>> >>>> On Sep 19, 2022, at 6:10 PM, Johnnie Walker <[email protected]> wrote: >>>> >>>> Hi, >>>> I'd had a recurring issue for some months that the weewx service loses >>>> connection to my Davis Vantage Pro2 / WeatherLink. Basic solution has been >>>> to reboot which brings things back into action, although I lose the data >>>> for the period during which the problem occurred. Shows up as breaks in the >>>> charts (for example): >>>> [image: dayhum.png] >>>> I have some breaks that are 1-2 days or more if I was on travel and >>>> didn't realize until I returned. >>>> I just took time to search /var/log/syslog for the instance that >>>> happened today - log is copied in below. And it looks like an out of memory >>>> error, so the process is killed. (Log is grep'd for "weewx" , I copied in >>>> the log info grep'd "oom" further below). The log entries around 16:12 and >>>> after are when I rebooted the server. >>>> I installed weewx well over 2 years ago so can't recollect how I >>>> installed. I can post the weewx.conf file if needed. >>>> Could anyone give me a pointer on how to fix, and is there a way to >>>> patch up the missing data sections? >>>> Thanks very much - JW >>>> >>>> *"weewx"* >>>> >>>> Sep 19 15:38:31 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> poll packet >>>> Sep 19 15:38:31 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> push (broadcast) packet >>>> Sep 19 15:38:38 sparta weewx[1081] message repeated 4 times: [ INFO >>>> user.weatherlink_live: Emitting push (broadcast) packet] >>>> Sep 19 15:38:41 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> poll packet >>>> Sep 19 15:38:41 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> push (broadcast) packet >>>> Sep 19 15:38:51 sparta weewx[1081] message repeated 4 times: [ INFO >>>> user.weatherlink_live: Emitting push (broadcast) packet] >>>> Sep 19 15:38:51 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> poll packet >>>> Sep 19 15:38:53 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> push (broadcast) packet >>>> Sep 19 15:39:01 sparta weewx[1081] message repeated 3 times: [ INFO >>>> user.weatherlink_live: Emitting push (broadcast) packet] >>>> Sep 19 15:39:01 sparta weewx[1081] INFO user.weatherlink_live: Emitting >>>> poll packet >>>> *Sep 19 15:39:02 sparta kernel: [1192066.955832] >>>> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=23c2c8bc241e51af783d9abaf3b8008403f4099db600e63a6284359bda22502e,mems_allowed=0,global_oom,task_memcg=/system.slice/weewx.service,task=python3,pid=1081,uid=0* >>>> Sep 19 16:12:45 sparta systemd[1]: Starting LSB: weewx weather system... >>>> Sep 19 16:12:45 sparta weewx[907]: * Starting weewx weather system >>>> weewx >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Initializing weewx >>>> version 4.8.0 >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Using Python 3.8.10 >>>> (default, Jun 22 2022, 20:18:18) #012[GCC 9.4.0] >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Platform >>>> Linux-5.4.0-125-generic-x86_64-with-glibc2.29 >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Locale is 'C.UTF-8' >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Using configuration >>>> file /etc/weewx/weewx.conf >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Debug is 0 >>>> Sep 19 16:12:46 sparta weewx[1003] INFO __main__: PID file is >>>> /var/run/weewx.pid >>>> Sep 19 16:12:46 sparta weewx[1090] INFO weewx.engine: Loading station >>>> type WeatherLinkLive (user.weatherlink_live) >>>> Sep 19 16:12:46 sparta weewx[907]: ...done. >>>> Sep 19 16:12:46 sparta systemd[1]: Started LSB: weewx weather system. >>>> >>>> *"oom"* >>>> >>>> Sep 19 15:39:02 sparta kernel: [1192066.955659] python3 invoked >>>> oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, >>>> oom_score_adj=0 >>>> Sep 19 15:39:02 sparta kernel: [1192066.955672] >>>> oom_kill_process.cold+0xb/0x10 >>>> Sep 19 15:39:02 sparta kernel: [1192066.955745] [ pid ] uid tgid >>>> total_vm rss pgtables_bytes swapents oom_score_adj name >>>> Sep 19 15:39:02 sparta kernel: [1192066.955832] >>>> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=23c2c8bc241e51af783d9abaf3b8008403f4099db600e63a6284359bda22502e,mems_allowed=0,global_oom,task_memcg=/system.slice/weewx.service,task=python3,pid=1081,uid=0 >>>> Sep 19 15:39:02 sparta kernel: [1192066.955996] Out of memory: Killed >>>> process 1081 (python3) total-vm:21086716kB, anon-rss:14282388kB, >>>> file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:36056kB oom_score_adj:0 >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/6300dba8-2595-45db-b386-465345ef89e2n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/weewx-user/6300dba8-2595-45db-b386-465345ef89e2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> [image: dayhum.png] >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/7F064749-72E2-4343-B079-3D2B980E075A%40johnkline.com >>>> <https://groups.google.com/d/msgid/weewx-user/7F064749-72E2-4343-B079-3D2B980E075A%40johnkline.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "weewx-user" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/weewx-user/yJNYSpx-ihI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/CAPq0zEBDRB7h1QBVOvVFe9Qj_k7WyZDPrqn4Oweix-pqBQ%3DUkA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEBDRB7h1QBVOvVFe9Qj_k7WyZDPrqn4Oweix-pqBQ%3DUkA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/CAM4CCC%2BQ5H0_SeHC%3DvTMgjbkSTjctmweYu06Gh6vdiFqtytXAQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/weewx-user/CAM4CCC%2BQ5H0_SeHC%3DvTMgjbkSTjctmweYu06Gh6vdiFqtytXAQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "weewx-user" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/weewx-user/yJNYSpx-ihI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/01e1792d-a63e-4ea7-af2c-7706f45d0782n%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/01e1792d-a63e-4ea7-af2c-7706f45d0782n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAM4CCCJ3mUX40oTb27HxVtqPwrjSYchJSH9sOBPQ8sEy1fg5HA%40mail.gmail.com.
