If I understand the situation correctly, you

   - Are running plain-vanilla wheezy on a stock RPi;
   - Installed weewx using setup.py;
   - Are using an Ultimeter station;
   - Have absolutely no extensions installed (in particular, rtg has been
   disabled);

yet are experiencing memory growth.

Please correct me if any of these assumptions are wrong. If they are
correct, I'll make up an instrumented version of weewxd that will show
which objects are growing in size and number.

-tk



On Tue, Jan 29, 2019 at 5:20 AM Steve2Q <[email protected]> wrote:

> Here is syslog from shortly before the crash.. Is there any other files
> that may be useful for analysis?
>
> Jan 28 23:44:15 raspberrypi weewx[9770]: manager: Added record 2019-01-28
> 23:44:00 EST (1548737040) to database 'weewx.sdb'
> Jan 28 23:44:16 raspberrypi weewx[9770]: manager: Added record 2019-01-28
> 23:44:00 EST (1548737040) to daily summary in 'weewx.sdb'
> Jan 28 23:44:18 raspberrypi weewx[9770]: restx: PWSWeather: Published
> record 2019-01-28 23:44:00 EST (1548737040)
> Jan 28 23:44:26 raspberrypi weewx[9770]: cheetahgenerator: Generated 14
> files for report StandardReport in 8.67 seconds
> Jan 28 23:44:36 raspberrypi weewx[9770]: imagegenerator: Generated 13
> images for StandardReport in 9.49 seconds
> Jan 28 23:44:36 raspberrypi weewx[9770]: copygenerator: copied 0 files to
> /home/weewx/public_html
> Jan 28 23:44:47 raspberrypi weewx[9770]: ftpgenerator: ftp'd 27 files in
> 11.38 seconds
> Jan 28 23:46:15 raspberrypi weewx[9770]: manager: Added record 2019-01-28
> 23:46:00 EST (1548737160) to database 'weewx.sdb'
> Jan 28 23:46:16 raspberrypi weewx[9770]: manager: Added record 2019-01-28
> 23:46:00 EST (1548737160) to daily summary in 'weewx.sdb'
> Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed to fork:
> Cannot allocate memory
> Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed to run
> 'start' task: Cannot allocate memory
> Jan 28 23:46:39 raspberrypi systemd[1]: Failed to start Daily apt download
> activities.
> Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.timer: Adding 1h 13min
> 18.732028s random time.
> Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Unit entered
> failed state.
> Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.timer: Adding 9h 23min
> 52.201118s random time.
> Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed with
> result 'resources'.
> Jan 28 23:48:31 raspberrypi weewx[9770]: engine: Garbage collected 248392
> objects
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633466] top invoked
> oom-killer: gfp_mask=0x14040d0(GFP_KERNEL|__GFP_COMP|__GFP_RECLAIMABLE),
> nodemask=(null),  order=0, oom_score_adj=0
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633480] top cpuset=/
> mems_allowed=0
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633495] CPU: 3 PID: 17293
> Comm: top Tainted: G         C      4.14.87-v7+ #1178
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633497] Hardware name: BCM2835
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633521] [<8010ff30>]
> (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633531] [<8010c174>]
> (show_stack) from [<8078b424>] (dump_stack+0xd4/0x118)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633543] [<8078b424>]
> (dump_stack) from [<80224bac>] (dump_header+0xac/0x208)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633553] [<80224bac>]
> (dump_header) from [<80223f14>] (oom_kill_process+0x478/0x584)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633560] [<80223f14>]
> (oom_kill_process) from [<80224874>] (out_of_memory+0x124/0x334)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633570] [<80224874>]
> (out_of_memory) from [<8022a3b8>] (__alloc_pages_nodemask+0x10cc/0x11c0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633579] [<8022a3b8>]
> (__alloc_pages_nodemask) from [<80275a60>] (new_slab+0x454/0x558)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633587] [<80275a60>]
> (new_slab) from [<802778a4>] (___slab_alloc.constprop.11+0x228/0x2c0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633595] [<802778a4>]
> (___slab_alloc.constprop.11) from [<80277980>]
> (__slab_alloc.constprop.10+0x44/0x90)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633602] [<80277980>]
> (__slab_alloc.constprop.10) from [<80278118>] (kmem_cache_alloc+0x1f4/0x230)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633611] [<80278118>]
> (kmem_cache_alloc) from [<802f764c>] (proc_alloc_inode+0x2c/0x5c)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633622] [<802f764c>]
> (proc_alloc_inode) from [<802a7ca8>] (alloc_inode+0x2c/0xb4)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633631] [<802a7ca8>]
> (alloc_inode) from [<802aa01c>] (new_inode_pseudo+0x18/0x5c)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633639] [<802aa01c>]
> (new_inode_pseudo) from [<802aa07c>] (new_inode+0x1c/0x30)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633646] [<802aa07c>]
> (new_inode) from [<802fb850>] (proc_pid_make_inode+0x24/0xc0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633654] [<802fb850>]
> (proc_pid_make_inode) from [<802fbdc8>] (proc_pident_instantiate+0x2c/0xb0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633660] [<802fbdc8>]
> (proc_pident_instantiate) from [<802fbee8>] (proc_pident_lookup+0x9c/0xf0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633667] [<802fbee8>]
> (proc_pident_lookup) from [<802fbf84>] (proc_tgid_base_lookup+0x20/0x28)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633675] [<802fbf84>]
> (proc_tgid_base_lookup) from [<8029ad60>] (path_openat+0xe0c/0x10c0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633684] [<8029ad60>]
> (path_openat) from [<8029c2c4>] (do_filp_open+0x70/0xd4)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633693] [<8029c2c4>]
> (do_filp_open) from [<80288f80>] (do_sys_open+0x120/0x1d0)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633701] [<80288f80>]
> (do_sys_open) from [<8028905c>] (SyS_open+0x2c/0x30)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633709] [<8028905c>]
> (SyS_open) from [<80108000>] (ret_fast_syscall+0x0/0x28)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633713] Mem-Info:
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633724] active_anon:111538
> inactive_anon:112531 isolated_anon:0
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633724]  active_file:502
> inactive_file:552 isolated_file:32
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633724]  unevictable:0
> dirty:3 writeback:13 unstable:0
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633724]
> slab_reclaimable:1872 slab_unreclaimable:2785
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633724]  mapped:962
> shmem:1453 pagetables:906 bounce:0
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633724]  free:4134
> free_pcp:28 free_cma:435
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633731] Node 0
> active_anon:446152kB inactive_anon:450124kB active_file:2008kB
> inactive_file:2208kB unevictable:0kB isolated(anon):0kB
> isolated(file):128kB mapped:3848kB dirty:12kB writeback:52kB shmem:5812kB
> writeback_tmp:0kB unstable:0kB all_unreclaimable? no
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633742] Normal free:16536kB
> min:16384kB low:20480kB high:24576kB active_anon:446152kB
> inactive_anon:449608kB active_file:2452kB inactive_file:2316kB
> unevictable:0kB writepending:0kB present:970752kB managed:949448kB
> mlocked:0kB kernel_stack:1144kB pagetables:3624kB bounce:0kB free_pcp:72kB
> local_pcp:0kB free_cma:1740kB
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633744] lowmem_reserve[]: 0 0
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633751] Normal: 123*4kB
> (UMEHC) 104*8kB (UEHC) 105*16kB (UEHC) 76*32kB (UEHC) 39*64kB (UEHC)
> 20*128kB (UEH) 16*256kB (UEHC) 2*512kB (H) 1*1024kB (C) 0*2048kB 0*4096kB =
> 16636kB
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633790] 2600 total pagecache
> pages
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633794] 18 pages in swap cache
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633797] Swap cache stats: add
> 967890, delete 967873, find 539252/1220763
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633799] Free swap  = 0kB
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633801] Total swap = 102396kB
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633803] 242688 pages RAM
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633806] 0 pages
> HighMem/MovableOnly
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633808] 5326 pages reserved
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633810] 2048 pages cma
> reserved
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633812] [ pid ]   uid  tgid
> total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633832] [   92]     0
> 92     8780       34      15       0       75             0 systemd-journal
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633839] [  124]     0
> 124     3638       13       8       0      157         -1000 systemd-udevd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633846] [  246]   100
> 246     4320        8       9       0      102             0 systemd-timesyn
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633852] [  294]     0
> 294     5969       12      10       0      245             0 rsyslogd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633857] [  298]   108
> 298     1601       28       7       0       63             0 avahi-daemon
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633863] [  299] 65534
> 299     1324        4       6       0       58             0 thd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633868] [  304]   105
> 304     1629       40       8       0       77          -900 dbus-daemon
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633874] [  326]   108
> 326     1601        0       6       0       77             0 avahi-daemon
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633879] [  336]     0
> 336     1845       10       7       0      101             0 systemd-logind
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633885] [  339]     0
> 339     1325       13       6       0       43             0 cron
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633890] [  424]     0
> 424     2533       16       9       0      119             0 wpa_supplicant
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633897] [  461]     0
> 461      524        0       4       0       31             0 hciattach
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633902] [  465]     0
> 465     1818        0       8       0       84             0 bluetoothd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633907] [  466]     0
> 466     8759        0      13       0      108             0 bluealsa
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633913] [  530]     0
> 530      737        8       6       0       94             0 dhcpcd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633918] [  546]     0
> 546     1470        2       8       0      115             0 login
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633923] [  571]     0
> 571     2552       11       8       0      127         -1000 sshd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633929] [  806]  1000
> 806     2414        2       9       0      178             0 systemd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633935] [  810]  1000
> 810     2815        0       9       0      315             0 (sd-pam)
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633940] [  817]  1000
> 817     1526        2       6       0      316             0 bash
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633945] [  841]     0
> 841     1808        2       8       0       93             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633951] [  845]     0
> 845      877        0       5       0       22             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633957] [  857]     0
> 857     1808        2       7       0       93             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633962] [  861]     0
> 861      877        0       5       0       22             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633968] [ 2317]     0
> 2317    11818       48      18       0      308             0 packagekitd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633974] [ 2321]     0
> 2321    10046       40      14       0      160             0 polkitd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633979] [ 2424]     0
> 2424     1808        2       7       0       93             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633985] [ 2428]     0
> 2428      877        0       5       0       22             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633990] [ 2491]     0
> 2491     1808        2       8       0       93             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.633995] [ 2495]     0
> 2495      877        0       5       0       22             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634001] [ 2570]     0
> 2570     1808        2       7       0       93             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634007] [ 2574]     0
> 2574      877        0       7       0       22             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634012] [ 5138]     0
> 5138     1808        2       8       0       92             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634018] [ 5142]     0
> 5142      877        0       6       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634023] [ 7788]     0
> 7788     1808        2       7       0       92             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634029] [ 7792]     0
> 7792      877        0       6       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634035] [ 7924]     0
> 7924     1808        2       7       0       91             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634040] [ 7928]     0
> 7928      877        0       6       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634045] [ 9770]     0  9770
> 243016   221290     473       0     9449             0 weewxd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634051] [ 9793]     0
> 9793     1808        2       7       0       91             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634056] [ 9797]     0
> 9797      877        0       5       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634061] [ 6926]  1000
> 6926     2028        1       8       0      126             0 top
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634067] [12607]     0
> 12607     1808        0       7       0       92             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634072] [12611]     0
> 12611      877        0       6       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634077] [12661]     0
> 12661     1808        2       8       0       92             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634083] [12665]     0
> 12665      877        0       6       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634089] [12895]     0
> 12895     1808        2       8       0       92             0 sudo
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634094] [12899]     0
> 12899      877        0       6       0       21             0 tail
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634099] [17160]     0
> 17160     2882      184      10       0        2             0 sshd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634105] [17171]  1000
> 17171     2915      195      10       0        3             0 sshd
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634111] [17174]  1000
> 17174     1526      281       7       0       38             0 bash
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634116] [17194]  1000
> 17194     2028      128       9       0        1             0 top
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634122] [17211]  1000
> 17211     2028        0       8       0      129             0 top
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634127] [17290]  1000
> 17290     2028      113      10       0        1             0 top
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634132] [17293]  1000
> 17293     2028      391       8       0        1             0 top
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634140] Out of memory: Kill
> process 9770 (weewxd) score 852 or sacrifice child
> Jan 28 23:48:32 raspberrypi kernel: [3815265.634171] Killed process 9770
> (weewxd) total-vm:972064kB, anon-rss:885160kB, file-rss:0kB, shmem-rss:0kB
> Jan 28 23:48:32 raspberrypi kernel: [3815265.972280] oom_reaper: reaped
> process 9770 (weewxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> Jan 29 00:17:01 raspberrypi CRON[17411]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 01:17:01 raspberrypi CRON[17456]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 02:17:01 raspberrypi CRON[17500]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 03:17:01 raspberrypi CRON[17526]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 04:17:01 raspberrypi CRON[17555]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 05:17:01 raspberrypi CRON[17581]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 06:13:38 raspberrypi systemd[1]: Starting Daily apt upgrade and
> clean activities...
> Jan 29 06:13:41 raspberrypi systemd[1]: Started Daily apt upgrade and
> clean activities.
> Jan 29 06:13:41 raspberrypi systemd[1]: apt-daily-upgrade.timer: Adding
> 24min 23.836691s random time.
> Jan 29 06:13:41 raspberrypi systemd[1]: apt-daily-upgrade.timer: Adding
> 4min 9.519638s random time.
> Jan 29 06:17:02 raspberrypi CRON[17652]: (root) CMD (   cd / && run-parts
> --report /etc/cron.hourly)
> Jan 29 06:25:01 raspberrypi CRON[17667]: (root) CMD (test -x
> /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
> Jan 29 06:25:02 raspberrypi liblogging-stdlog:  [origin
> software="rsyslogd" swVersion="8.24.0" x-pid="294" x-info="
> http://www.rsyslog.com";] rsyslogd was HUPed
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
For more options, visit https://groups.google.com/d/optout.

Reply via email to