Why not try running the benchmark (link) <https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation> and see what your numbers look like ?
Since you're running the simulator, stop weewx to quiet things down. The benchmark basically is cpu bound though, but any data would help. You might also look at the output of my mem extension that you have running too, just to see if you're out of memory. If you have a memory leak and you are out of RAM, you would be swapping to SD card which is going to be really slow and the pi can slow down a lot as a result. (I'm running 2x your number of records just fine on a pi4 - takes just a few seconds for my customized variant of the old Standard skin) On Wednesday, November 29, 2023 at 8:12:59 AM UTC-8 Tom Keffer wrote: > The templates are taking 3 minutes or more to run. That could be because > they are very complex, they use tags that require whole database searches, > your disk is on the verge of failure, or the CPUs are busy doing something > else. > > On Wed, Nov 29, 2023 at 7:33 AM Paul Dunphy <[email protected]> wrote: > >> Nov 29 11:13:33 jade python3[362677]: weewx[362677] INFO user.sftp: >> sftpgenerator: transferred 27 files in 5.47 seconds >> Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:15:16 AST (1701270916) to database 'mem.sdb' >> Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:15:16 AST (1701270916) to daily summary in >> 'mem.sdb' >> Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:15:00 AST (1701270900) to database 'weewx.sdb' >> Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:15:00 AST (1701270900) to daily summary in >> 'weewx.sdb' >> Nov 29 11:17:01 jade CRON[539036]: pam_unix(cron:session): session opened >> for user root(uid=0) by (uid=0) >> Nov 29 11:17:01 jade CRON[539037]: (root) CMD (cd / && run-parts --report >> /etc/cron.hourly) >> Nov 29 11:17:01 jade CRON[539036]: pam_unix(cron:session): session closed >> for user root >> Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO >> weewx.cheetahgenerator: Generated 13 files for report StandardReport in >> 191.18 seconds >> Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO >> weewx.imagegenerator: Generated 12 images for report StandardReport in 0.41 >> seconds >> Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO >> weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html >> Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO >> weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds >> Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO >> weewx.imagegenerator: Generated 1 images for report mem in 0.04 seconds >> Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO >> weewx.reportengine: Copied 0 files to >> /home/pdunphy/weewx-data/public_html/mem >> Nov 29 11:18:28 jade python3[362677]: weewx[362677] INFO >> paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1) >> Nov 29 11:18:28 jade python3[362677]: weewx[362677] INFO >> paramiko.transport: Authentication (publickey) successful! >> Nov 29 11:18:30 jade python3[362677]: weewx[362677] INFO >> paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3) >> Nov 29 11:18:34 jade python3[362677]: weewx[362677] INFO >> paramiko.transport.sftp: [chan 0] sftp session closed. >> Nov 29 11:18:34 jade python3[362677]: weewx[362677] INFO user.sftp: >> sftpgenerator: transferred 27 files in 6.95 seconds >> Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:20:16 AST (1701271216) to database 'mem.sdb' >> Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:20:16 AST (1701271216) to daily summary in >> 'mem.sdb' >> Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:20:00 AST (1701271200) to database 'weewx.sdb' >> Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:20:00 AST (1701271200) to daily summary in >> 'weewx.sdb' >> Nov 29 11:23:25 jade python3[362677]: weewx[362677] INFO >> weewx.cheetahgenerator: Generated 13 files for report StandardReport in >> 189.78 seconds >> Nov 29 11:23:25 jade python3[362677]: weewx[362677] INFO >> weewx.imagegenerator: Generated 12 images for report StandardReport in 0.43 >> seconds >> Nov 29 11:23:25 jade python3[362677]: weewx[362677] INFO >> weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html >> Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO >> weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds >> Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO >> weewx.imagegenerator: Generated 1 images for report mem in 0.03 seconds >> Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO >> weewx.reportengine: Copied 0 files to >> /home/pdunphy/weewx-data/public_html/mem >> Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO >> paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1) >> Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO >> paramiko.transport: Authentication (publickey) successful! >> Nov 29 11:23:28 jade python3[362677]: weewx[362677] INFO >> paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3) >> Nov 29 11:23:31 jade python3[362677]: weewx[362677] INFO >> paramiko.transport.sftp: [chan 0] sftp session closed. >> Nov 29 11:23:31 jade python3[362677]: weewx[362677] INFO user.sftp: >> sftpgenerator: transferred 27 files in 5.72 seconds >> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:25:16 AST (1701271516) to database 'mem.sdb' >> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:25:16 AST (1701271516) to daily summary in >> 'mem.sdb' >> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:25:00 AST (1701271500) to database 'weewx.sdb' >> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: >> Added record 2023-11-29 11:25:00 AST (1701271500) to daily summary in >> 'weewx.sdb' >> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO >> weewx.cheetahgenerator: Generated 13 files for report StandardReport in >> 189.62 seconds >> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO >> weewx.imagegenerator: Generated 12 images for report StandardReport in 0.41 >> seconds >> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO >> weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html >> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO >> weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds >> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO >> weewx.imagegenerator: Generated 1 images for report mem in 0.04 seconds >> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO >> weewx.reportengine: Copied 0 files to >> /home/pdunphy/weewx-data/public_html/mem >> Nov 29 11:28:26 jade python3[362677]: weewx[362677] INFO >> paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1) >> Nov 29 11:28:26 jade python3[362677]: weewx[362677] INFO >> paramiko.transport: Authentication (publickey) successful! >> Nov 29 11:28:27 jade python3[362677]: weewx[362677] INFO >> paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3) >> Nov 29 11:28:31 jade python3[362677]: weewx[362677] INFO >> paramiko.transport.sftp: [chan 0] sftp session closed. >> Nov 29 11:28:31 jade python3[362677]: weewx[362677] INFO user.sftp: >> sftpgenerator: transferred 27 files in 5.43 seconds >> >> >> On Wednesday, November 29, 2023 at 11:29:18 AM UTC-4 Tom Keffer wrote: >> >>> Impossible to say without seeing the log. >>> >>> On Wed, Nov 29, 2023 at 6:57 AM Paul Dunphy <[email protected]> wrote: >>> >>>> I am testing v5.0.0b15 in a venv (installed on a Debian-12 based 4 GB >>>> Pi 4B using Vince's script.) I'm using the Simulator to add records to a >>>> copy of my database with about 860,000 records. It's working well, but I >>>> note the new record is added at XX:05, XX:10, etc, and then the CPU usage >>>> jumps up for ~3 minutes. Then it uploads the HTML code to my VPS, and the >>>> CPU drops to idle until the next interval ~2 minutes later. >>>> >>>> Am I correct that the system is sweeping through the database, >>>> calculating the derived parameters' min, max, etc.? With versions before >>>> v5, the upload occurred seconds after the new record was added. >>>> >>>> - Paul VE1DX >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/2feedd15-3641-48d8-a585-301d63d1b201n%40googlegroups.com.
