Richard, just curious do you have an Acurite weather station and using Belcherstown? I'm just trying to see if we have the same type of setup before going back to V5 and trying your solution. Thanks!
On Friday, January 19, 2024 at 5:49:18 AM UTC-5 Richard Whitcombe wrote: > FWIW i just needed to add appTemp and its worked - report generation now > takes 3 seconds not 190. > I would caveat that by saying im not using the other missing columns at > all as my station doesnt provide the data so maybe if you are they need to > be added too. > Missing data calculations took about 8 mins for 2017 - 2024 on a Pi4. > > On Friday 19 January 2024 at 15:59:15 UTC+9 vince wrote: > >> Re: the Belchertown issue many seem to be facing, I notice that Blaine's >> db uses the original old wview schema, not the newer bigger >> 'wview_extended' schema that came out a few years ago. I suspect this is >> the pattern we've missed. Folks who have the old schema do not have any of >> the elements for many of the Extended Observations in Belchertown's >> skin.conf file. >> >> Blaine's db schema misses these items in Belchertown's skin.conf extended >> observations list.... >> appTemp = Apparent Temperature >> cloudbase = Cloud Base >> visibility = Visibility >> windrun = Wind Run >> cloud_cover = Cloud Cover >> aqi = AQI >> >> The wview_extended schema is missing these, so I'd expect users with data >> in those somehow might have the same slowness issue until they add db >> elements and calc-missing to generate summary tables for them..... >> visibility = Visibility >> cloud_cover = Cloud Cover >> >> Re: Michael's question about what to do about the NOAA files, I'd suggest >> pre-calculating them offline and dropping them into place in his NOAA >> subdirectory with the appropriate owner/group/mode so they don't have to be >> recalculated. I'd omit the current month+year which should regenerate >> quickly on weewx startup. That would be a good thing to try. We've >> verified the db is ok, so it 'should' work on Blaine's system too >> (hopefully). The test then would be to simply look at the date+time last >> modified for the current month and year NOAA file. It should change every >> archive period as StdReport runs. >> >> (reminder - if you have Belchertown 'and' Seasons, you need to drop the >> pre-calculated NOAA files into both places to preseed both skin output >> directories) >> >> On Thursday, January 18, 2024 at 8:36:20 PM UTC-8 michael.k...@gmx.at >> wrote: >> >>> Or, to speed things up, do the "calc-missing" on another machine with a >>> more potent CPU. >>> >>> Back to Blaine's problem: if the NOAA files are generated on other >>> installations with his database, there is obviously another problem with >>> his installation.I don't have an idea what to look for, if deleted NOAA >>> data will be regenerated in the same way, as before. >>> >>> vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1: >>> >>>> Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ >>>> but what I did for a pip installation using Blaine's db was: >>>> >>>> source ~/weewx-venv/bin/activate >>>> weectl database add-column appTemp >>>> weectl calc-missing >>>> (answer y when prompted above) >>>> >>>> If you have a packaged install of weewx you can skip the activate step >>>> above. >>>> >>>> Note - the calc-missing takes quite a while and will peg your cpu while >>>> working, so if you have a lot of data so you might need to use the --from >>>> and --to options or even the --tranche option to split up the processing a >>>> bit into pieces. I got lucky with this db on pi3plus with a bit of >>>> patience waiting for calc-missing to complete.... >>>> >>>> (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y >>>> Using configuration file /home/pi/weewx-data/weewx.conf >>>> Missing derived observations will be calculated for all records. >>>> Calculating missing derived observations... >>>> Processing record: 959536; Last record: 2024-01-19 00:00:00 PST >>>> (1705651200) >>>> Recalculating daily summaries... >>>> Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300) >>>> Finished recalculating daily summaries >>>> Missing derived observations calculated in 2986.21 seconds >>>> >>>> And after restarting weewx things look good !!!! >>>> >>>> 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to >>>> database 'weewx.sdb' >>>> 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to daily >>>> summary in 'weewx.sdb' >>>> 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92 >>>> seconds >>>> 2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14 >>>> seconds >>>> 2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html >>>> 2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: INFO >>>> user.belchertown: version 1.3.1 >>>> 2024-01-18T19:45:37.910258-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 16.96 >>>> seconds >>>> 2024-01-18T19:45:37.947603-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.reportengine: Copied 40 files to >>>> /home/pi/weewx-data/public_html/belchertown >>>> >>>> 2024-01-18T19:50:15.410986-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to >>>> database 'weewx.sdb' >>>> 2024-01-18T19:50:15.438866-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to daily >>>> summary in 'weewx.sdb' >>>> 2024-01-18T19:50:18.223483-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.65 >>>> seconds >>>> 2024-01-18T19:50:19.311313-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.08 >>>> seconds >>>> 2024-01-18T19:50:19.313932-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.reportengine: Copied 0 files to /home/pi/weewx-data/public_html >>>> 2024-01-18T19:50:31.266819-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 11.89 >>>> seconds >>>> 2024-01-18T19:50:31.277235-08:00 pi3plus weewxd[1797]: INFO >>>> weewx.reportengine: Copied 3 files to >>>> /home/pi/weewx-data/public_html/belchertown >>>> >>>> So the commands above 'did' fix this particular db. >>>> (thanks Tom) >>>> >>>> On Thursday, January 18, 2024 at 6:21:01 PM UTC-8 Richard Whitcombe >>>> wrote: >>>> >>>>> Can you explain in laymans terms how to do that? >>>>> Do you mean add a new DB field using SQL or similar and leave it blank >>>>> or something else? >>>>> >>>>> Or outline trouble shooting steps to isolate the cause in simple terms >>>>> (ie familiar with linux and the basics of weewx but not the core detailed >>>>> functionality). >>>>> >>>>> On Friday 19 January 2024 at 08:12:14 UTC+9 Tom Keffer wrote: >>>>> >>>>>> Using Blaine's database, I was able to isolate the performance >>>>>> problems. >>>>>> >>>>>> It's in the template records/index.html.tmpl >>>>>> >>>>>> Specifically, apparent temperature (appTemp). It does not appear in >>>>>> the database, so a simple tag such as $alltime.appTemp.max requires >>>>>> searching the engine database, calculating apparent temperature for each >>>>>> record, then finding the max value. >>>>>> >>>>>> In V4, this would not have even been attempted. >>>>>> >>>>>> The solution is to add appTemp to the database. >>>>>> >>>>>> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote: >>>>>> >>>>>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+. >>>>>>> The Seasons skin NOAA file for 2024-01 'is' being updated every >>>>>>> archive period. >>>>>>> The historical files dating back to 2014 generated fine. >>>>>>> >>>>>>> Unrelated but interesting..... >>>>>>> >>>>>>> - this db shows the Belchertown taking too long issue >>>>>>> - running htop I see it peg two cpus and StdReport aborts due to >>>>>>> time vs. the Simulator 300 sec on a pi3+ >>>>>>> - Belchertown 'always' takes longer than 300 secs and weewx >>>>>>> always aborts due to StdReport not completing in 300 secs >>>>>>> - yet the same installation with 'my' db works fine with no >>>>>>> issues, so it 'has' to be content of the db being used doesn't it ? >>>>>>> - I have been unable to get Belchertown to succeed even at 600 >>>>>>> secs archive period. >>>>>>> >>>>>>> Question - why would the db be 'locked' if a skin is reading it in >>>>>>> order to process its contents ?????? >>>>>>> >>>>>>> last restart/failure... >>>>>>> >>>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up >>>>>>> weewx version 5.0.0 >>>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error >>>>>>> is -0.21 seconds (positive is fast) >>>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using >>>>>>> binding 'wx_binding' to database 'weewx.sdb' >>>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting >>>>>>> backfill of daily summaries >>>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily >>>>>>> summaries up to date >>>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting >>>>>>> main packet loop. >>>>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added >>>>>>> record 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb' >>>>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added >>>>>>> record 2024-01-18 13:20:00 PST (1705612800) to daily summary in >>>>>>> 'weewx.sdb' >>>>>>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: >>>>>>> Generated 8 files for report SeasonsReport in 3.96 seconds >>>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator: >>>>>>> Generated 30 images for report SeasonsReport in 4.80 seconds >>>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: >>>>>>> Copied 5 files to /home/pi/weewx-data/public_html >>>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version >>>>>>> 1.3.1 >>>>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added >>>>>>> record 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb' >>>>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added >>>>>>> record 2024-01-18 13:30:00 PST (1705613400) to daily summary in >>>>>>> 'weewx.sdb' >>>>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop >>>>>>> exiting. Shutting engine down. >>>>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting >>>>>>> down StdReport thread >>>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to >>>>>>> shut down StdReport thread >>>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database >>>>>>> OperationalError exception: database is locked >>>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: **** >>>>>>> Waiting 2 minutes then retrying... >>>>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: >>>>>>> Generated 12 files for report Belchertown in 617.64 seconds >>>>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: >>>>>>> Copied 40 files to /home/pi/weewx-data/public_html/belchertown >>>>>>> >>>>>>> >>>>>>> weewx related open files: >>>>>>> >>>>>>> pi@pi3plus:~/weewx-data $ lsof|grep weew >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 217360 272588 >>>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 224592 272587 >>>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 1630340 271670 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imaging.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 2528660 271317 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ >>>>>>> libastro.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 56296 271675 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 93216 271673 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 45064 271812 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ >>>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so >>>>>>> >>>>>>> When its cpus are pegged trying to process Belchertown: >>>>>>> >>>>>>> python3 1633 pi cwd DIR 179,2 >>>>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 217360 272588 >>>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 224592 272587 >>>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 1630340 271670 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imaging.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 2528660 271317 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ >>>>>>> libastro.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 56296 271675 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 93216 271673 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi mem REG 179,2 >>>>>>> 45064 271812 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ >>>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 pi 4uw REG 179,2 >>>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>>>>>> python3 1633 pi 5uw REG 179,2 >>>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>>>>>> python3 1633 pi 6u REG 179,2 >>>>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal >>>>>>> python3 1633 1853 python3 pi cwd DIR 179,2 >>>>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 217360 272588 >>>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 224592 272587 >>>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 1630340 271670 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imaging.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 2528660 271317 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ >>>>>>> libastro.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 56296 271675 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 93216 271673 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 1853 python3 pi mem REG 179,2 >>>>>>> 45064 271812 >>>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ >>>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so >>>>>>> python3 1633 1853 python3 pi 4uw REG 179,2 >>>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>>>>>> python3 1633 1853 python3 pi 5uw REG 179,2 >>>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>>>>>> python3 1633 1853 python3 pi 6u REG 179,2 >>>>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal >>>>>>> >>>>>>> >>>>>>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote: >>>>>>> >>>>>>>> Thank you for the reply! Yes, I have tried this many times. It >>>>>>>> continues to regenerate them with the missing data. >>>>>>>> >>>>>>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote: >>>>>>>> >>>>>>>> Have you tried completely deleting all NOAA reports and letting >>>>>>>> weewx regenerate them? >>>>>>>> >>>>>>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote: >>>>>>>> >>>>>>>> Way back in 2020 my NOAA reports stopped populating their daily >>>>>>>> values. The average values for the month continue to be populated at >>>>>>>> the >>>>>>>> bottom of the table. Since 2020 every year or so I have spent hours >>>>>>>> banging >>>>>>>> my head against the wall following any and all instructions on the >>>>>>>> wiki, >>>>>>>> github, google and this user group to attempt to repair this issue. >>>>>>>> None of >>>>>>>> the published instructions seem to help. I have never been successful. >>>>>>>> >>>>>>>> If this is a problem that can be solved I would be more than happy >>>>>>>> to pay someone to help fix this issue knowing full well there are no >>>>>>>> guarantees recovery is possible. This data is important to me and >>>>>>>> having >>>>>>>> essentially lost years worth with no chance of recovery is a real >>>>>>>> bummer. >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-user" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>>> >>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5bdf2f84-dfb9-4805-b0e1-d2c97b5a639en%40googlegroups.com.
