Rewording Tom's post to align with one of my professors decades ago, "That
will work, but let's do it the right way." Using 'weectl database'
(followed by the proper parameters) is the right way. Humidex is now stored
in the database, and '$alltime.humidex.max' takes a split second to return
the correct result. My Rube Goldberg software machine has been dismantled.
- Paul VE1DX
On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
> I mean this in the gentlest way, but this is basically what the SQLite
> database is doing for you: saving various values in a file where they can
> be retrieved via a general query statement.
>
> On Fri, Jan 19, 2024 at 6:16 AM WindnFog <foc...@gmail.com> wrote:
>
>> I encountered the same problem trying to calculate the all time humidex
>> in the index.html.tmpl template. This was taking about 3 minutes on a Pi
>> 4. Rather than add humidex to the database (probably the best solution), I
>> calculated all-time humidex once a day using a small program and invoking
>> it with cron. I saved it in a text file that I could then use the Cheetah
>> include statement to read it into index.the html.tmpl template. This runs
>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
>> appTemp, a similar approach might work. I was not comfortable with
>> modifying the database because that's not my area of expertise:
>>
>> #!/usr/bin/env python3
>> # -*- coding: utf-8 -*-
>>
>> """
>> Program: WeeWX Humidex calculator
>> Author: Paul M Dunphy
>> Date: 12 December 2023
>> Version: 1.0.0
>> Description: This script sweeps through the Weewx database and
>> calculates the
>> derived parameter humidex for each record. It then
>> returns the
>> maximum humidex and the date/time when it occured. This
>> takes
>> about 1.5 seconds on a 900,000 record database as opposed
>> to 3.5
>> minutes by the WeeWX system if the recommended aggregate
>> values
>> are implemented in the Standard/index.html.tmpl file:
>>
>> $alltime.humidex.max ($alltime.humidex.max.degree_F)
>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>
>> It's not clear why the weeWX system takes so long.
>> Possibly
>> because it's doing many other things at the same time.
>>
>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>> """
>>
>> # Required Libraries
>>
>> import sqlite3
>> import math
>> from datetime import datetime
>> import pytz
>> from pytz import timezone
>>
>>
>> def calculate_humidex(temperature, dewpoint):
>> # Constants for the formula
>> k = 273.15 # Conversion factor for Celsius to Kelvin
>>
>> # Convert temperatures to Kelvin
>> temperature + k
>> dewpoint_k = dewpoint + k
>>
>> # Calculate vapor pressure in hPa
>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
>> dewpoint_k)))
>>
>> # Humidex calculation
>> humidex = temperature + ((vapor_pressure - 10) * 0.5555)
>> return humidex
>>
>>
>> def get_max_humidex(database_path):
>> # Connect to the WeeWX database
>> conn = sqlite3.connect(database_path)
>> cursor = conn.cursor()
>>
>> # Query for temperature, dewpoint, and dateTime for all records
>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>> cursor.execute(query)
>>
>> # Variables to track the maximum humidex and its date
>> maximum_humidex = None
>> maximum_humidex_date = None
>>
>> for row in cursor:
>> temperature, dewpoint, date_time = row
>> if temperature is not None and dewpoint is not None:
>> humidex = calculate_humidex(temperature, dewpoint)
>> if maximum_humidex is None or humidex > maximum_humidex:
>> maximum_humidex = humidex
>> maximum_humidex_date = date_time
>>
>> conn.close()
>> return maximum_humidex, maximum_humidex_date
>>
>>
>> def convert_utc_to_ast_adt(utc_datetime):
>> """
>> Convert a UTC datetime to AST or ADT depending on the date.
>>
>> Parameters:
>> utc_datetime (datetime): UTC datetime object.
>>
>> Returns:
>> datetime: Datetime object in AST or ADT.
>> """
>> # Define the time zone for AST/ADT
>> ast_tz = timezone('America/Halifax')
>>
>> # Localize the UTC datetime to the AST/ADT timezone
>> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>>
>> return local_datetime
>>
>>
>> def format_date(unix_timestamp):
>> """
>> Format a UNIX timestamp into a human-readable string in AST or ADT.
>>
>> Parameters:
>> unix_timestamp (int): UNIX timestamp.
>>
>> Returns:
>> str: Formatted date string.
>> """
>> # Create a UTC datetime from the UNIX timestamp
>> utc_datetime =
>> datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)
>>
>> # Convert UTC to AST or ADT
>> local_datetime = convert_utc_to_ast_adt(utc_datetime)
>>
>> # Format the local datetime
>> return local_datetime.strftime("%I:%M %p on %d %b %Y")
>>
>>
>> db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
>> max_humidex, max_humidex_date = get_max_humidex(db_path)
>>
>> # Convert Humidex to Fahrenheit
>> max_humidex_f = max_humidex * 9 / 5 + 32
>>
>> # Format and print the result
>> print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
>> {format_date(max_humidex_date)}")
>>
>> # Format the result
>> output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
>> {format_date(max_humidex_date)}"
>>
>> # Write the result to a file
>> output_file_path =
>> '/home/pdunphy/weewx-data/skins/Standard/Standard_all_time_humidex.txt'
>> with open(output_file_path, 'w') as file:
>> file.write(output)
>>
>> print(f"Output written to {output_file_path}")
>>
>>
>>
>> On Thursday, January 18, 2024 at 7:12:14 PM UTC-4 Tom Keffer wrote:
>>
>>> Using Blaine's database, I was able to isolate the performance problems.
>>>
>>> It's in the template records/index.html.tmpl
>>>
>>> Specifically, apparent temperature (appTemp). It does not appear in the
>>> database, so a simple tag such as $alltime.appTemp.max requires searching
>>> the engine database, calculating apparent temperature for each record, then
>>> finding the max value.
>>>
>>> In V4, this would not have even been attempted.
>>>
>>> The solution is to add appTemp to the database.
>>>
>>> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote:
>>>
>>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
>>>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive
>>>> period.
>>>> The historical files dating back to 2014 generated fine.
>>>>
>>>> Unrelated but interesting.....
>>>>
>>>> - this db shows the Belchertown taking too long issue
>>>> - running htop I see it peg two cpus and StdReport aborts due to
>>>> time vs. the Simulator 300 sec on a pi3+
>>>> - Belchertown 'always' takes longer than 300 secs and weewx always
>>>> aborts due to StdReport not completing in 300 secs
>>>> - yet the same installation with 'my' db works fine with no issues,
>>>> so it 'has' to be content of the db being used doesn't it ?
>>>> - I have been unable to get Belchertown to succeed even at 600 secs
>>>> archive period.
>>>>
>>>> Question - why would the db be 'locked' if a skin is reading it in
>>>> order to process its contents ??????
>>>>
>>>> last restart/failure...
>>>>
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
>>>> version 5.0.0
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is
>>>> -0.21 seconds (positive is fast)
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
>>>> 'wx_binding' to database 'weewx.sdb'
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
>>>> backfill of daily summaries
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily
>>>> summaries up to date
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
>>>> packet loop.
>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>>>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>> Generated 8 files for report SeasonsReport in 3.96 seconds
>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator:
>>>> Generated 30 images for report SeasonsReport in 4.80 seconds
>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5
>>>> files to /home/pi/weewx-data/public_html
>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version
>>>> 1.3.1
>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
>>>> exiting. Shutting engine down.
>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
>>>> StdReport thread
>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to
>>>> shut down StdReport thread
>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
>>>> OperationalError exception: database is locked
>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: ****
>>>> Waiting 2 minutes then retrying...
>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>> Generated 12 files for report Belchertown in 617.64 seconds
>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied
>>>> 40 files to /home/pi/weewx-data/public_html/belchertown
>>>>
>>>>
>>>> weewx related open files:
>>>>
>>>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>>>> python3 1633 pi mem REG 179,2
>>>> 217360 272588
>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>
>>>> When its cpus are pegged trying to process Belchertown:
>>>>
>>>> python3 1633 pi cwd DIR 179,2
>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>> python3 1633 pi mem REG 179,2
>>>> 217360 272588
>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi 4uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 pi 5uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 pi 6u REG 179,2
>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>> python3 1633 1853 python3 pi cwd DIR 179,2
>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 217360 272588
>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi 4uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 1853 python3 pi 5uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 1853 python3 pi 6u REG 179,2
>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>>
>>>>
>>>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote:
>>>>
>>>>> Thank you for the reply! Yes, I have tried this many times. It
>>>>> continues to regenerate them with the missing data.
>>>>>
>>>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:
>>>>>
>>>>> Have you tried completely deleting all NOAA reports and letting weewx
>>>>> regenerate them?
>>>>>
>>>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote:
>>>>>
>>>>> Way back in 2020 my NOAA reports stopped populating their daily
>>>>> values. The average values for the month continue to be populated at the
>>>>> bottom of the table. Since 2020 every year or so I have spent hours
>>>>> banging
>>>>> my head against the wall following any and all instructions on the wiki,
>>>>> github, google and this user group to attempt to repair this issue. None
>>>>> of
>>>>> the published instructions seem to help. I have never been successful.
>>>>>
>>>>> If this is a problem that can be solved I would be more than happy to
>>>>> pay someone to help fix this issue knowing full well there are no
>>>>> guarantees recovery is possible. This data is important to me and having
>>>>> essentially lost years worth with no chance of recovery is a real bummer.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to weewx-user+...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to weewx-user+...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-user+...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/b6dcfd7b-f866-4e90-a1a1-40472a833279n%40googlegroups.com.
