Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[vince]

I don't know why, but my reply was deleted by Google.   I did a test and 
your commands look ok and worked for me here, so you might want to check 
your wpm.conf file for errors.  You need a wpm_binding stanza defined under 
DataBindings and that database defined under Databases.

Post your [DataBindings] and [Databases] sections of wpm.conf if you can't 
see any errors.

It should look like the following.  The DataBindings tag's database needs 
to point to a defined tag under Databases

[DataBindings]
[[wpm_binding]]
database = archive_sqlite_wpm
table_name = archive
manager = weewx.manager.DaySummaryManager
schema = schemas.wview_extended.schema

[Databases]
[[archive_sqlite_wpm]]
database_name = wpm.sdb
database_type = SQLite

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/14958239-795d-4c8e-946a-e762bb2030bdn%40googlegroups.com.

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[loeriver]

I tried to follow the proposal of an extension of the DB schema, but as I 
am using a quite different setup (similar to / derived from the Brultech 
monitor) I failed in the step of updating.

My attempt was to add the until now calculated value "heating_COP" to the 
database.

Adding the column worked:

root@localhost:/home/weewx2# ./bin/weectl database add-column heating_COP 
--config=wpm.conf --binding=wpm_binding
Using configuration file wpm.conf
Add new column 'heating_COP' of type 'REAL' to database 'wpm.sdb' (y/n)? y
New column heating_COP of type REAL added to database.

But updating did not work, it insists in using the default binding that I 
do not use:

root@localhost:/home/weewx2# ./bin/weectl database calc-missing 
--config=wpm.conf --binding=wpm_binding
Using configuration file wpm.conf
Missing derived observations will be calculated for all records.
Proceed (y/n)? y
Calculating missing derived observations...
Error: 'Unknown data binding ''wx_binding'''
Perhaps StdWXCalculate is using a different binding. Check configuration 
file [StdWXCalculate] stanza
Nothing done. Aborting.

Any ideas?

vince schrieb am Sonntag, 25. Februar 2024 um 21:01:22 UTC+1:

> This has come up a lot of times vs. the Seasons skin if that's what you're 
> referring to.  v5 calculates things it expects are available in the db as 
> needed, which is slow.  In most cases adding the columns to the schema 
> makes the problem go away.  This seems to be coming up for folks starting 
> in v3 weewx who are using the original wview-compatible schema rather than 
> the wview_extended bigger schema that first appeared in v4 weewx.
>
> You can check to see if you're already running the extended schema in your 
> db already by just counting the number of fields:
>  echo "SELECT count() FROM PRAGMA_TABLE_INFO('archive');" | 
> sqlite3 weewx.sdb
>
> The extended schema will return a count of 114.   The original schema had 
> around 53 or so if I recall correctly.  If you're weewx 3.x you are 
> probably going to see 53 as your answer.
>
> If you are running the small original schema you can either figure out 
> which columns are missing and add just those, or update your db to the 
> newer/bigger schema and not worry it.  Given you have a fast PC to do this 
> on, personally I'd upgrade the db to the big one.  Doesn't take that long 
> if you throw some compute at it.
>
> On Sunday, February 25, 2024 at 11:30:14 AM UTC-8 loeriver wrote:
>
>> Sorry for this late question:
>>
>> I went through this performance issue also and to have an apples to 
>> apples comparison I set up weewx 4.10.2 and 5.0.2 on the same computer and 
>> had it running with a large db (280MB). 
>>
>> For 4.10.2 file generation takes around 0.3s, for 5.0.2 it takes 13s (on 
>> a well-equipped PC, on my Raspi-type system it is >100s).
>>
>> Following the explanation from Tom Keffer ("... In V4, this would not 
>> have even been attempted." ...) I would expect that in the output for 
>> v4.10.2 some data should be missing / empty or different compared to the 
>> 5.0.2 output. But this I do not see: Can somebody point me to the data I 
>> should compare for this?
>>
>> WindnFog schrieb am Samstag, 20. Januar 2024 um 14:14:08 UTC+1:
>>
>>> Rewording Tom's post to align with one of my professors decades ago, 
>>> "That will work, but let's do it the right way." Using 'weectl database' 
>>> (followed by the proper parameters) is the right way. Humidex is now stored 
>>> in the database, and '$alltime.humidex.max' takes a split second to return 
>>> the correct result. My Rube Goldberg software machine has been dismantled.
>>>
>>> - Paul VE1DX
>>>
>>> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>>>
 I mean this in the gentlest way, but this is basically what the SQLite 
 database is doing for you: saving various values in a file where they can 
 be retrieved via a general query statement.

 On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:

> I encountered the same problem trying to calculate the all time 
> humidex in the index.html.tmpl template.  This was taking about 3 minutes 
> on a Pi 4. Rather than add humidex to the database (probably the best 
> solution), I calculated all-time humidex once a day using a small program 
> and invoking it with cron. I saved it in a text file that I could then 
> use 
> the Cheetah include statement to read it into index.the html.tmpl 
> template.  This runs in 2 seconds on a Pi 4 and 1 second on a pi 5; if 
> you 
> have an algorithm for appTemp, a similar approach might work.  I was not 
> comfortable with modifying the database because that's not my area of 
> expertise:
>
> #!/usr/bin/env python3
> # -*- coding: utf-8 -*-
>
> """
> Program:WeeWX Humidex calculator
> Author: Paul M Dunphy
> Date:   12 December 2023
> Version:1.0.0
> Description:

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[vince]

This has come up a lot of times vs. the Seasons skin if that's what you're 
referring to.  v5 calculates things it expects are available in the db as 
needed, which is slow.  In most cases adding the columns to the schema 
makes the problem go away.  This seems to be coming up for folks starting 
in v3 weewx who are using the original wview-compatible schema rather than 
the wview_extended bigger schema that first appeared in v4 weewx.

You can check to see if you're already running the extended schema in your 
db already by just counting the number of fields:
 echo "SELECT count() FROM PRAGMA_TABLE_INFO('archive');" | sqlite3 
weewx.sdb

The extended schema will return a count of 114.   The original schema had 
around 53 or so if I recall correctly.  If you're weewx 3.x you are 
probably going to see 53 as your answer.

If you are running the small original schema you can either figure out 
which columns are missing and add just those, or update your db to the 
newer/bigger schema and not worry it.  Given you have a fast PC to do this 
on, personally I'd upgrade the db to the big one.  Doesn't take that long 
if you throw some compute at it.

On Sunday, February 25, 2024 at 11:30:14 AM UTC-8 loeriver wrote:

> Sorry for this late question:
>
> I went through this performance issue also and to have an apples to apples 
> comparison I set up weewx 4.10.2 and 5.0.2 on the same computer and had it 
> running with a large db (280MB). 
>
> For 4.10.2 file generation takes around 0.3s, for 5.0.2 it takes 13s (on a 
> well-equipped PC, on my Raspi-type system it is >100s).
>
> Following the explanation from Tom Keffer ("... In V4, this would not have 
> even been attempted." ...) I would expect that in the output for v4.10.2 
> some data should be missing / empty or different compared to the 5.0.2 
> output. But this I do not see: Can somebody point me to the data I should 
> compare for this?
>
> WindnFog schrieb am Samstag, 20. Januar 2024 um 14:14:08 UTC+1:
>
>> Rewording Tom's post to align with one of my professors decades ago, 
>> "That will work, but let's do it the right way." Using 'weectl database' 
>> (followed by the proper parameters) is the right way. Humidex is now stored 
>> in the database, and '$alltime.humidex.max' takes a split second to return 
>> the correct result. My Rube Goldberg software machine has been dismantled.
>>
>> - Paul VE1DX
>>
>> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>>
>>> I mean this in the gentlest way, but this is basically what the SQLite 
>>> database is doing for you: saving various values in a file where they can 
>>> be retrieved via a general query statement.
>>>
>>> On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:
>>>
 I encountered the same problem trying to calculate the all time humidex 
 in the index.html.tmpl template.  This was taking about 3 minutes on a Pi 
 4. Rather than add humidex to the database (probably the best solution), I 
 calculated all-time humidex once a day using a small program and invoking 
 it with cron. I saved it in a text file that I could then use the Cheetah 
 include statement to read it into index.the html.tmpl template.  This runs 
 in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm 
 for 
 appTemp, a similar approach might work.  I was not comfortable with 
 modifying the database because that's not my area of expertise:

 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-

 """
 Program:WeeWX Humidex calculator
 Author: Paul M Dunphy
 Date:   12 December 2023
 Version:1.0.0
 Description:This script sweeps through the Weewx database and 
 calculates the 
 derived parameter humidex for each record.  It then 
 returns the
 maximum humidex and the date/time when it occured.  
 This takes
 about 1.5 seconds on a 900,000 record database as 
 opposed to 3.5
 minutes by the WeeWX system if the recommended 
 aggregate values
 are implemented in the Standard/index.html.tmpl file:
 
 $alltime.humidex.max ($alltime.humidex.max.degree_F) 
 $alltime.humidex.maxtime.format($ALLTIMEFMT)

 It's not clear why the weeWX system takes so long.  
 Possibly
 because it's doing many other things at the same time.  

 This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 
 12
 """

 # Required Libraries

 import sqlite3
 import math
 from datetime import datetime
 import pytz
 from pytz import timezone


 def calculate_humidex(temperature, dewpoint):
 # Constants for the formula
 k = 273.15  # Conversion factor for Celsius to Kelvin

 # Convert temperatures to 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[loeriver]

Sorry for this late question:

I went through this performance issue also and to have an apples to apples 
comparison I set up weewx 4.10.2 and 5.0.2 on the same computer and had it 
running with a large db (280MB). 

For 4.10.2 file generation takes around 0.3s, for 5.0.2 it takes 13s (on a 
well-equipped PC, on my Raspi-type system it is >100s).

Following the explanation from Tom Keffer ("... In V4, this would not have 
even been attempted." ...) I would expect that in the output for v4.10.2 
some data should be missing / empty or different compared to the 5.0.2 
output. But this I do not see: Can somebody point me to the data I should 
compare for this?

WindnFog schrieb am Samstag, 20. Januar 2024 um 14:14:08 UTC+1:

> Rewording Tom's post to align with one of my professors decades ago, "That 
> will work, but let's do it the right way." Using 'weectl database' 
> (followed by the proper parameters) is the right way. Humidex is now stored 
> in the database, and '$alltime.humidex.max' takes a split second to return 
> the correct result. My Rube Goldberg software machine has been dismantled.
>
> - Paul VE1DX
>
> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>
>> I mean this in the gentlest way, but this is basically what the SQLite 
>> database is doing for you: saving various values in a file where they can 
>> be retrieved via a general query statement.
>>
>> On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:
>>
>>> I encountered the same problem trying to calculate the all time humidex 
>>> in the index.html.tmpl template.  This was taking about 3 minutes on a Pi 
>>> 4. Rather than add humidex to the database (probably the best solution), I 
>>> calculated all-time humidex once a day using a small program and invoking 
>>> it with cron. I saved it in a text file that I could then use the Cheetah 
>>> include statement to read it into index.the html.tmpl template.  This runs 
>>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for 
>>> appTemp, a similar approach might work.  I was not comfortable with 
>>> modifying the database because that's not my area of expertise:
>>>
>>> #!/usr/bin/env python3
>>> # -*- coding: utf-8 -*-
>>>
>>> """
>>> Program:WeeWX Humidex calculator
>>> Author: Paul M Dunphy
>>> Date:   12 December 2023
>>> Version:1.0.0
>>> Description:This script sweeps through the Weewx database and 
>>> calculates the 
>>> derived parameter humidex for each record.  It then 
>>> returns the
>>> maximum humidex and the date/time when it occured.  This 
>>> takes
>>> about 1.5 seconds on a 900,000 record database as 
>>> opposed to 3.5
>>> minutes by the WeeWX system if the recommended aggregate 
>>> values
>>> are implemented in the Standard/index.html.tmpl file:
>>> 
>>> $alltime.humidex.max ($alltime.humidex.max.degree_F) 
>>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>>
>>> It's not clear why the weeWX system takes so long.  
>>> Possibly
>>> because it's doing many other things at the same time.  
>>>
>>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>>> """
>>>
>>> # Required Libraries
>>>
>>> import sqlite3
>>> import math
>>> from datetime import datetime
>>> import pytz
>>> from pytz import timezone
>>>
>>>
>>> def calculate_humidex(temperature, dewpoint):
>>> # Constants for the formula
>>> k = 273.15  # Conversion factor for Celsius to Kelvin
>>>
>>> # Convert temperatures to Kelvin
>>> temperature + k
>>> dewpoint_k = dewpoint + k
>>>
>>> # Calculate vapor pressure in hPa
>>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 / 
>>> dewpoint_k)))
>>>
>>> # Humidex calculation
>>> humidex = temperature + ((vapor_pressure - 10) * 0.)
>>> return humidex
>>>
>>>
>>> def get_max_humidex(database_path):
>>> # Connect to the WeeWX database
>>> conn = sqlite3.connect(database_path)
>>> cursor = conn.cursor()
>>>
>>> # Query for temperature, dewpoint, and dateTime for all records
>>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>>> cursor.execute(query)
>>>
>>> # Variables to track the maximum humidex and its date
>>> maximum_humidex = None
>>> maximum_humidex_date = None
>>>
>>> for row in cursor:
>>> temperature, dewpoint, date_time = row
>>> if temperature is not None and dewpoint is not None:
>>> humidex = calculate_humidex(temperature, dewpoint)
>>> if maximum_humidex is None or humidex > maximum_humidex:
>>> maximum_humidex = humidex
>>> maximum_humidex_date = date_time
>>>
>>> conn.close()
>>> return maximum_humidex, maximum_humidex_date
>>>
>>>
>>> def convert_utc_to_ast_adt(utc_datetime):
>>> """
>>> Convert a 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[WindnFog]

Rewording Tom's post to align with one of my professors decades ago, "That 
will work, but let's do it the right way." Using 'weectl database' 
(followed by the proper parameters) is the right way. Humidex is now stored 
in the database, and '$alltime.humidex.max' takes a split second to return 
the correct result. My Rube Goldberg software machine has been dismantled.

- Paul VE1DX

On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:

> I mean this in the gentlest way, but this is basically what the SQLite 
> database is doing for you: saving various values in a file where they can 
> be retrieved via a general query statement.
>
> On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:
>
>> I encountered the same problem trying to calculate the all time humidex 
>> in the index.html.tmpl template.  This was taking about 3 minutes on a Pi 
>> 4. Rather than add humidex to the database (probably the best solution), I 
>> calculated all-time humidex once a day using a small program and invoking 
>> it with cron. I saved it in a text file that I could then use the Cheetah 
>> include statement to read it into index.the html.tmpl template.  This runs 
>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for 
>> appTemp, a similar approach might work.  I was not comfortable with 
>> modifying the database because that's not my area of expertise:
>>
>> #!/usr/bin/env python3
>> # -*- coding: utf-8 -*-
>>
>> """
>> Program:WeeWX Humidex calculator
>> Author: Paul M Dunphy
>> Date:   12 December 2023
>> Version:1.0.0
>> Description:This script sweeps through the Weewx database and 
>> calculates the 
>> derived parameter humidex for each record.  It then 
>> returns the
>> maximum humidex and the date/time when it occured.  This 
>> takes
>> about 1.5 seconds on a 900,000 record database as opposed 
>> to 3.5
>> minutes by the WeeWX system if the recommended aggregate 
>> values
>> are implemented in the Standard/index.html.tmpl file:
>> 
>> $alltime.humidex.max ($alltime.humidex.max.degree_F) 
>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>
>> It's not clear why the weeWX system takes so long.  
>> Possibly
>> because it's doing many other things at the same time.  
>>
>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>> """
>>
>> # Required Libraries
>>
>> import sqlite3
>> import math
>> from datetime import datetime
>> import pytz
>> from pytz import timezone
>>
>>
>> def calculate_humidex(temperature, dewpoint):
>> # Constants for the formula
>> k = 273.15  # Conversion factor for Celsius to Kelvin
>>
>> # Convert temperatures to Kelvin
>> temperature + k
>> dewpoint_k = dewpoint + k
>>
>> # Calculate vapor pressure in hPa
>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 / 
>> dewpoint_k)))
>>
>> # Humidex calculation
>> humidex = temperature + ((vapor_pressure - 10) * 0.)
>> return humidex
>>
>>
>> def get_max_humidex(database_path):
>> # Connect to the WeeWX database
>> conn = sqlite3.connect(database_path)
>> cursor = conn.cursor()
>>
>> # Query for temperature, dewpoint, and dateTime for all records
>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>> cursor.execute(query)
>>
>> # Variables to track the maximum humidex and its date
>> maximum_humidex = None
>> maximum_humidex_date = None
>>
>> for row in cursor:
>> temperature, dewpoint, date_time = row
>> if temperature is not None and dewpoint is not None:
>> humidex = calculate_humidex(temperature, dewpoint)
>> if maximum_humidex is None or humidex > maximum_humidex:
>> maximum_humidex = humidex
>> maximum_humidex_date = date_time
>>
>> conn.close()
>> return maximum_humidex, maximum_humidex_date
>>
>>
>> def convert_utc_to_ast_adt(utc_datetime):
>> """
>> Convert a UTC datetime to AST or ADT depending on the date.
>>
>> Parameters:
>> utc_datetime (datetime): UTC datetime object.
>>
>> Returns:
>> datetime: Datetime object in AST or ADT.
>> """
>> # Define the time zone for AST/ADT
>> ast_tz = timezone('America/Halifax')
>>
>> # Localize the UTC datetime to the AST/ADT timezone
>> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>>
>> return local_datetime
>>
>>
>> def format_date(unix_timestamp):
>> """
>> Format a UNIX timestamp into a human-readable string in AST or ADT.
>>
>> Parameters:
>> unix_timestamp (int): UNIX timestamp.
>>
>> Returns:
>> str: Formatted date string.
>> """
>> # Create a UTC datetime from the UNIX timestamp
>> utc_datetime = 
>> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Richard Whitcombe]

I have an Aercus Weathsleuth so not sure (run via the interceptor driver).  
Backup the DB and theres nothing to lose by trying the solution really.
On Saturday 20 January 2024 at 01:33:14 UTC+9 Michael Sanphillipo wrote:

> Richard, just curious do you have an Acurite weather station and using 
> Belcherstown? I'm just trying to see if we have the same type of setup 
> before going back to V5 and trying your solution. Thanks!
>
> On Friday, January 19, 2024 at 5:49:18 AM UTC-5 Richard Whitcombe wrote:
>
>> FWIW i just needed to add appTemp and its worked - report generation now 
>> takes 3 seconds not 190.
>> I would caveat that by saying im not using the other missing columns at 
>> all as my station doesnt provide the data so maybe if you are they need to 
>> be added too.
>> Missing data calculations took about 8 mins for 2017 - 2024 on a Pi4.
>>
>> On Friday 19 January 2024 at 15:59:15 UTC+9 vince wrote:
>>
>>> Re: the Belchertown issue many seem to be facing, I notice that Blaine's 
>>> db uses the original old wview schema, not the newer bigger 
>>> 'wview_extended' schema that came out a few years ago.  I suspect this is 
>>> the pattern we've missed.  Folks who have the old schema do not have any of 
>>> the elements for many of the Extended Observations in Belchertown's 
>>> skin.conf file. 
>>>
>>> Blaine's db schema misses these items in Belchertown's skin.conf 
>>> extended observations list
>>> appTemp  = Apparent Temperature
>>> cloudbase= Cloud Base
>>> visibility   = Visibility
>>> windrun  = Wind Run
>>> cloud_cover  = Cloud Cover
>>> aqi  = AQI
>>>
>>> The wview_extended schema is missing these, so I'd expect users with 
>>> data in those somehow might have the same slowness issue until they add db 
>>> elements and calc-missing to generate summary tables for them.
>>> visibility   = Visibility
>>> cloud_cover  = Cloud Cover
>>>
>>> Re: Michael's question about what to do about the NOAA files, I'd 
>>> suggest pre-calculating them offline and dropping them into place in his 
>>> NOAA subdirectory with the appropriate owner/group/mode so they don't have 
>>> to be recalculated.  I'd omit the current month+year which should 
>>> regenerate quickly on weewx startup.  That would be a good thing to try. 
>>>  We've verified the db is ok, so it 'should' work on Blaine's system too 
>>> (hopefully).  The test then would be to simply look at the date+time last 
>>> modified for the current month and year NOAA file.  It should change every 
>>> archive period as StdReport runs.
>>>
>>> (reminder - if you have Belchertown 'and' Seasons, you need to drop the 
>>> pre-calculated NOAA files into both places to preseed both skin output 
>>> directories)
>>>
>>> On Thursday, January 18, 2024 at 8:36:20 PM UTC-8 michael.k...@gmx.at 
>>> wrote:
>>>
 Or, to speed things up, do the "calc-missing" on another machine with a 
 more potent CPU.

 Back to Blaine's problem: if the NOAA files are generated on other 
 installations with his database, there is obviously another problem with 
 his installation.I don't have an idea what to look for, if deleted NOAA 
 data will be regenerated in the same way, as before.

 vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1:

> Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ 
> but what I did for a pip installation using Blaine's db was:
>
> source ~/weewx-venv/bin/activate
> weectl database add-column appTemp
> weectl calc-missing
> (answer y when prompted above)
>
> If you have a packaged install of weewx you can skip the activate step 
> above.
>
> Note - the calc-missing takes quite a while and will peg your cpu 
> while working, so if you have a lot of data so you might need to use the 
> --from and --to options or even the --tranche option to split up the 
> processing a bit into pieces.  I got lucky with this db on pi3plus with a 
> bit of patience waiting for calc-missing to complete
>
> (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
> Using configuration file /home/pi/weewx-data/weewx.conf
> Missing derived observations will be calculated for all records.
> Calculating missing derived observations...
> Processing record: 959536; Last record: 2024-01-19 00:00:00 PST 
> (1705651200)
> Recalculating daily summaries...
> Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
> Finished recalculating daily summaries
> Missing derived observations calculated in 2986.21 seconds
>
> And after restarting weewx things look good 
>
> 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO 
> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to 
> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Tom Keffer]

We've seen this problem with largish databases. Try setting the option
--tranche to 5, or even 1

weectl database calc-missing --tranch=1


On Fri, Jan 19, 2024 at 4:03 PM Michael Sanphillipo 
wrote:

> What happens when the sudo weectl database calc-missing command stops on a
> date and seems to stall?
>
> On Friday, January 19, 2024 at 5:53:11 PM UTC-5 Tom Keffer wrote:
>
>> Good sleuthing, Paul!
>>
>> On Fri, Jan 19, 2024 at 10:23 AM Paul R Anderson 
>> wrote:
>>
>>> This was really fun to track down! Thanks for the challenge !
>>> I have been able to reproduce your issue... it's a  little complicated,
>>> but it can be fixed  easily :)
>>> It's caused by 2 factors
>>> *1. Your station seems to have stopped reporting Barometric Pressure on
>>> or about 2019-12-31 @ 16:45 *
>>> *No idea why, hopefully you are aware of this and know the cause.*
>>>
>>> *2. Your using an older version of  NOAA-%Y-%m.txt.tmpl*
>>> The older version used this statement to determine if there was data for
>>> each day
>>> #for $day in $month.days
>>> #if $day.barometer.count.raw
>>> So it fails because there is no  $day.barometer.count.raw
>>> Your Monthly Report will look like this:
>>>
>>>MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>>>
>>>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>>>
>>>  HEAT   COOL AVG
>>>   MEAN   DEGDEG  WIND
>>> DOM
>>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED
>>> HIGH   TIMEDIR
>>> 
>>> ---
>>>  01
>>>  02
>>>  03
>>>  04
>>>  05
>>>  06
>>>  07
>>>  08
>>>  09
>>>  10
>>>  11
>>>  12
>>>  13
>>>  14
>>>  15
>>>  16
>>>  17
>>>  18
>>>  19
>>>  20
>>>  21
>>>  22
>>>  23
>>>  24
>>>  25
>>>  26
>>>  27
>>>  28
>>>  29
>>>  30
>>>  31
>>> 
>>> ---
>>>   55.4   81.3 31   44.4 12  294.20.7   0.150.4
>>> 14.0 29302
>>>
>>> *Newer version of NOAA-%Y-%m.txt.tmp*
>>> Changed how it determines if there was data for each day
>>> #for $day in $month.days
>>> #if $day.outTemp.has_data or $day.rain.has_data or $day.wind.has_data
>>> So it will pass even if there is no  $day.barometer.count.raw
>>> Your Monthly Report will now look like this:
>>>
>>>  MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>>>
>>>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>>>
>>>  HEAT   COOL AVG
>>>   MEAN   DEGDEG  WIND
>>> DOM
>>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED
>>> HIGH   TIMEDIR
>>>
>>> ---
>>>  01   55.7   63.8  10:55   47.4  03:409.30.0   0.000.3
>>>  9.0  08:00294
>>>  02   55.9   64.1  06:30   51.5  03:109.10.0   0.000.3
>>>  4.0  05:45322
>>>  03   62.5   72.2  16:55   52.5  23:152.50.0   0.000.6
>>>  6.0  14:50301
>>>  04   57.3   71.7  16:55   48.7  07:207.70.0   0.000.4
>>>  6.0  13:30297
>>>  05   56.9   70.0  16:45   47.8  10:158.10.0   0.000.4
>>>  6.0  16:50302
>>>  06   64.5   80.7  16:20   53.1  10:250.50.0   0.000.2
>>>  6.0  14:50297
>>>  07   60.8   79.3  15:40   50.5  00:404.20.0   0.000.3
>>>  7.0  15:55299
>>>  08   51.9   62.5  15:05   47.1  08:30   13.10.0   0.000.5
>>>  9.0  15:30317
>>>  09   50.9   60.8  14:20   46.6  01:25   14.10.0   0.000.6
>>> 11.0  15:35301
>>>  10   51.6   64.1  16:30   44.9  09:35   13.40.0   0.010.5
>>>  7.0  15:50294
>>>  11   51.0   60.6  15:20   45.0  07:25   14.00.0   0.000.7
>>> 13.0  15:25325
>>>  12   51.2   62.5  15:10   44.4  09:15   13.80.0   0.000.5
>>> 10.0  16:25298
>>>  13   51.2   59.8  17:45   46.3  08:50   13.80.0   0.000.5
>>>  8.0  17:30300
>>>  14   50.4   60.5  16:35   45.2  10:00   14.60.0   0.010.4
>>>  9.0  17:00294
>>>  15   51.1   65.1  16:00   44.4  09:40   13.90.0   0.000.4
>>>  8.0  17:25296
>>>  16   50.3   58.4  15:20   45.4  10:00   14.70.0   0.050.8
>>> 11.0  01:25315
>>>  17   51.2   60.1  16:55   45.5  10:20   13.80.0   0.050.6
>>> 12.0  03:40316
>>>  18   56.1   69.6  17:30   46.9  06:108.90.0   0.000.2
>>>  6.0  16:10294
>>>  19   58.6   76.1  14:50   49.0  10:056.40.0   0.000.2
>>>  9.0  15:10300
>>>  20   54.9   59.3  15:45   52.4  04:50   10.10.0   0.000.3
>>> 11.0  15:50306
>>>  21   52.0   56.8  14:20   47.7  02:15   13.00.0   0.030.3
>>>  7.0  14:35310
>>>  22   52.6   61.1  14:25   47.6  04:35   12.40.0   0.000.5
>>>  8.0  16:25  

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[vince]

Providing logs always helps folks try to helpbut you can add the --from 
and --to switches to do a range of dates.  Possibly add the --tranche 
option to set the number of records in a batch it runs.

On Friday, January 19, 2024 at 4:03:44 PM UTC-8 Michael Sanphillipo wrote:

> What happens when the sudo weectl database calc-missing command stops on a 
> date and seems to stall?
>
> On Friday, January 19, 2024 at 5:53:11 PM UTC-5 Tom Keffer wrote:
>
>> Good sleuthing, Paul!
>>
>> On Fri, Jan 19, 2024 at 10:23 AM Paul R Anderson  
>> wrote:
>>
>>> This was really fun to track down! Thanks for the challenge !
>>> I have been able to reproduce your issue... it's a  little complicated, 
>>> but it can be fixed  easily :)
>>> It's caused by 2 factors
>>> *1. Your station seems to have stopped reporting Barometric Pressure on 
>>> or about 2019-12-31 @ 16:45 *
>>> *No idea why, hopefully you are aware of this and know the cause.*
>>>
>>> *2. Your using an older version of  NOAA-%Y-%m.txt.tmpl*
>>> The older version used this statement to determine if there was data for 
>>> each day
>>> #for $day in $month.days
>>> #if $day.barometer.count.raw
>>> So it fails because there is no  $day.barometer.count.raw
>>> Your Monthly Report will look like this:
>>>
>>>MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>>>
>>>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>>>
>>>  HEAT   COOL AVG
>>>   MEAN   DEGDEG  WIND   
>>> DOM
>>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   
>>> HIGH   TIMEDIR
>>> 
>>> ---
>>>  01
>>>  02
>>>  03
>>>  04
>>>  05
>>>  06
>>>  07
>>>  08
>>>  09
>>>  10
>>>  11
>>>  12
>>>  13
>>>  14
>>>  15
>>>  16
>>>  17
>>>  18
>>>  19
>>>  20
>>>  21
>>>  22
>>>  23
>>>  24
>>>  25
>>>  26
>>>  27
>>>  28
>>>  29
>>>  30
>>>  31
>>> 
>>> ---
>>>   55.4   81.3 31   44.4 12  294.20.7   0.150.4   
>>> 14.0 29302
>>>
>>> *Newer version of NOAA-%Y-%m.txt.tmp*
>>> Changed how it determines if there was data for each day
>>> #for $day in $month.days
>>> #if $day.outTemp.has_data or $day.rain.has_data or $day.wind.has_data
>>> So it will pass even if there is no  $day.barometer.count.raw
>>> Your Monthly Report will now look like this:
>>>  
>>>  MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>>>
>>>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>>>
>>>  HEAT   COOL AVG
>>>   MEAN   DEGDEG  WIND   
>>> DOM
>>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   
>>> HIGH   TIMEDIR
>>>
>>> ---
>>>  01   55.7   63.8  10:55   47.4  03:409.30.0   0.000.3   
>>>  9.0  08:00294
>>>  02   55.9   64.1  06:30   51.5  03:109.10.0   0.000.3   
>>>  4.0  05:45322
>>>  03   62.5   72.2  16:55   52.5  23:152.50.0   0.000.6   
>>>  6.0  14:50301
>>>  04   57.3   71.7  16:55   48.7  07:207.70.0   0.000.4   
>>>  6.0  13:30297
>>>  05   56.9   70.0  16:45   47.8  10:158.10.0   0.000.4   
>>>  6.0  16:50302
>>>  06   64.5   80.7  16:20   53.1  10:250.50.0   0.000.2   
>>>  6.0  14:50297
>>>  07   60.8   79.3  15:40   50.5  00:404.20.0   0.000.3   
>>>  7.0  15:55299
>>>  08   51.9   62.5  15:05   47.1  08:30   13.10.0   0.000.5   
>>>  9.0  15:30317
>>>  09   50.9   60.8  14:20   46.6  01:25   14.10.0   0.000.6   
>>> 11.0  15:35301
>>>  10   51.6   64.1  16:30   44.9  09:35   13.40.0   0.010.5   
>>>  7.0  15:50294
>>>  11   51.0   60.6  15:20   45.0  07:25   14.00.0   0.000.7   
>>> 13.0  15:25325
>>>  12   51.2   62.5  15:10   44.4  09:15   13.80.0   0.000.5   
>>> 10.0  16:25298
>>>  13   51.2   59.8  17:45   46.3  08:50   13.80.0   0.000.5   
>>>  8.0  17:30300
>>>  14   50.4   60.5  16:35   45.2  10:00   14.60.0   0.010.4   
>>>  9.0  17:00294
>>>  15   51.1   65.1  16:00   44.4  09:40   13.90.0   0.000.4   
>>>  8.0  17:25296
>>>  16   50.3   58.4  15:20   45.4  10:00   14.70.0   0.050.8   
>>> 11.0  01:25315
>>>  17   51.2   60.1  16:55   45.5  10:20   13.80.0   0.050.6   
>>> 12.0  03:40316
>>>  18   56.1   69.6  17:30   46.9  06:108.90.0   0.000.2   
>>>  6.0  16:10294
>>>  19   58.6   76.1  14:50   49.0  10:056.40.0   0.000.2   
>>>  9.0  15:10300
>>>  20   54.9   59.3  15:45   52.4  04:50   10.10.0   0.000.3   
>>> 11.0  15:50306
>>>  21   

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Michael Sanphillipo]

What happens when the sudo weectl database calc-missing command stops on a 
date and seems to stall?

On Friday, January 19, 2024 at 5:53:11 PM UTC-5 Tom Keffer wrote:

> Good sleuthing, Paul!
>
> On Fri, Jan 19, 2024 at 10:23 AM Paul R Anderson  
> wrote:
>
>> This was really fun to track down! Thanks for the challenge !
>> I have been able to reproduce your issue... it's a  little complicated, 
>> but it can be fixed  easily :)
>> It's caused by 2 factors
>> *1. Your station seems to have stopped reporting Barometric Pressure on 
>> or about 2019-12-31 @ 16:45 *
>> *No idea why, hopefully you are aware of this and know the cause.*
>>
>> *2. Your using an older version of  NOAA-%Y-%m.txt.tmpl*
>> The older version used this statement to determine if there was data for 
>> each day
>> #for $day in $month.days
>> #if $day.barometer.count.raw
>> So it fails because there is no  $day.barometer.count.raw
>> Your Monthly Report will look like this:
>>
>>MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>>
>>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>>
>>  HEAT   COOL AVG
>>   MEAN   DEGDEG  WIND 
>>   DOM
>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH 
>>   TIMEDIR
>> 
>> ---
>>  01
>>  02
>>  03
>>  04
>>  05
>>  06
>>  07
>>  08
>>  09
>>  10
>>  11
>>  12
>>  13
>>  14
>>  15
>>  16
>>  17
>>  18
>>  19
>>  20
>>  21
>>  22
>>  23
>>  24
>>  25
>>  26
>>  27
>>  28
>>  29
>>  30
>>  31
>> 
>> ---
>>   55.4   81.3 31   44.4 12  294.20.7   0.150.4   14.0 
>> 29302
>>
>> *Newer version of NOAA-%Y-%m.txt.tmp*
>> Changed how it determines if there was data for each day
>> #for $day in $month.days
>> #if $day.outTemp.has_data or $day.rain.has_data or $day.wind.has_data
>> So it will pass even if there is no  $day.barometer.count.raw
>> Your Monthly Report will now look like this:
>>  
>>  MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>>
>>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>>
>>  HEAT   COOL AVG
>>   MEAN   DEGDEG  WIND 
>>   DOM
>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH 
>>   TIMEDIR
>>
>> ---
>>  01   55.7   63.8  10:55   47.4  03:409.30.0   0.000.39.0 
>>  08:00294
>>  02   55.9   64.1  06:30   51.5  03:109.10.0   0.000.34.0 
>>  05:45322
>>  03   62.5   72.2  16:55   52.5  23:152.50.0   0.000.66.0 
>>  14:50301
>>  04   57.3   71.7  16:55   48.7  07:207.70.0   0.000.46.0 
>>  13:30297
>>  05   56.9   70.0  16:45   47.8  10:158.10.0   0.000.46.0 
>>  16:50302
>>  06   64.5   80.7  16:20   53.1  10:250.50.0   0.000.26.0 
>>  14:50297
>>  07   60.8   79.3  15:40   50.5  00:404.20.0   0.000.37.0 
>>  15:55299
>>  08   51.9   62.5  15:05   47.1  08:30   13.10.0   0.000.59.0 
>>  15:30317
>>  09   50.9   60.8  14:20   46.6  01:25   14.10.0   0.000.6   11.0 
>>  15:35301
>>  10   51.6   64.1  16:30   44.9  09:35   13.40.0   0.010.57.0 
>>  15:50294
>>  11   51.0   60.6  15:20   45.0  07:25   14.00.0   0.000.7   13.0 
>>  15:25325
>>  12   51.2   62.5  15:10   44.4  09:15   13.80.0   0.000.5   10.0 
>>  16:25298
>>  13   51.2   59.8  17:45   46.3  08:50   13.80.0   0.000.58.0 
>>  17:30300
>>  14   50.4   60.5  16:35   45.2  10:00   14.60.0   0.010.49.0 
>>  17:00294
>>  15   51.1   65.1  16:00   44.4  09:40   13.90.0   0.000.48.0 
>>  17:25296
>>  16   50.3   58.4  15:20   45.4  10:00   14.70.0   0.050.8   11.0 
>>  01:25315
>>  17   51.2   60.1  16:55   45.5  10:20   13.80.0   0.050.6   12.0 
>>  03:40316
>>  18   56.1   69.6  17:30   46.9  06:108.90.0   0.000.26.0 
>>  16:10294
>>  19   58.6   76.1  14:50   49.0  10:056.40.0   0.000.29.0 
>>  15:10300
>>  20   54.9   59.3  15:45   52.4  04:50   10.10.0   0.000.3   11.0 
>>  15:50306
>>  21   52.0   56.8  14:20   47.7  02:15   13.00.0   0.030.37.0 
>>  14:35310
>>  22   52.6   61.1  14:25   47.6  04:35   12.40.0   0.000.58.0 
>>  16:25296
>>  23   59.9   73.0  16:20   50.0  07:205.10.0   0.000.47.0 
>>  15:40306
>>  24   59.0   74.2  16:55   51.0  02:056.00.0   0.000.35.0 
>>  13:05297
>>  25   54.9   69.8  17:10   46.9  08:45   

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Tom Keffer]

Good sleuthing, Paul!

On Fri, Jan 19, 2024 at 10:23 AM Paul R Anderson  wrote:

> This was really fun to track down! Thanks for the challenge !
> I have been able to reproduce your issue... it's a  little complicated,
> but it can be fixed  easily :)
> It's caused by 2 factors
> *1. Your station seems to have stopped reporting Barometric Pressure on or
> about 2019-12-31 @ 16:45 *
> *No idea why, hopefully you are aware of this and know the cause.*
>
> *2. Your using an older version of  NOAA-%Y-%m.txt.tmpl*
> The older version used this statement to determine if there was data for
> each day
> #for $day in $month.days
> #if $day.barometer.count.raw
> So it fails because there is no  $day.barometer.count.raw
> Your Monthly Report will look like this:
>
>MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>
>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>
>  HEAT   COOL AVG
>   MEAN   DEGDEG  WIND
>   DOM
> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH
>   TIMEDIR
> 
> ---
>  01
>  02
>  03
>  04
>  05
>  06
>  07
>  08
>  09
>  10
>  11
>  12
>  13
>  14
>  15
>  16
>  17
>  18
>  19
>  20
>  21
>  22
>  23
>  24
>  25
>  26
>  27
>  28
>  29
>  30
>  31
> 
> ---
>   55.4   81.3 31   44.4 12  294.20.7   0.150.4   14.0
> 29302
>
> *Newer version of NOAA-%Y-%m.txt.tmp*
> Changed how it determines if there was data for each day
> #for $day in $month.days
> #if $day.outTemp.has_data or $day.rain.has_data or $day.wind.has_data
> So it will pass even if there is no  $day.barometer.count.raw
> Your Monthly Report will now look like this:
>
>  MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>
>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>
>  HEAT   COOL AVG
>   MEAN   DEGDEG  WIND
>   DOM
> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH
>   TIMEDIR
>
> ---
>  01   55.7   63.8  10:55   47.4  03:409.30.0   0.000.39.0
>  08:00294
>  02   55.9   64.1  06:30   51.5  03:109.10.0   0.000.34.0
>  05:45322
>  03   62.5   72.2  16:55   52.5  23:152.50.0   0.000.66.0
>  14:50301
>  04   57.3   71.7  16:55   48.7  07:207.70.0   0.000.46.0
>  13:30297
>  05   56.9   70.0  16:45   47.8  10:158.10.0   0.000.46.0
>  16:50302
>  06   64.5   80.7  16:20   53.1  10:250.50.0   0.000.26.0
>  14:50297
>  07   60.8   79.3  15:40   50.5  00:404.20.0   0.000.37.0
>  15:55299
>  08   51.9   62.5  15:05   47.1  08:30   13.10.0   0.000.59.0
>  15:30317
>  09   50.9   60.8  14:20   46.6  01:25   14.10.0   0.000.6   11.0
>  15:35301
>  10   51.6   64.1  16:30   44.9  09:35   13.40.0   0.010.57.0
>  15:50294
>  11   51.0   60.6  15:20   45.0  07:25   14.00.0   0.000.7   13.0
>  15:25325
>  12   51.2   62.5  15:10   44.4  09:15   13.80.0   0.000.5   10.0
>  16:25298
>  13   51.2   59.8  17:45   46.3  08:50   13.80.0   0.000.58.0
>  17:30300
>  14   50.4   60.5  16:35   45.2  10:00   14.60.0   0.010.49.0
>  17:00294
>  15   51.1   65.1  16:00   44.4  09:40   13.90.0   0.000.48.0
>  17:25296
>  16   50.3   58.4  15:20   45.4  10:00   14.70.0   0.050.8   11.0
>  01:25315
>  17   51.2   60.1  16:55   45.5  10:20   13.80.0   0.050.6   12.0
>  03:40316
>  18   56.1   69.6  17:30   46.9  06:108.90.0   0.000.26.0
>  16:10294
>  19   58.6   76.1  14:50   49.0  10:056.40.0   0.000.29.0
>  15:10300
>  20   54.9   59.3  15:45   52.4  04:50   10.10.0   0.000.3   11.0
>  15:50306
>  21   52.0   56.8  14:20   47.7  02:15   13.00.0   0.030.37.0
>  14:35310
>  22   52.6   61.1  14:25   47.6  04:35   12.40.0   0.000.58.0
>  16:25296
>  23   59.9   73.0  16:20   50.0  07:205.10.0   0.000.47.0
>  15:40306
>  24   59.0   74.2  16:55   51.0  02:056.00.0   0.000.35.0
>  13:05297
>  25   54.9   69.8  17:10   46.9  08:45   10.10.0   0.000.37.0
>  14:20296
>  26   51.2   57.1  17:10   47.4  00:45   13.80.0   0.000.17.0
>  17:10299
>  27   55.4   70.4  16:55   45.7  08:259.60.0   0.000.38.0
>  17:15284
>  28   58.9   73.9  17:15   50.9  08:156.10.0   0.000.67.0
>  15:15296
>  29   58.9   71.3  18:15   45.7  

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

['michael.k...@gmx.at' via weewx-user]

Next time I won't wait until the weekend! Great job!

Paul R Anderson schrieb am Freitag, 19. Januar 2024 um 19:23:48 UTC+1:

> This was really fun to track down! Thanks for the challenge !
> I have been able to reproduce your issue... it's a  little complicated, 
> but it can be fixed  easily :)
> It's caused by 2 factors
> *1. Your station seems to have stopped reporting Barometric Pressure on or 
> about 2019-12-31 @ 16:45 *
> *No idea why, hopefully you are aware of this and know the cause.*
>
> *2. Your using an older version of  NOAA-%Y-%m.txt.tmpl*
> The older version used this statement to determine if there was data for 
> each day
> #for $day in $month.days
> #if $day.barometer.count.raw
> So it fails because there is no  $day.barometer.count.raw
> Your Monthly Report will look like this:
>
>MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>
>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>
>  HEAT   COOL AVG
>   MEAN   DEGDEG  WIND 
>   DOM
> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH 
>   TIMEDIR
> 
> ---
>  01
>  02
>  03
>  04
>  05
>  06
>  07
>  08
>  09
>  10
>  11
>  12
>  13
>  14
>  15
>  16
>  17
>  18
>  19
>  20
>  21
>  22
>  23
>  24
>  25
>  26
>  27
>  28
>  29
>  30
>  31
> 
> ---
>   55.4   81.3 31   44.4 12  294.20.7   0.150.4   14.0 
> 29302
>
> *Newer version of NOAA-%Y-%m.txt.tmp*
> Changed how it determines if there was data for each day
> #for $day in $month.days
> #if $day.outTemp.has_data or $day.rain.has_data or $day.wind.has_data
> So it will pass even if there is no  $day.barometer.count.raw
> Your Monthly Report will now look like this:
>  
>  MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020
>
>TEMPERATURE (F), RAIN (in), WIND SPEED (mph)
>
>  HEAT   COOL AVG
>   MEAN   DEGDEG  WIND 
>   DOM
> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH 
>   TIMEDIR
>
> ---
>  01   55.7   63.8  10:55   47.4  03:409.30.0   0.000.39.0 
>  08:00294
>  02   55.9   64.1  06:30   51.5  03:109.10.0   0.000.34.0 
>  05:45322
>  03   62.5   72.2  16:55   52.5  23:152.50.0   0.000.66.0 
>  14:50301
>  04   57.3   71.7  16:55   48.7  07:207.70.0   0.000.46.0 
>  13:30297
>  05   56.9   70.0  16:45   47.8  10:158.10.0   0.000.46.0 
>  16:50302
>  06   64.5   80.7  16:20   53.1  10:250.50.0   0.000.26.0 
>  14:50297
>  07   60.8   79.3  15:40   50.5  00:404.20.0   0.000.37.0 
>  15:55299
>  08   51.9   62.5  15:05   47.1  08:30   13.10.0   0.000.59.0 
>  15:30317
>  09   50.9   60.8  14:20   46.6  01:25   14.10.0   0.000.6   11.0 
>  15:35301
>  10   51.6   64.1  16:30   44.9  09:35   13.40.0   0.010.57.0 
>  15:50294
>  11   51.0   60.6  15:20   45.0  07:25   14.00.0   0.000.7   13.0 
>  15:25325
>  12   51.2   62.5  15:10   44.4  09:15   13.80.0   0.000.5   10.0 
>  16:25298
>  13   51.2   59.8  17:45   46.3  08:50   13.80.0   0.000.58.0 
>  17:30300
>  14   50.4   60.5  16:35   45.2  10:00   14.60.0   0.010.49.0 
>  17:00294
>  15   51.1   65.1  16:00   44.4  09:40   13.90.0   0.000.48.0 
>  17:25296
>  16   50.3   58.4  15:20   45.4  10:00   14.70.0   0.050.8   11.0 
>  01:25315
>  17   51.2   60.1  16:55   45.5  10:20   13.80.0   0.050.6   12.0 
>  03:40316
>  18   56.1   69.6  17:30   46.9  06:108.90.0   0.000.26.0 
>  16:10294
>  19   58.6   76.1  14:50   49.0  10:056.40.0   0.000.29.0 
>  15:10300
>  20   54.9   59.3  15:45   52.4  04:50   10.10.0   0.000.3   11.0 
>  15:50306
>  21   52.0   56.8  14:20   47.7  02:15   13.00.0   0.030.37.0 
>  14:35310
>  22   52.6   61.1  14:25   47.6  04:35   12.40.0   0.000.58.0 
>  16:25296
>  23   59.9   73.0  16:20   50.0  07:205.10.0   0.000.47.0 
>  15:40306
>  24   59.0   74.2  16:55   51.0  02:056.00.0   0.000.35.0 
>  13:05297
>  25   54.9   69.8  17:10   46.9  08:45   10.10.0   0.000.37.0 
>  14:20296
>  26   51.2   57.1  17:10   47.4  00:45   13.80.0   0.000.17.0 
>  17:10299
>  27   55.4   70.4  16:55   45.7  08:259.60.0   0.000.38.0 
>  17:15284
>  28   58.9   73.9  17:15   

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Paul R Anderson]

This was really fun to track down! Thanks for the challenge !
I have been able to reproduce your issue... it's a  little complicated, but
it can be fixed  easily :)
It's caused by 2 factors
*1. Your station seems to have stopped reporting Barometric Pressure on or
about 2019-12-31 @ 16:45 *
*No idea why, hopefully you are aware of this and know the cause.*

*2. Your using an older version of  NOAA-%Y-%m.txt.tmpl*
The older version used this statement to determine if there was data for
each day
#for $day in $month.days
#if $day.barometer.count.raw
So it fails because there is no  $day.barometer.count.raw
Your Monthly Report will look like this:

   MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020

   TEMPERATURE (F), RAIN (in), WIND SPEED (mph)

 HEAT   COOL AVG
  MEAN   DEGDEG  WIND
DOM
DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH
TIMEDIR

---
 01
 02
 03
 04
 05
 06
 07
 08
 09
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31

---
  55.4   81.3 31   44.4 12  294.20.7   0.150.4   14.0
  29302

*Newer version of NOAA-%Y-%m.txt.tmp*
Changed how it determines if there was data for each day
#for $day in $month.days
#if $day.outTemp.has_data or $day.rain.has_data or $day.wind.has_data
So it will pass even if there is no  $day.barometer.count.raw
Your Monthly Report will now look like this:

 MONTHLY CLIMATOLOGICAL SUMMARY for Jan 2020

   TEMPERATURE (F), RAIN (in), WIND SPEED (mph)

 HEAT   COOL AVG
  MEAN   DEGDEG  WIND
DOM
DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH
TIMEDIR
---
 01   55.7   63.8  10:55   47.4  03:409.30.0   0.000.39.0
 08:00294
 02   55.9   64.1  06:30   51.5  03:109.10.0   0.000.34.0
 05:45322
 03   62.5   72.2  16:55   52.5  23:152.50.0   0.000.66.0
 14:50301
 04   57.3   71.7  16:55   48.7  07:207.70.0   0.000.46.0
 13:30297
 05   56.9   70.0  16:45   47.8  10:158.10.0   0.000.46.0
 16:50302
 06   64.5   80.7  16:20   53.1  10:250.50.0   0.000.26.0
 14:50297
 07   60.8   79.3  15:40   50.5  00:404.20.0   0.000.37.0
 15:55299
 08   51.9   62.5  15:05   47.1  08:30   13.10.0   0.000.59.0
 15:30317
 09   50.9   60.8  14:20   46.6  01:25   14.10.0   0.000.6   11.0
 15:35301
 10   51.6   64.1  16:30   44.9  09:35   13.40.0   0.010.57.0
 15:50294
 11   51.0   60.6  15:20   45.0  07:25   14.00.0   0.000.7   13.0
 15:25325
 12   51.2   62.5  15:10   44.4  09:15   13.80.0   0.000.5   10.0
 16:25298
 13   51.2   59.8  17:45   46.3  08:50   13.80.0   0.000.58.0
 17:30300
 14   50.4   60.5  16:35   45.2  10:00   14.60.0   0.010.49.0
 17:00294
 15   51.1   65.1  16:00   44.4  09:40   13.90.0   0.000.48.0
 17:25296
 16   50.3   58.4  15:20   45.4  10:00   14.70.0   0.050.8   11.0
 01:25315
 17   51.2   60.1  16:55   45.5  10:20   13.80.0   0.050.6   12.0
 03:40316
 18   56.1   69.6  17:30   46.9  06:108.90.0   0.000.26.0
 16:10294
 19   58.6   76.1  14:50   49.0  10:056.40.0   0.000.29.0
 15:10300
 20   54.9   59.3  15:45   52.4  04:50   10.10.0   0.000.3   11.0
 15:50306
 21   52.0   56.8  14:20   47.7  02:15   13.00.0   0.030.37.0
 14:35310
 22   52.6   61.1  14:25   47.6  04:35   12.40.0   0.000.58.0
 16:25296
 23   59.9   73.0  16:20   50.0  07:205.10.0   0.000.47.0
 15:40306
 24   59.0   74.2  16:55   51.0  02:056.00.0   0.000.35.0
 13:05297
 25   54.9   69.8  17:10   46.9  08:45   10.10.0   0.000.37.0
 14:20296
 26   51.2   57.1  17:10   47.4  00:45   13.80.0   0.000.17.0
 17:10299
 27   55.4   70.4  16:55   45.7  08:259.60.0   0.000.38.0
 17:15284
 28   58.9   73.9  17:15   50.9  08:156.10.0   0.000.67.0
 15:15296
 29   58.9   71.3  18:15   45.7  08:356.10.0   0.000.1   14.0
 16:10176
 30   59.0   71.3  16:50   48.9  09:506.00.0   0.000.59.0
 15:30302
 31   65.7   81.3  17:50   53.1  08:550.00.7   0.000.27.0
 15:25304
---
  55.4   81.3 31   44.4 12  

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Michael Sanphillipo]

Richard, just curious do you have an Acurite weather station and using 
Belcherstown? I'm just trying to see if we have the same type of setup 
before going back to V5 and trying your solution. Thanks!

On Friday, January 19, 2024 at 5:49:18 AM UTC-5 Richard Whitcombe wrote:

> FWIW i just needed to add appTemp and its worked - report generation now 
> takes 3 seconds not 190.
> I would caveat that by saying im not using the other missing columns at 
> all as my station doesnt provide the data so maybe if you are they need to 
> be added too.
> Missing data calculations took about 8 mins for 2017 - 2024 on a Pi4.
>
> On Friday 19 January 2024 at 15:59:15 UTC+9 vince wrote:
>
>> Re: the Belchertown issue many seem to be facing, I notice that Blaine's 
>> db uses the original old wview schema, not the newer bigger 
>> 'wview_extended' schema that came out a few years ago.  I suspect this is 
>> the pattern we've missed.  Folks who have the old schema do not have any of 
>> the elements for many of the Extended Observations in Belchertown's 
>> skin.conf file. 
>>
>> Blaine's db schema misses these items in Belchertown's skin.conf extended 
>> observations list
>> appTemp  = Apparent Temperature
>> cloudbase= Cloud Base
>> visibility   = Visibility
>> windrun  = Wind Run
>> cloud_cover  = Cloud Cover
>> aqi  = AQI
>>
>> The wview_extended schema is missing these, so I'd expect users with data 
>> in those somehow might have the same slowness issue until they add db 
>> elements and calc-missing to generate summary tables for them.
>> visibility   = Visibility
>> cloud_cover  = Cloud Cover
>>
>> Re: Michael's question about what to do about the NOAA files, I'd suggest 
>> pre-calculating them offline and dropping them into place in his NOAA 
>> subdirectory with the appropriate owner/group/mode so they don't have to be 
>> recalculated.  I'd omit the current month+year which should regenerate 
>> quickly on weewx startup.  That would be a good thing to try.  We've 
>> verified the db is ok, so it 'should' work on Blaine's system too 
>> (hopefully).  The test then would be to simply look at the date+time last 
>> modified for the current month and year NOAA file.  It should change every 
>> archive period as StdReport runs.
>>
>> (reminder - if you have Belchertown 'and' Seasons, you need to drop the 
>> pre-calculated NOAA files into both places to preseed both skin output 
>> directories)
>>
>> On Thursday, January 18, 2024 at 8:36:20 PM UTC-8 michael.k...@gmx.at 
>> wrote:
>>
>>> Or, to speed things up, do the "calc-missing" on another machine with a 
>>> more potent CPU.
>>>
>>> Back to Blaine's problem: if the NOAA files are generated on other 
>>> installations with his database, there is obviously another problem with 
>>> his installation.I don't have an idea what to look for, if deleted NOAA 
>>> data will be regenerated in the same way, as before.
>>>
>>> vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1:
>>>
 Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ 
 but what I did for a pip installation using Blaine's db was:

 source ~/weewx-venv/bin/activate
 weectl database add-column appTemp
 weectl calc-missing
 (answer y when prompted above)

 If you have a packaged install of weewx you can skip the activate step 
 above.

 Note - the calc-missing takes quite a while and will peg your cpu while 
 working, so if you have a lot of data so you might need to use the --from 
 and --to options or even the --tranche option to split up the processing a 
 bit into pieces.  I got lucky with this db on pi3plus with a bit of 
 patience waiting for calc-missing to complete

 (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
 Using configuration file /home/pi/weewx-data/weewx.conf
 Missing derived observations will be calculated for all records.
 Calculating missing derived observations...
 Processing record: 959536; Last record: 2024-01-19 00:00:00 PST 
 (1705651200)
 Recalculating daily summaries...
 Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
 Finished recalculating daily summaries
 Missing derived observations calculated in 2986.21 seconds

 And after restarting weewx things look good 

 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO 
 weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to 
 database 'weewx.sdb'
 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO 
 weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to daily 
 summary in 'weewx.sdb'
 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO 
 weewx.cheetahgenerator: Generated 8 files for report 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

['michael.k...@gmx.at' via weewx-user]

Good news: let the provided tools do the job, no need to touch that nasty 
thing with your own hands :D :D :D Maybe Tom can add an alias for 
"database" when "weectl database" is called ;) Or I might create an PR for 
this :D

WindnFog schrieb am Freitag, 19. Januar 2024 um 16:47:39 UTC+1:

> No need to be gentle, Tom.  My approach was probably the most complicated 
> (worst?) solution possible.  Without going into detail, when I was working, 
> I messed up a mission-critical Oracle database by trying to change 
> something I didn't understand. I can program fairly well, but I'm still 
> paranoid about databases!  So, I need to get my head around adding fields 
> to the weeWX database and not devise these complex ways to avoid touching 
> it.  :-)
>
> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>
>> I mean this in the gentlest way, but this is basically what the SQLite 
>> database is doing for you: saving various values in a file where they can 
>> be retrieved via a general query statement.
>>
>> On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:
>>
>>> I encountered the same problem trying to calculate the all time humidex 
>>> in the index.html.tmpl template.  This was taking about 3 minutes on a Pi 
>>> 4. Rather than add humidex to the database (probably the best solution), I 
>>> calculated all-time humidex once a day using a small program and invoking 
>>> it with cron. I saved it in a text file that I could then use the Cheetah 
>>> include statement to read it into index.the html.tmpl template.  This runs 
>>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for 
>>> appTemp, a similar approach might work.  I was not comfortable with 
>>> modifying the database because that's not my area of expertise:
>>>
>>> #!/usr/bin/env python3
>>> # -*- coding: utf-8 -*-
>>>
>>> """
>>> Program:WeeWX Humidex calculator
>>> Author: Paul M Dunphy
>>> Date:   12 December 2023
>>> Version:1.0.0
>>> Description:This script sweeps through the Weewx database and 
>>> calculates the 
>>> derived parameter humidex for each record.  It then 
>>> returns the
>>> maximum humidex and the date/time when it occured.  This 
>>> takes
>>> about 1.5 seconds on a 900,000 record database as 
>>> opposed to 3.5
>>> minutes by the WeeWX system if the recommended aggregate 
>>> values
>>> are implemented in the Standard/index.html.tmpl file:
>>> 
>>> $alltime.humidex.max ($alltime.humidex.max.degree_F) 
>>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>>
>>> It's not clear why the weeWX system takes so long.  
>>> Possibly
>>> because it's doing many other things at the same time.  
>>>
>>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>>> """
>>>
>>> # Required Libraries
>>>
>>> import sqlite3
>>> import math
>>> from datetime import datetime
>>> import pytz
>>> from pytz import timezone
>>>
>>>
>>> def calculate_humidex(temperature, dewpoint):
>>> # Constants for the formula
>>> k = 273.15  # Conversion factor for Celsius to Kelvin
>>>
>>> # Convert temperatures to Kelvin
>>> temperature + k
>>> dewpoint_k = dewpoint + k
>>>
>>> # Calculate vapor pressure in hPa
>>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 / 
>>> dewpoint_k)))
>>>
>>> # Humidex calculation
>>> humidex = temperature + ((vapor_pressure - 10) * 0.)
>>> return humidex
>>>
>>>
>>> def get_max_humidex(database_path):
>>> # Connect to the WeeWX database
>>> conn = sqlite3.connect(database_path)
>>> cursor = conn.cursor()
>>>
>>> # Query for temperature, dewpoint, and dateTime for all records
>>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>>> cursor.execute(query)
>>>
>>> # Variables to track the maximum humidex and its date
>>> maximum_humidex = None
>>> maximum_humidex_date = None
>>>
>>> for row in cursor:
>>> temperature, dewpoint, date_time = row
>>> if temperature is not None and dewpoint is not None:
>>> humidex = calculate_humidex(temperature, dewpoint)
>>> if maximum_humidex is None or humidex > maximum_humidex:
>>> maximum_humidex = humidex
>>> maximum_humidex_date = date_time
>>>
>>> conn.close()
>>> return maximum_humidex, maximum_humidex_date
>>>
>>>
>>> def convert_utc_to_ast_adt(utc_datetime):
>>> """
>>> Convert a UTC datetime to AST or ADT depending on the date.
>>>
>>> Parameters:
>>> utc_datetime (datetime): UTC datetime object.
>>>
>>> Returns:
>>> datetime: Datetime object in AST or ADT.
>>> """
>>> # Define the time zone for AST/ADT
>>> ast_tz = timezone('America/Halifax')
>>>
>>> # Localize the UTC datetime to the AST/ADT timezone
>>> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[WindnFog]

No need to be gentle, Tom.  My approach was probably the most complicated 
(worst?) solution possible.  Without going into detail, when I was working, 
I messed up a mission-critical Oracle database by trying to change 
something I didn't understand. I can program fairly well, but I'm still 
paranoid about databases!  So, I need to get my head around adding fields 
to the weeWX database and not devise these complex ways to avoid touching 
it.  :-)

On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:

> I mean this in the gentlest way, but this is basically what the SQLite 
> database is doing for you: saving various values in a file where they can 
> be retrieved via a general query statement.
>
> On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:
>
>> I encountered the same problem trying to calculate the all time humidex 
>> in the index.html.tmpl template.  This was taking about 3 minutes on a Pi 
>> 4. Rather than add humidex to the database (probably the best solution), I 
>> calculated all-time humidex once a day using a small program and invoking 
>> it with cron. I saved it in a text file that I could then use the Cheetah 
>> include statement to read it into index.the html.tmpl template.  This runs 
>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for 
>> appTemp, a similar approach might work.  I was not comfortable with 
>> modifying the database because that's not my area of expertise:
>>
>> #!/usr/bin/env python3
>> # -*- coding: utf-8 -*-
>>
>> """
>> Program:WeeWX Humidex calculator
>> Author: Paul M Dunphy
>> Date:   12 December 2023
>> Version:1.0.0
>> Description:This script sweeps through the Weewx database and 
>> calculates the 
>> derived parameter humidex for each record.  It then 
>> returns the
>> maximum humidex and the date/time when it occured.  This 
>> takes
>> about 1.5 seconds on a 900,000 record database as opposed 
>> to 3.5
>> minutes by the WeeWX system if the recommended aggregate 
>> values
>> are implemented in the Standard/index.html.tmpl file:
>> 
>> $alltime.humidex.max ($alltime.humidex.max.degree_F) 
>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>
>> It's not clear why the weeWX system takes so long.  
>> Possibly
>> because it's doing many other things at the same time.  
>>
>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>> """
>>
>> # Required Libraries
>>
>> import sqlite3
>> import math
>> from datetime import datetime
>> import pytz
>> from pytz import timezone
>>
>>
>> def calculate_humidex(temperature, dewpoint):
>> # Constants for the formula
>> k = 273.15  # Conversion factor for Celsius to Kelvin
>>
>> # Convert temperatures to Kelvin
>> temperature + k
>> dewpoint_k = dewpoint + k
>>
>> # Calculate vapor pressure in hPa
>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 / 
>> dewpoint_k)))
>>
>> # Humidex calculation
>> humidex = temperature + ((vapor_pressure - 10) * 0.)
>> return humidex
>>
>>
>> def get_max_humidex(database_path):
>> # Connect to the WeeWX database
>> conn = sqlite3.connect(database_path)
>> cursor = conn.cursor()
>>
>> # Query for temperature, dewpoint, and dateTime for all records
>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>> cursor.execute(query)
>>
>> # Variables to track the maximum humidex and its date
>> maximum_humidex = None
>> maximum_humidex_date = None
>>
>> for row in cursor:
>> temperature, dewpoint, date_time = row
>> if temperature is not None and dewpoint is not None:
>> humidex = calculate_humidex(temperature, dewpoint)
>> if maximum_humidex is None or humidex > maximum_humidex:
>> maximum_humidex = humidex
>> maximum_humidex_date = date_time
>>
>> conn.close()
>> return maximum_humidex, maximum_humidex_date
>>
>>
>> def convert_utc_to_ast_adt(utc_datetime):
>> """
>> Convert a UTC datetime to AST or ADT depending on the date.
>>
>> Parameters:
>> utc_datetime (datetime): UTC datetime object.
>>
>> Returns:
>> datetime: Datetime object in AST or ADT.
>> """
>> # Define the time zone for AST/ADT
>> ast_tz = timezone('America/Halifax')
>>
>> # Localize the UTC datetime to the AST/ADT timezone
>> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>>
>> return local_datetime
>>
>>
>> def format_date(unix_timestamp):
>> """
>> Format a UNIX timestamp into a human-readable string in AST or ADT.
>>
>> Parameters:
>> unix_timestamp (int): UNIX timestamp.
>>
>> Returns:
>> str: Formatted date string.
>> """
>> # Create a UTC datetime from the UNIX timestamp
>> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Tom Keffer]

I mean this in the gentlest way, but this is basically what the SQLite
database is doing for you: saving various values in a file where they can
be retrieved via a general query statement.

On Fri, Jan 19, 2024 at 6:16 AM WindnFog  wrote:

> I encountered the same problem trying to calculate the all time humidex in
> the index.html.tmpl template.  This was taking about 3 minutes on a Pi 4.
> Rather than add humidex to the database (probably the best solution), I
> calculated all-time humidex once a day using a small program and invoking
> it with cron. I saved it in a text file that I could then use the Cheetah
> include statement to read it into index.the html.tmpl template.  This runs
> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
> appTemp, a similar approach might work.  I was not comfortable with
> modifying the database because that's not my area of expertise:
>
> #!/usr/bin/env python3
> # -*- coding: utf-8 -*-
>
> """
> Program:WeeWX Humidex calculator
> Author: Paul M Dunphy
> Date:   12 December 2023
> Version:1.0.0
> Description:This script sweeps through the Weewx database and
> calculates the
> derived parameter humidex for each record.  It then
> returns the
> maximum humidex and the date/time when it occured.  This
> takes
> about 1.5 seconds on a 900,000 record database as opposed
> to 3.5
> minutes by the WeeWX system if the recommended aggregate
> values
> are implemented in the Standard/index.html.tmpl file:
>
> $alltime.humidex.max ($alltime.humidex.max.degree_F)
> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>
> It's not clear why the weeWX system takes so long.
> Possibly
> because it's doing many other things at the same time.
>
> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
> """
>
> # Required Libraries
>
> import sqlite3
> import math
> from datetime import datetime
> import pytz
> from pytz import timezone
>
>
> def calculate_humidex(temperature, dewpoint):
> # Constants for the formula
> k = 273.15  # Conversion factor for Celsius to Kelvin
>
> # Convert temperatures to Kelvin
> temperature + k
> dewpoint_k = dewpoint + k
>
> # Calculate vapor pressure in hPa
> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
> dewpoint_k)))
>
> # Humidex calculation
> humidex = temperature + ((vapor_pressure - 10) * 0.)
> return humidex
>
>
> def get_max_humidex(database_path):
> # Connect to the WeeWX database
> conn = sqlite3.connect(database_path)
> cursor = conn.cursor()
>
> # Query for temperature, dewpoint, and dateTime for all records
> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
> cursor.execute(query)
>
> # Variables to track the maximum humidex and its date
> maximum_humidex = None
> maximum_humidex_date = None
>
> for row in cursor:
> temperature, dewpoint, date_time = row
> if temperature is not None and dewpoint is not None:
> humidex = calculate_humidex(temperature, dewpoint)
> if maximum_humidex is None or humidex > maximum_humidex:
> maximum_humidex = humidex
> maximum_humidex_date = date_time
>
> conn.close()
> return maximum_humidex, maximum_humidex_date
>
>
> def convert_utc_to_ast_adt(utc_datetime):
> """
> Convert a UTC datetime to AST or ADT depending on the date.
>
> Parameters:
> utc_datetime (datetime): UTC datetime object.
>
> Returns:
> datetime: Datetime object in AST or ADT.
> """
> # Define the time zone for AST/ADT
> ast_tz = timezone('America/Halifax')
>
> # Localize the UTC datetime to the AST/ADT timezone
> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>
> return local_datetime
>
>
> def format_date(unix_timestamp):
> """
> Format a UNIX timestamp into a human-readable string in AST or ADT.
>
> Parameters:
> unix_timestamp (int): UNIX timestamp.
>
> Returns:
> str: Formatted date string.
> """
> # Create a UTC datetime from the UNIX timestamp
> utc_datetime =
> datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)
>
> # Convert UTC to AST or ADT
> local_datetime = convert_utc_to_ast_adt(utc_datetime)
>
> # Format the local datetime
> return local_datetime.strftime("%I:%M %p on %d %b %Y")
>
>
> db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
> max_humidex, max_humidex_date = get_max_humidex(db_path)
>
> # Convert Humidex to Fahrenheit
> max_humidex_f = max_humidex * 9 / 5 + 32
>
> # Format and print the result
> print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
> {format_date(max_humidex_date)}")
>
> # Format the result
> output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[WindnFog]

I encountered the same problem trying to calculate the all time humidex in 
the index.html.tmpl template.  This was taking about 3 minutes on a Pi 4. 
Rather than add humidex to the database (probably the best solution), I 
calculated all-time humidex once a day using a small program and invoking 
it with cron. I saved it in a text file that I could then use the Cheetah 
include statement to read it into index.the html.tmpl template.  This runs 
in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for 
appTemp, a similar approach might work.  I was not comfortable with 
modifying the database because that's not my area of expertise:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""
Program:WeeWX Humidex calculator
Author: Paul M Dunphy
Date:   12 December 2023
Version:1.0.0
Description:This script sweeps through the Weewx database and 
calculates the 
derived parameter humidex for each record.  It then returns 
the
maximum humidex and the date/time when it occured.  This 
takes
about 1.5 seconds on a 900,000 record database as opposed 
to 3.5
minutes by the WeeWX system if the recommended aggregate 
values
are implemented in the Standard/index.html.tmpl file:

$alltime.humidex.max ($alltime.humidex.max.degree_F) 
$alltime.humidex.maxtime.format($ALLTIMEFMT)

It's not clear why the weeWX system takes so long.  Possibly
because it's doing many other things at the same time.  

This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
"""

# Required Libraries

import sqlite3
import math
from datetime import datetime
import pytz
from pytz import timezone


def calculate_humidex(temperature, dewpoint):
# Constants for the formula
k = 273.15  # Conversion factor for Celsius to Kelvin

# Convert temperatures to Kelvin
temperature + k
dewpoint_k = dewpoint + k

# Calculate vapor pressure in hPa
vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 / 
dewpoint_k)))

# Humidex calculation
humidex = temperature + ((vapor_pressure - 10) * 0.)
return humidex


def get_max_humidex(database_path):
# Connect to the WeeWX database
conn = sqlite3.connect(database_path)
cursor = conn.cursor()

# Query for temperature, dewpoint, and dateTime for all records
query = "SELECT outTemp, dewpoint, dateTime FROM archive"
cursor.execute(query)

# Variables to track the maximum humidex and its date
maximum_humidex = None
maximum_humidex_date = None

for row in cursor:
temperature, dewpoint, date_time = row
if temperature is not None and dewpoint is not None:
humidex = calculate_humidex(temperature, dewpoint)
if maximum_humidex is None or humidex > maximum_humidex:
maximum_humidex = humidex
maximum_humidex_date = date_time

conn.close()
return maximum_humidex, maximum_humidex_date


def convert_utc_to_ast_adt(utc_datetime):
"""
Convert a UTC datetime to AST or ADT depending on the date.

Parameters:
utc_datetime (datetime): UTC datetime object.

Returns:
datetime: Datetime object in AST or ADT.
"""
# Define the time zone for AST/ADT
ast_tz = timezone('America/Halifax')

# Localize the UTC datetime to the AST/ADT timezone
local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))

return local_datetime


def format_date(unix_timestamp):
"""
Format a UNIX timestamp into a human-readable string in AST or ADT.

Parameters:
unix_timestamp (int): UNIX timestamp.

Returns:
str: Formatted date string.
"""
# Create a UTC datetime from the UNIX timestamp
utc_datetime = 
datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)

# Convert UTC to AST or ADT
local_datetime = convert_utc_to_ast_adt(utc_datetime)

# Format the local datetime
return local_datetime.strftime("%I:%M %p on %d %b %Y")


db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
max_humidex, max_humidex_date = get_max_humidex(db_path)

# Convert Humidex to Fahrenheit
max_humidex_f = max_humidex * 9 / 5 + 32

# Format and print the result
print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at 
{format_date(max_humidex_date)}")

# Format the result
output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at 
{format_date(max_humidex_date)}"

# Write the result to a file
output_file_path = 
'/home/pdunphy/weewx-data/skins/Standard/Standard_all_time_humidex.txt'
with open(output_file_path, 'w') as file:
file.write(output)

print(f"Output written to {output_file_path}") 



On Thursday, January 18, 2024 at 7:12:14 PM UTC-4 Tom Keffer wrote:

> Using Blaine's database, I was able to isolate the performance problems.
>
> It's in the template records/index.html.tmpl

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Richard Whitcombe]

FWIW i just needed to add appTemp and its worked - report generation now 
takes 3 seconds not 190.
I would caveat that by saying im not using the other missing columns at all 
as my station doesnt provide the data so maybe if you are they need to be 
added too.
Missing data calculations took about 8 mins for 2017 - 2024 on a Pi4.

On Friday 19 January 2024 at 15:59:15 UTC+9 vince wrote:

> Re: the Belchertown issue many seem to be facing, I notice that Blaine's 
> db uses the original old wview schema, not the newer bigger 
> 'wview_extended' schema that came out a few years ago.  I suspect this is 
> the pattern we've missed.  Folks who have the old schema do not have any of 
> the elements for many of the Extended Observations in Belchertown's 
> skin.conf file. 
>
> Blaine's db schema misses these items in Belchertown's skin.conf extended 
> observations list
> appTemp  = Apparent Temperature
> cloudbase= Cloud Base
> visibility   = Visibility
> windrun  = Wind Run
> cloud_cover  = Cloud Cover
> aqi  = AQI
>
> The wview_extended schema is missing these, so I'd expect users with data 
> in those somehow might have the same slowness issue until they add db 
> elements and calc-missing to generate summary tables for them.
> visibility   = Visibility
> cloud_cover  = Cloud Cover
>
> Re: Michael's question about what to do about the NOAA files, I'd suggest 
> pre-calculating them offline and dropping them into place in his NOAA 
> subdirectory with the appropriate owner/group/mode so they don't have to be 
> recalculated.  I'd omit the current month+year which should regenerate 
> quickly on weewx startup.  That would be a good thing to try.  We've 
> verified the db is ok, so it 'should' work on Blaine's system too 
> (hopefully).  The test then would be to simply look at the date+time last 
> modified for the current month and year NOAA file.  It should change every 
> archive period as StdReport runs.
>
> (reminder - if you have Belchertown 'and' Seasons, you need to drop the 
> pre-calculated NOAA files into both places to preseed both skin output 
> directories)
>
> On Thursday, January 18, 2024 at 8:36:20 PM UTC-8 michael.k...@gmx.at 
> wrote:
>
>> Or, to speed things up, do the "calc-missing" on another machine with a 
>> more potent CPU.
>>
>> Back to Blaine's problem: if the NOAA files are generated on other 
>> installations with his database, there is obviously another problem with 
>> his installation.I don't have an idea what to look for, if deleted NOAA 
>> data will be regenerated in the same way, as before.
>>
>> vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1:
>>
>>> Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ 
>>> but what I did for a pip installation using Blaine's db was:
>>>
>>> source ~/weewx-venv/bin/activate
>>> weectl database add-column appTemp
>>> weectl calc-missing
>>> (answer y when prompted above)
>>>
>>> If you have a packaged install of weewx you can skip the activate step 
>>> above.
>>>
>>> Note - the calc-missing takes quite a while and will peg your cpu while 
>>> working, so if you have a lot of data so you might need to use the --from 
>>> and --to options or even the --tranche option to split up the processing a 
>>> bit into pieces.  I got lucky with this db on pi3plus with a bit of 
>>> patience waiting for calc-missing to complete
>>>
>>> (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
>>> Using configuration file /home/pi/weewx-data/weewx.conf
>>> Missing derived observations will be calculated for all records.
>>> Calculating missing derived observations...
>>> Processing record: 959536; Last record: 2024-01-19 00:00:00 PST 
>>> (1705651200)
>>> Recalculating daily summaries...
>>> Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
>>> Finished recalculating daily summaries
>>> Missing derived observations calculated in 2986.21 seconds
>>>
>>> And after restarting weewx things look good 
>>>
>>> 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO 
>>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to 
>>> database 'weewx.sdb'
>>> 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO 
>>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to daily 
>>> summary in 'weewx.sdb'
>>> 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO 
>>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92 
>>> seconds
>>> 2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO 
>>> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14 
>>> seconds
>>> 2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO 
>>> weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html
>>> 2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[vince]

Re: the Belchertown issue many seem to be facing, I notice that Blaine's db 
uses the original old wview schema, not the newer bigger 'wview_extended' 
schema that came out a few years ago.  I suspect this is the pattern we've 
missed.  Folks who have the old schema do not have any of the elements for 
many of the Extended Observations in Belchertown's skin.conf file. 

Blaine's db schema misses these items in Belchertown's skin.conf extended 
observations list
appTemp  = Apparent Temperature
cloudbase= Cloud Base
visibility   = Visibility
windrun  = Wind Run
cloud_cover  = Cloud Cover
aqi  = AQI

The wview_extended schema is missing these, so I'd expect users with data 
in those somehow might have the same slowness issue until they add db 
elements and calc-missing to generate summary tables for them.
visibility   = Visibility
cloud_cover  = Cloud Cover

Re: Michael's question about what to do about the NOAA files, I'd suggest 
pre-calculating them offline and dropping them into place in his NOAA 
subdirectory with the appropriate owner/group/mode so they don't have to be 
recalculated.  I'd omit the current month+year which should regenerate 
quickly on weewx startup.  That would be a good thing to try.  We've 
verified the db is ok, so it 'should' work on Blaine's system too 
(hopefully).  The test then would be to simply look at the date+time last 
modified for the current month and year NOAA file.  It should change every 
archive period as StdReport runs.

(reminder - if you have Belchertown 'and' Seasons, you need to drop the 
pre-calculated NOAA files into both places to preseed both skin output 
directories)

On Thursday, January 18, 2024 at 8:36:20 PM UTC-8 michael.k...@gmx.at wrote:

> Or, to speed things up, do the "calc-missing" on another machine with a 
> more potent CPU.
>
> Back to Blaine's problem: if the NOAA files are generated on other 
> installations with his database, there is obviously another problem with 
> his installation.I don't have an idea what to look for, if deleted NOAA 
> data will be regenerated in the same way, as before.
>
> vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1:
>
>> Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ 
>> but what I did for a pip installation using Blaine's db was:
>>
>> source ~/weewx-venv/bin/activate
>> weectl database add-column appTemp
>> weectl calc-missing
>> (answer y when prompted above)
>>
>> If you have a packaged install of weewx you can skip the activate step 
>> above.
>>
>> Note - the calc-missing takes quite a while and will peg your cpu while 
>> working, so if you have a lot of data so you might need to use the --from 
>> and --to options or even the --tranche option to split up the processing a 
>> bit into pieces.  I got lucky with this db on pi3plus with a bit of 
>> patience waiting for calc-missing to complete
>>
>> (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
>> Using configuration file /home/pi/weewx-data/weewx.conf
>> Missing derived observations will be calculated for all records.
>> Calculating missing derived observations...
>> Processing record: 959536; Last record: 2024-01-19 00:00:00 PST 
>> (1705651200)
>> Recalculating daily summaries...
>> Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
>> Finished recalculating daily summaries
>> Missing derived observations calculated in 2986.21 seconds
>>
>> And after restarting weewx things look good 
>>
>> 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to 
>> database 'weewx.sdb'
>> 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to daily 
>> summary in 'weewx.sdb'
>> 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92 
>> seconds
>> 2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14 
>> seconds
>> 2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html
>> 2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: INFO 
>> user.belchertown: version 1.3.1
>> 2024-01-18T19:45:37.910258-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 16.96 
>> seconds
>> 2024-01-18T19:45:37.947603-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.reportengine: Copied 40 files to 
>> /home/pi/weewx-data/public_html/belchertown
>>
>> 2024-01-18T19:50:15.410986-08:00 pi3plus weewxd[1797]: INFO 
>> weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to 
>> database 'weewx.sdb'
>> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Graham Eddy]

i found putting the db into a RAM disk for the duration helps in these 
situations (i had to do that for a weewx migration some time ago)
⊣GE⊢

> On 19 Jan 2024, at 3:36 pm, 'michael.k...@gmx.at' via weewx-user 
>  wrote:
> 
> Or, to speed things up, do the "calc-missing" on another machine with a more 
> potent CPU.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/0EE301A5-5FC2-4461-9AD4-67D2678ED9FA%40geddy.au.

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

['michael.k...@gmx.at' via weewx-user]

Or, to speed things up, do the "calc-missing" on another machine with a 
more potent CPU.

Back to Blaine's problem: if the NOAA files are generated on other 
installations with his database, there is obviously another problem with 
his installation.I don't have an idea what to look for, if deleted NOAA 
data will be regenerated in the same way, as before.

vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1:

> Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ but 
> what I did for a pip installation using Blaine's db was:
>
> source ~/weewx-venv/bin/activate
> weectl database add-column appTemp
> weectl calc-missing
> (answer y when prompted above)
>
> If you have a packaged install of weewx you can skip the activate step 
> above.
>
> Note - the calc-missing takes quite a while and will peg your cpu while 
> working, so if you have a lot of data so you might need to use the --from 
> and --to options or even the --tranche option to split up the processing a 
> bit into pieces.  I got lucky with this db on pi3plus with a bit of 
> patience waiting for calc-missing to complete
>
> (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
> Using configuration file /home/pi/weewx-data/weewx.conf
> Missing derived observations will be calculated for all records.
> Calculating missing derived observations...
> Processing record: 959536; Last record: 2024-01-19 00:00:00 PST 
> (1705651200)
> Recalculating daily summaries...
> Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
> Finished recalculating daily summaries
> Missing derived observations calculated in 2986.21 seconds
>
> And after restarting weewx things look good 
>
> 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
> Added record 2024-01-18 19:45:00 PST (1705635900) to database 'weewx.sdb'
> 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
> Added record 2024-01-18 19:45:00 PST (1705635900) to daily summary in 
> 'weewx.sdb'
> 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO 
> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92 
> seconds
> 2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO 
> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14 
> seconds
> 2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO 
> weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html
> 2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: INFO 
> user.belchertown: version 1.3.1
> 2024-01-18T19:45:37.910258-08:00 pi3plus weewxd[1797]: INFO 
> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 16.96 
> seconds
> 2024-01-18T19:45:37.947603-08:00 pi3plus weewxd[1797]: INFO 
> weewx.reportengine: Copied 40 files to 
> /home/pi/weewx-data/public_html/belchertown
>
> 2024-01-18T19:50:15.410986-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
> Added record 2024-01-18 19:50:00 PST (1705636200) to database 'weewx.sdb'
> 2024-01-18T19:50:15.438866-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
> Added record 2024-01-18 19:50:00 PST (1705636200) to daily summary in 
> 'weewx.sdb'
> 2024-01-18T19:50:18.223483-08:00 pi3plus weewxd[1797]: INFO 
> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.65 
> seconds
> 2024-01-18T19:50:19.311313-08:00 pi3plus weewxd[1797]: INFO 
> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.08 
> seconds
> 2024-01-18T19:50:19.313932-08:00 pi3plus weewxd[1797]: INFO 
> weewx.reportengine: Copied 0 files to /home/pi/weewx-data/public_html
> 2024-01-18T19:50:31.266819-08:00 pi3plus weewxd[1797]: INFO 
> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 11.89 
> seconds
> 2024-01-18T19:50:31.277235-08:00 pi3plus weewxd[1797]: INFO 
> weewx.reportengine: Copied 3 files to 
> /home/pi/weewx-data/public_html/belchertown
>
> So the commands above 'did' fix this particular db.
> (thanks Tom)
>
> On Thursday, January 18, 2024 at 6:21:01 PM UTC-8 Richard Whitcombe wrote:
>
>> Can you explain in laymans terms how to do that?
>> Do you mean add a new DB field using SQL or similar and leave it blank or 
>> something else?
>>
>> Or outline trouble shooting steps to isolate the cause in simple terms 
>> (ie familiar with linux and the basics of weewx but not the core detailed 
>> functionality).
>>
>> On Friday 19 January 2024 at 08:12:14 UTC+9 Tom Keffer wrote:
>>
>>> Using Blaine's database, I was able to isolate the performance problems.
>>>
>>> It's in the template records/index.html.tmpl
>>>
>>> Specifically, apparent temperature (appTemp). It does not appear in the 
>>> database, so a simple tag such as $alltime.appTemp.max requires searching 
>>> the engine database, calculating apparent temperature for each record, then 
>>> finding the max value.
>>>
>>> In V4, this would not have even been attempted.
>>>
>>> The solution is to add appTemp to the database. 
>>>
>>> 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[vince]

Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ but 
what I did for a pip installation using Blaine's db was:

source ~/weewx-venv/bin/activate
weectl database add-column appTemp
weectl calc-missing
(answer y when prompted above)

If you have a packaged install of weewx you can skip the activate step 
above.

Note - the calc-missing takes quite a while and will peg your cpu while 
working, so if you have a lot of data so you might need to use the --from 
and --to options or even the --tranche option to split up the processing a 
bit into pieces.  I got lucky with this db on pi3plus with a bit of 
patience waiting for calc-missing to complete

(weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
Using configuration file /home/pi/weewx-data/weewx.conf
Missing derived observations will be calculated for all records.
Calculating missing derived observations...
Processing record: 959536; Last record: 2024-01-19 00:00:00 PST (1705651200)
Recalculating daily summaries...
Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
Finished recalculating daily summaries
Missing derived observations calculated in 2986.21 seconds

And after restarting weewx things look good 

2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
Added record 2024-01-18 19:45:00 PST (1705635900) to database 'weewx.sdb'
2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
Added record 2024-01-18 19:45:00 PST (1705635900) to daily summary in 
'weewx.sdb'
2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO 
weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92 
seconds
2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO 
weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14 
seconds
2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO 
weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html
2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: INFO 
user.belchertown: version 1.3.1
2024-01-18T19:45:37.910258-08:00 pi3plus weewxd[1797]: INFO 
weewx.cheetahgenerator: Generated 12 files for report Belchertown in 16.96 
seconds
2024-01-18T19:45:37.947603-08:00 pi3plus weewxd[1797]: INFO 
weewx.reportengine: Copied 40 files to 
/home/pi/weewx-data/public_html/belchertown

2024-01-18T19:50:15.410986-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
Added record 2024-01-18 19:50:00 PST (1705636200) to database 'weewx.sdb'
2024-01-18T19:50:15.438866-08:00 pi3plus weewxd[1797]: INFO weewx.manager: 
Added record 2024-01-18 19:50:00 PST (1705636200) to daily summary in 
'weewx.sdb'
2024-01-18T19:50:18.223483-08:00 pi3plus weewxd[1797]: INFO 
weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.65 
seconds
2024-01-18T19:50:19.311313-08:00 pi3plus weewxd[1797]: INFO 
weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.08 
seconds
2024-01-18T19:50:19.313932-08:00 pi3plus weewxd[1797]: INFO 
weewx.reportengine: Copied 0 files to /home/pi/weewx-data/public_html
2024-01-18T19:50:31.266819-08:00 pi3plus weewxd[1797]: INFO 
weewx.cheetahgenerator: Generated 12 files for report Belchertown in 11.89 
seconds
2024-01-18T19:50:31.277235-08:00 pi3plus weewxd[1797]: INFO 
weewx.reportengine: Copied 3 files to 
/home/pi/weewx-data/public_html/belchertown

So the commands above 'did' fix this particular db.
(thanks Tom)

On Thursday, January 18, 2024 at 6:21:01 PM UTC-8 Richard Whitcombe wrote:

> Can you explain in laymans terms how to do that?
> Do you mean add a new DB field using SQL or similar and leave it blank or 
> something else?
>
> Or outline trouble shooting steps to isolate the cause in simple terms (ie 
> familiar with linux and the basics of weewx but not the core detailed 
> functionality).
>
> On Friday 19 January 2024 at 08:12:14 UTC+9 Tom Keffer wrote:
>
>> Using Blaine's database, I was able to isolate the performance problems.
>>
>> It's in the template records/index.html.tmpl
>>
>> Specifically, apparent temperature (appTemp). It does not appear in the 
>> database, so a simple tag such as $alltime.appTemp.max requires searching 
>> the engine database, calculating apparent temperature for each record, then 
>> finding the max value.
>>
>> In V4, this would not have even been attempted.
>>
>> The solution is to add appTemp to the database. 
>>
>> On Thu, Jan 18, 2024 at 1:33 PM vince  wrote:
>>
>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.  
>>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive 
>>> period.
>>> The historical files dating back to 2014 generated fine.
>>>
>>> Unrelated but interesting.
>>>
>>>- this db shows the Belchertown taking too long issue
>>>- running htop I see it peg two cpus and StdReport aborts due to 
>>>time vs. the Simulator 300 sec on a pi3+
>>>- Belchertown 'always' takes longer than 300 secs and weewx always 
>>>aborts due 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Richard Whitcombe]

Can you explain in laymans terms how to do that?
Do you mean add a new DB field using SQL or similar and leave it blank or 
something else?

Or outline trouble shooting steps to isolate the cause in simple terms (ie 
familiar with linux and the basics of weewx but not the core detailed 
functionality).

On Friday 19 January 2024 at 08:12:14 UTC+9 Tom Keffer wrote:

> Using Blaine's database, I was able to isolate the performance problems.
>
> It's in the template records/index.html.tmpl
>
> Specifically, apparent temperature (appTemp). It does not appear in the 
> database, so a simple tag such as $alltime.appTemp.max requires searching 
> the engine database, calculating apparent temperature for each record, then 
> finding the max value.
>
> In V4, this would not have even been attempted.
>
> The solution is to add appTemp to the database. 
>
> On Thu, Jan 18, 2024 at 1:33 PM vince  wrote:
>
>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.  
>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive 
>> period.
>> The historical files dating back to 2014 generated fine.
>>
>> Unrelated but interesting.
>>
>>- this db shows the Belchertown taking too long issue
>>- running htop I see it peg two cpus and StdReport aborts due to time 
>>vs. the Simulator 300 sec on a pi3+
>>- Belchertown 'always' takes longer than 300 secs and weewx always 
>>aborts due to StdReport not completing in 300 secs
>>- yet the same installation with 'my' db works fine with no issues, 
>>so it 'has' to be content of the db being used doesn't it ?
>>- I have been unable to get Belchertown to succeed even at 600 secs 
>>archive period.
>>
>> Question - why would the db be 'locked' if a skin is reading it in order 
>> to process its contents ??
>>
>> last restart/failure...
>>
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx 
>> version 5.0.0
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is 
>> -0.21 seconds (positive is fast)
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding 
>> 'wx_binding' to database 'weewx.sdb'
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting 
>> backfill of daily summaries
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily summaries 
>> up to date
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main 
>> packet loop.
>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: 
>> Generated 8 files for report SeasonsReport in 3.96 seconds
>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator: 
>> Generated 30 images for report SeasonsReport in 4.80 seconds
>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5 
>> files to /home/pi/weewx-data/public_html
>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version 1.3.1
>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop 
>> exiting. Shutting engine down.
>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down 
>> StdReport thread
>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut 
>> down StdReport thread
>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database 
>> OperationalError exception: database is locked
>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__:  
>>  Waiting 2 minutes then retrying...
>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: 
>> Generated 12 files for report Belchertown in 617.64 seconds
>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40 
>> files to /home/pi/weewx-data/public_html/belchertown
>>
>>
>> weewx related open files:
>>
>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>> python3   1633pi  mem   REG  179,2   
>>  217360 272588 
>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>> python3   1633pi  mem   REG  179,2   
>>  224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>> python3   1633pi  mem   REG  179,2   
>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imaging.cpython-311-arm-linux-gnueabihf.so
>> python3   1633pi  mem   REG  179,2   
>> 2528660 271317 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Tom Keffer]

Using Blaine's database, I was able to isolate the performance problems.

It's in the template records/index.html.tmpl

Specifically, apparent temperature (appTemp). It does not appear in the
database, so a simple tag such as $alltime.appTemp.max requires searching
the engine database, calculating apparent temperature for each record, then
finding the max value.

In V4, this would not have even been attempted.

The solution is to add appTemp to the database.

On Thu, Jan 18, 2024 at 1:33 PM vince  wrote:

> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
> The Seasons skin NOAA file for 2024-01 'is' being updated every archive
> period.
> The historical files dating back to 2014 generated fine.
>
> Unrelated but interesting.
>
>- this db shows the Belchertown taking too long issue
>- running htop I see it peg two cpus and StdReport aborts due to time
>vs. the Simulator 300 sec on a pi3+
>- Belchertown 'always' takes longer than 300 secs and weewx always
>aborts due to StdReport not completing in 300 secs
>- yet the same installation with 'my' db works fine with no issues, so
>it 'has' to be content of the db being used doesn't it ?
>- I have been unable to get Belchertown to succeed even at 600 secs
>archive period.
>
> Question - why would the db be 'locked' if a skin is reading it in order
> to process its contents ??
>
> last restart/failure...
>
> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
> version 5.0.0
> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is
> -0.21 seconds (positive is fast)
> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
> 'wx_binding' to database 'weewx.sdb'
> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
> backfill of daily summaries
> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily summaries
> up to date
> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
> packet loop.
> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
> Generated 8 files for report SeasonsReport in 3.96 seconds
> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator: Generated
> 30 images for report SeasonsReport in 4.80 seconds
> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5
> files to /home/pi/weewx-data/public_html
> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version 1.3.1
> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
> exiting. Shutting engine down.
> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
> StdReport thread
> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut
> down StdReport thread
> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
> OperationalError exception: database is locked
> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__:   Waiting
> 2 minutes then retrying...
> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
> Generated 12 files for report Belchertown in 617.64 seconds
> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40
> files to /home/pi/weewx-data/public_html/belchertown
>
>
> weewx related open files:
>
> pi@pi3plus:~/weewx-data $ lsof|grep weew
> python3   1633pi  mem   REG  179,2
>  217360 272588
> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
> python3   1633pi  mem   REG  179,2
>  224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
> python3   1633pi  mem   REG  179,2
> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
> imaging.cpython-311-arm-linux-gnueabihf.so
> python3   1633pi  mem   REG  179,2
> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
> libastro.cpython-311-arm-linux-gnueabihf.so
> python3   1633pi  mem   REG  179,2
> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
> imagingmath.cpython-311-arm-linux-gnueabihf.so
> python3   1633pi  mem   REG  179,2
> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
> imagingft.cpython-311-arm-linux-gnueabihf.so
> python3   1633pi  

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[vince]

Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.  
The Seasons skin NOAA file for 2024-01 'is' being updated every archive 
period.
The historical files dating back to 2014 generated fine.

Unrelated but interesting.

   - this db shows the Belchertown taking too long issue
   - running htop I see it peg two cpus and StdReport aborts due to time 
   vs. the Simulator 300 sec on a pi3+
   - Belchertown 'always' takes longer than 300 secs and weewx always 
   aborts due to StdReport not completing in 300 secs
   - yet the same installation with 'my' db works fine with no issues, so 
   it 'has' to be content of the db being used doesn't it ?
   - I have been unable to get Belchertown to succeed even at 600 secs 
   archive period.

Question - why would the db be 'locked' if a skin is reading it in order to 
process its contents ??

last restart/failure...

Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx 
version 5.0.0
Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is 
-0.21 seconds (positive is fast)
Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding 
'wx_binding' to database 'weewx.sdb'
Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting backfill 
of daily summaries
Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily summaries 
up to date
Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main 
packet loop.
Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: 
Generated 8 files for report SeasonsReport in 3.96 seconds
Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator: Generated 
30 images for report SeasonsReport in 4.80 seconds
Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5 
files to /home/pi/weewx-data/public_html
Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version 1.3.1
Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record 
2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop exiting. 
Shutting engine down.
Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down 
StdReport thread
Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut 
down StdReport thread
Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database 
OperationalError exception: database is locked
Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__:   Waiting 
2 minutes then retrying...
Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: 
Generated 12 files for report Belchertown in 617.64 seconds
Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40 
files to /home/pi/weewx-data/public_html/belchertown


weewx related open files:

pi@pi3plus:~/weewx-data $ lsof|grep weew
python3   1633pi  mem   REG  179,2   
 217360 272588 
/home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
python3   1633pi  mem   REG  179,2   
 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
python3   1633pi  mem   REG  179,2   
1630340 271670 
/home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_imaging.cpython-311-arm-linux-gnueabihf.so
python3   1633pi  mem   REG  179,2   
2528660 271317 
/home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_libastro.cpython-311-arm-linux-gnueabihf.so
python3   1633pi  mem   REG  179,2 
56296 271675 
/home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_imagingmath.cpython-311-arm-linux-gnueabihf.so
python3   1633pi  mem   REG  179,2 
93216 271673 
/home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_imagingft.cpython-311-arm-linux-gnueabihf.so
python3   1633pi  mem   REG  179,2 
45064 271812 
/home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_namemapper.cpython-311-arm-linux-gnueabihf.so

When its cpus are pegged trying to process Belchertown:

python3   1633pi  cwd   DIR  179,2 
 4096 272808 /home/pi/weewx-data/skins/Belchertown
python3   1633pi  mem   REG  179,2   
 217360 272588 
/home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
python3   1633pi  mem   REG  179,2   
 224592 272587 

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Blaine]

Thank you for the reply! Yes, I have tried this many times. It continues to 
regenerate them with the missing data.

On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:

Have you tried completely deleting all NOAA reports and letting weewx 
regenerate them?

On Wed, Jan 17, 2024 at 9:32 AM Blaine  wrote:

Way back in 2020 my NOAA reports stopped populating their daily values. The 
average values for the month continue to be populated at the bottom of the 
table. Since 2020 every year or so I have spent hours banging my head 
against the wall following any and all instructions on the wiki, github, 
google and this user group to attempt to repair this issue. None of the 
published instructions seem to help. I have never been successful.

If this is a problem that can be solved I would be more than happy to pay 
someone to help fix this issue knowing full well there are no guarantees 
recovery is possible. This data is important to me and having essentially 
lost years worth with no chance of recovery is a real bummer.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/34583a3a-2093-4c1b-8b0a-6e7e99fa11ecn%40googlegroups.com.

Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Tom Keffer]

Have you tried completely deleting all NOAA reports and letting weewx
regenerate them?

On Wed, Jan 17, 2024 at 9:32 AM Blaine  wrote:

> Way back in 2020 my NOAA reports stopped populating their daily values.
> The average values for the month continue to be populated at the bottom of
> the table. Since 2020 every year or so I have spent hours banging my head
> against the wall following any and all instructions on the wiki, github,
> google and this user group to attempt to repair this issue. None of the
> published instructions seem to help. I have never been successful.
>
> If this is a problem that can be solved I would be more than happy to pay
> someone to help fix this issue knowing full well there are no guarantees
> recovery is possible. This data is important to me and having essentially
> lost years worth with no chance of recovery is a real bummer.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEDiu5ykPGVe7_6Kh2thW916%3Dt6MxaxeGkXYQcZg6G6JhQ%40mail.gmail.com.

[weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?

[Blaine]

Way back in 2020 my NOAA reports stopped populating their daily values. The 
average values for the month continue to be populated at the bottom of the 
table. Since 2020 every year or so I have spent hours banging my head 
against the wall following any and all instructions on the wiki, github, 
google and this user group to attempt to repair this issue. None of the 
published instructions seem to help. I have never been successful.

If this is a problem that can be solved I would be more than happy to pay 
someone to help fix this issue knowing full well there are no guarantees 
recovery is possible. This data is important to me and having essentially 
lost years worth with no chance of recovery is a real bummer.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com.