[weewx-development] Re: V5.0 release candidate available

['Cameron D' via weewx-development]

A bit more info on the reports.

   1. I disabled any reports from the GW1000 and cpu usage dropped to zero.
   2. I set the WMR300 to just use the Seasons skin as-shipped, but there 
   was no improvement.
   3. I changed wmr300 config to write to a new empty directory (with 
   suitable ownership and permissions) and things got much worse:


   - NOAA dir was created and populated.
   - A set of PNG files and inxex.html were written as expected.
   - the reports now timeout every minute, rather than every 2 minutes.
   - various support files are never copied across - seasons.css and js; 
   background, icons, font, lang folders do not exist (not sure whether some 
   are hangovers from prev versions.)
   - Most files are not updated when expected.


 For example, I deleted all day*.png files and, at the next update, noted 
the following:
   
   - the barometer and tempdew were the only images to reappear. 
   -  rss.xml was updated
   - html files celestial, tabular and telemetry were 1 minute old
   - statistics.html was 2 minutes old
   - index.html was 4 min old
   - NOAA had not been updated for 7 min
   - other files were older - css and js files were dated from when I 
   copied them over manually.
   - the files that do get updated within a report period are often dated 
   within 0.1 seconds of each other, but sometimes are seconds apart.


On Monday 25 December 2023 at 5:34:29 pm UTC+10 Cameron D wrote:


*Reports not completing:*
Both systems report via customised Seasons skins, and they *appear *to be 
giving reasonable updates to the reports. The WMR300 merges data from both 
databases and has a separate issue that I will report in detail later. The 
gw100 seems to give updated reports, including sensible plots and current 
conditions every 2 minutes.
They sample at 1 minute intervals and  every 2 minutes *both *systems 
independently report:
*INFO weewx.engine: Launch of report thread aborted: existing report thread 
still running*

*Both instances are each consuming 60 to 100% cpu permanently.*

When I stop (allow it to die) and restart, there is not much cpu activity 
for the first minute, so I would guess it is tied into thje report 
generation.

I'll have a go with the default skin...

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/4aacf7dc-ca61-4c71-9544-36c424dd8c59n%40googlegroups.com.

Re: [weewx-development] Use tags in Python extension code?

[Joel Bion]

Indeed! I share your appreciation for Weewx as a Python example. I’m retired, so didn’t have a “need” to learn Python, but did object oriented programming in Smalltalk and then C++ a long while ago, but ended up entering the management track, for good or ill. Nearing retirement, I set up WeeWX on an at-home server, and slowly used it as one of the examples to teach me Python, along with fail2ban and a couple of other things. I don’t say I am fluent in the language as I reserve that term for when I’ve built a multi-source file program “package” totaling thousands of lines, but I’ve grown quite comfortable editing source code for my particular unusual circumstances that are not generalizable - like tying in my sprinkler system to rain totals and soil moisture levels, etc. Sent from my iPhoneOn Dec 25, 2023, at 10:49 AM, nfbarg...@gmail.com  wrote:Thanks, Tom.I appreciate the tips.  Right now I am issuing SQL statements directly into the 'archive' and 'archive_day_rain' tablesfor the barometer trend and day rain values.  This works as the access only occurs with each new record.My basic understanding of Python is fine, it's all of the abstraction through the various classes where I tend to get lost.Still this little project of mine has helped me immensely as it has given me a reason to understand the WeeWX internalsmuch better and the way an object oriented project ties together.- NateOn Sunday, December 24, 2023 at 7:39:34 PM UTC-6 Tom Keffer wrote:In order to do this, a lot of infrastructure has to be in place. Take a look at the test file test_daily.py, function testTags (line 206).It sets up a TimeBinder object (line 222). Once you have that, you can do things like    print(tagStats.trend.barometer)For many of these aggregates, you can use the xtypes system, which is a lot simpler to access. For example, to get the day's rain    val_tuple = xtypes.get_aggregate('rain', timespan, 'sum', db_manager)where timespan is the start/stop times over which the aggregation is to be taken, and db_manager is an open database manager object. It returns a ValueTuple.If you're not a Python programmer, it's going to be tough. :-(-tkOn Sun, Dec 17, 2023 at 3:38 PM nfbarg...@gmail.com  wrote:This likely has everything to do with my lack of Python acuity than WeeWX.When obtaining values for a user extension that does not have an associated skin so I am not using the Cheetah Generator, how might I call into the tag system to get the data such as 'trend.barometer'?  in cheetahgenerator.py I see the Stats class that looks like it should be what should be called but I don't see it subclassed anywhere but is part of 'default_search_list'.Primarily I am interested in 'trend.barometer' and 'day.rain'.  The rest I am getting from the latest archive record.  I've not found any extensions that use the tag system like this that I can be inspired by so I am asking the experts.TIA- Nate



-- 
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b4a1b9d6-2e1e-4132-8731-477c2d6018f8n%40googlegroups.com.





-- 
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/882990f1-4e76-4bb7-a63d-40e8c25b387cn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/04AEE55C-110D-4BB2-8BC7-63BD50F991B9%40gmail.com.

[weewx-development] Re: V5.0 release candidate available

['Cameron D' via weewx-development]

Back to the reports themselves.
I am running the GW1000 system without any reports, and will probably end 
up running cron jobs to occasionally update the reports.


   - I created an empty folder and pointed the config file there.
   - This is a heavily customised Seasons report, because it is basically 
   indoor air/living quality only.
   - I ranweectl report run SeasonsReport 
--config=/etc/weewx/ewitt.conf
   - then  ls -lrt --full-time to show the increment in file creation times.
   - my customised report has 5 daily, 4 weekly, 3 monthly and 5 yearly 
   plots, so not a particular burdon.

The results show the report  files are generated in short bursts, then a 
gap, then a few more, etc.  The sequence is...

   - start at 2:14.6 seconds
   - 14.8 s NOAA folder populated by 14.8 seconds
   - 2.5 s later index.html, followed statistics
   - 0.9 s gap to  telemetry, tabular &  celestial, which were within 0.02 s
   - 40 s gap to rss.xml ,
   - 0.04 sec gap to the start of the PNG files daybarom and daytempdew
   - 13s gap to dayhumidity, dayPM, dayco2, weekbarom and weektempdew (all 
   within 0.25s)
   - 21s gap to weekPM, weekco2, monthbarom, monthtempdew (all within 0.2 s)
   - 10.3 s gap to monthco2, yearbarom, and the remaining 4 year summaries. 
   (within 0.1s)
   - 10.2s gap to seasons.js, css, font and favicon.ico (all at identical 
   times - to the nanosecond!)



On Monday 25 December 2023 at 7:41:15 pm UTC+10 Cameron D wrote:

> A bit more info on the reports.
>
>1. I disabled any reports from the GW1000 and cpu usage dropped to 
>zero.
>2. I set the WMR300 to just use the Seasons skin as-shipped, but there 
>was no improvement.
>3. I changed wmr300 config to write to a new empty directory (with 
>suitable ownership and permissions) and things got much worse:
>
>
>- NOAA dir was created and populated.
>- A set of PNG files and inxex.html were written as expected.
>- the reports now timeout every minute, rather than every 2 minutes.
>- various support files are never copied across - seasons.css and js; 
>background, icons, font, lang folders do not exist (not sure whether some 
>are hangovers from prev versions.)
>- Most files are not updated when expected.
>
>
>  For example, I deleted all day*.png files and, at the next update, noted 
> the following:
>
>- the barometer and tempdew were the only images to reappear. 
>-  rss.xml was updated
>- html files celestial, tabular and telemetry were 1 minute old
>- statistics.html was 2 minutes old
>- index.html was 4 min old
>- NOAA had not been updated for 7 min
>- other files were older - css and js files were dated from when I 
>copied them over manually.
>- the files that do get updated within a report period are often dated 
>within 0.1 seconds of each other, but sometimes are seconds apart.
>
>
> On Monday 25 December 2023 at 5:34:29 pm UTC+10 Cameron D wrote:
>
>
> *Reports not completing:*
> Both systems report via customised Seasons skins, and they *appear *to be 
> giving reasonable updates to the reports. The WMR300 merges data from both 
> databases and has a separate issue that I will report in detail later. The 
> gw100 seems to give updated reports, including sensible plots and current 
> conditions every 2 minutes.
> They sample at 1 minute intervals and  every 2 minutes *both *systems 
> independently report:
> *INFO weewx.engine: Launch of report thread aborted: existing report 
> thread still running*
>
> *Both instances are each consuming 60 to 100% cpu permanently.*
>
> When I stop (allow it to die) and restart, there is not much cpu activity 
> for the first minute, so I would guess it is tied into thje report 
> generation.
>
> I'll have a go with the default skin...
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/a6b116db-2356-499c-8bd4-5e61455cd4e8n%40googlegroups.com.

[weewx-development] Re: V5.0 release candidate available

[Vince Skahan]

Cameron several of us have run v5 with very large db of over 10 years data 
on pi4 or lesser boxes without such issue, so a bit more data from you 
would be helpful. How big a size are you running ? On what hardware ?

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/3666d062-813b-4777-9350-9e75baa43b1fn%40googlegroups.com.

[weewx-development] Re: V5.0 release candidate available

['Cameron D' via weewx-development]

The high cpu usage seems to *relate to the size of the DB*.

I tried with my schema and a new sqlite db - no problems

I migrated my mysql DB to sqlite and got the same problems, if not worse - 
elapsed time 18 minutes clock (10 min cpu user, 2 min sys)  to run a 
default Seasons report and it failed to create the index.html file.

Another time it took 10 minutes, created an index.html, but only generated 
a single png file.

I'm out of ideas.

On Tuesday 26 December 2023 at 12:42:59 pm UTC+10 Cameron D wrote:

> Back to the reports themselves.
> I am running the GW1000 system without any reports, and will probably end 
> up running cron jobs to occasionally update the reports.
>
>
>- I created an empty folder and pointed the config file there.
>- This is a heavily customised Seasons report, because it is basically 
>indoor air/living quality only.
>- I ranweectl report run SeasonsReport 
> --config=/etc/weewx/ewitt.conf
>- then  ls -lrt --full-time to show the increment in file creation 
>times.
>- my customised report has 5 daily, 4 weekly, 3 monthly and 5 yearly 
>plots, so not a particular burdon.
>
> The results show the report  files are generated in short bursts, then a 
> gap, then a few more, etc.  The sequence is...
>
>- start at 2:14.6 seconds
>- 14.8 s NOAA folder populated by 14.8 seconds
>- 2.5 s later index.html, followed statistics
>- 0.9 s gap to  telemetry, tabular &  celestial, which were within 
>0.02 s
>- 40 s gap to rss.xml ,
>- 0.04 sec gap to the start of the PNG files daybarom and daytempdew
>- 13s gap to dayhumidity, dayPM, dayco2, weekbarom and weektempdew 
>(all within 0.25s)
>- 21s gap to weekPM, weekco2, monthbarom, monthtempdew (all within 0.2 
>s)
>- 10.3 s gap to monthco2, yearbarom, and the remaining 4 year 
>summaries. (within 0.1s)
>- 10.2s gap to seasons.js, css, font and favicon.ico (all at identical 
>times - to the nanosecond!)
>
>
>
> On Monday 25 December 2023 at 7:41:15 pm UTC+10 Cameron D wrote:
>
>> A bit more info on the reports.
>>
>>1. I disabled any reports from the GW1000 and cpu usage dropped to 
>>zero.
>>2. I set the WMR300 to just use the Seasons skin as-shipped, but 
>>there was no improvement.
>>3. I changed wmr300 config to write to a new empty directory (with 
>>suitable ownership and permissions) and things got much worse:
>>
>>
>>- NOAA dir was created and populated.
>>- A set of PNG files and inxex.html were written as expected.
>>- the reports now timeout every minute, rather than every 2 minutes.
>>- various support files are never copied across - seasons.css and js; 
>>background, icons, font, lang folders do not exist (not sure whether some 
>>are hangovers from prev versions.)
>>- Most files are not updated when expected.
>>
>>
>>  For example, I deleted all day*.png files and, at the next update, noted 
>> the following:
>>
>>- the barometer and tempdew were the only images to reappear. 
>>-  rss.xml was updated
>>- html files celestial, tabular and telemetry were 1 minute old
>>- statistics.html was 2 minutes old
>>- index.html was 4 min old
>>- NOAA had not been updated for 7 min
>>- other files were older - css and js files were dated from when I 
>>copied them over manually.
>>- the files that do get updated within a report period are often 
>>dated within 0.1 seconds of each other, but sometimes are seconds apart.
>>
>>
>> On Monday 25 December 2023 at 5:34:29 pm UTC+10 Cameron D wrote:
>>
>>
>> *Reports not completing:*
>> Both systems report via customised Seasons skins, and they *appear *to 
>> be giving reasonable updates to the reports. The WMR300 merges data from 
>> both databases and has a separate issue that I will report in detail later. 
>> The gw100 seems to give updated reports, including sensible plots and 
>> current conditions every 2 minutes.
>> They sample at 1 minute intervals and  every 2 minutes *both *systems 
>> independently report:
>> *INFO weewx.engine: Launch of report thread aborted: existing report 
>> thread still running*
>>
>> *Both instances are each consuming 60 to 100% cpu permanently.*
>>
>> When I stop (allow it to die) and restart, there is not much cpu activity 
>> for the first minute, so I would guess it is tied into thje report 
>> generation.
>>
>> I'll have a go with the default skin...
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/b05efd6e-f8a9-4d02-9a46-ebeb894d2f5dn%40googlegroups.com.

Re: [weewx-development] V5.0 release candidate available

['Cameron D' via weewx-development]

I got basically this same result as Vince, trying to run a report manually.
Running as user weewx or root gave same result.
I have no venv, and no pip install.

A pointer to the cause would be that *renaming 
/usr/share/weewx/user-2023xxx  back to user fixed the problem*.

I also tried a symlink from /usr/share/weewx/user pointing to 
/etc/weewx/bin/user and *that worked also*.  Including being able to run as 
user weewx.

And got me back to my other problem, that the report took a long time to 
run - 1 minute and 30 seconds to generate the full list. I'll report on 
that separately.

On Saturday 23 December 2023 at 11:30:21 am UTC+10 Vince Skahan wrote:

> I hand-edited the diff into 
> weewx-venv/lib/python3.11/site-packages/weecfg/extension.py which worked 
> for installing the extensions, but I get a similar message trying to run 
> reports manually
>
> You might try this on a current fully patched up version to see if it 
> works with all the updates you made after release to pypi just in case...
>
> (weewx-venv) vagrant@deb12:~/weewx-data$ weectl report run 
> --config=simulator.conf
> Using configuration file simulator.conf
> All enabled reports will be run.
> Generating as of last timestamp in the database.
> Traceback (most recent call last):
>   File "/home/vagrant/weewx-venv/bin/weectl", line 8, in 
> sys.exit(main())
>  ^^
>   File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py", 
> line 66, in main
> namespace.func(namespace)
>   File 
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py",
>  
> line 96, in dispatch
> namespace.action_func(config_dict, namespace)
>   File 
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/report_cmd.py",
>  
> line 92, in run_reports
> weectllib.report_actions.run_reports(config_dict,
>   File 
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/report_actions.py",
>  
> line 84, in run_reports
> engine = weewx.engine.DummyEngine(config_dict)
>  ^
>   File 
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", 
> line 89, in __init__
> self.loadServices(config_dict)
>   File 
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", 
> line 157, in loadServices
> obj = weeutil.weeutil.get_object(svc)(self, config_dict)
>   ^^^
>   File 
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weeutil/weeutil.py", 
> line 1404, in get_object
> module = importlib.import_module(module_name)
>  
>   File "/usr/lib/python3.11/importlib/__init__.py", line 126, in 
> import_module
> return _bootstrap._gcd_import(name[level:], package, level)
>
>   File "", line 1206, in _gcd_import
>   File "", line 1178, in _find_and_load
>   File "", line 1128, in 
> _find_and_load_unlocked
>   File "", line 241, in 
> _call_with_frames_removed
>   File "", line 1206, in _gcd_import
>   File "", line 1178, in _find_and_load
>   File "", line 1142, in 
> _find_and_load_unlocked
>
>
> On Friday, December 22, 2023 at 4:51:49 PM UTC-8 Tom Keffer wrote:
>
>> Thanks for spotting that, Vince!
>>
>> Fixed in commit 256cac5 
>> 
>> .
>>
>> On Fri, Dec 22, 2023 at 4:31 PM Vince Skahan  wrote:
>>
>>> Tom - your xaggs extension isn't installing.  I see the identical issue 
>>> with one of my custom extensions too.
>>>
>>> (weewx-venv) vagrant@deb12:~/adds$ weectl extension install 
>>> --config=/home/vagrant/weewx-data/simulator.conf weewx-xaggs-master/
>>> Using configuration file /home/vagrant/weewx-data/simulator.conf
>>> Request to install 'weewx-xaggs-master/'.
>>> Traceback (most recent call last):
>>>   File "/home/vagrant/weewx-venv/bin/weectl", line 8, in 
>>> sys.exit(main())
>>>  ^^
>>>   File 
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py", line 66, 
>>> in main
>>> namespace.func(namespace)
>>>   File 
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py",
>>>  
>>> line 96, in dispatch
>>> namespace.action_func(config_dict, namespace)
>>>   File 
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/extension_cmd.py",
>>>  
>>> line 116, in install_extension
>>> ext.install_extension(namespace.source)
>>>   File 
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py",
>>>  
>>> line 125, in install_extension
>>> extension_name = self.install_from_dir(extension_path)
>>>  ^
>>>   File 
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py",
>>>  
>>> line 201, in install_from_dir
>>> save_config |= 

Re: [weewx-development] Use tags in Python extension code?

[nfbarg...@gmail.com]

Thanks, Tom.

I appreciate the tips.  Right now I am issuing SQL statements directly into 
the 'archive' and 'archive_day_rain' tables
for the barometer trend and day rain values.  This works as the access only 
occurs with each new record.

My basic understanding of Python is fine, it's all of the abstraction 
through the various classes where I tend to get lost.
Still this little project of mine has helped me immensely as it has given 
me a reason to understand the WeeWX internals
much better and the way an object oriented project ties together.

- Nate

On Sunday, December 24, 2023 at 7:39:34 PM UTC-6 Tom Keffer wrote:

> In order to do this, a lot of infrastructure has to be in place. Take a 
> look at the test file test_daily.py 
> , 
> function testTags 
> 
>  
> (line 206).
>
> It sets up a TimeBinder object (line 222). Once you have that, you can do 
> things like
>
> print(tagStats.trend.barometer)
>
> For many of these aggregates, you can use the xtypes 
>  system, which is a lot 
> simpler to access. For example, to get the day's rain
>
> val_tuple = xtypes.get_aggregate('rain', timespan, 'sum', db_manager)
>
> where timespan is the start/stop times over which the aggregation is to be 
> taken, and db_manager is an open database manager object. It returns a 
> ValueTuple .
>
> If you're not a Python programmer, it's going to be tough. :-(
>
> -tk
>
> On Sun, Dec 17, 2023 at 3:38 PM nfbarg...@gmail.com  
> wrote:
>
>> This likely has everything to do with my lack of Python acuity than WeeWX.
>>
>> When obtaining values for a user extension that does not have an 
>> associated skin so I am not using the Cheetah Generator, how might I call 
>> into the tag system to get the data such as 'trend.barometer'?  in 
>> cheetahgenerator.py I see the Stats class that looks like it should be what 
>> should be called but I don't see it subclassed anywhere but is part of 
>> 'default_search_list'.
>>
>> Primarily I am interested in 'trend.barometer' and 'day.rain'.  The rest 
>> I am getting from the latest archive record.  I've not found any extensions 
>> that use the tag system like this that I can be inspired by so I am asking 
>> the experts.
>>
>> TIA
>>
>> - Nate
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-developm...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-development/b4a1b9d6-2e1e-4132-8731-477c2d6018f8n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/882990f1-4e76-4bb7-a63d-40e8c25b387cn%40googlegroups.com.