[weewx-development] Re: get data from external database

Hello Matteo, I think it is quite possible to have your own code filling the weewx DB and running weewx simply to create the plots and calculate the long term summaries. It would be a huge task to have weewx directly read your db format, so maintaining both DBs (your original one and weewx)

Re: [weewx-development] Re: V5.0 release candidate available

On Sunday 31 December 2023 at 11:59:12 am UTC+10 Cameron D wrote: It was cutting unused parameters out of V5 Seasons skin.con that got the time down to 8 seconds with those DBs. I was still worried about it taking 8 seconds, but I finally realised that was because I was wiping the target

Re: [weewx-development] Re: V5.0 release candidate available

On Sunday 31 December 2023 at 3:11:51 am UTC+10 Vince Skahan wrote: On Saturday, December 30, 2023 at 12:51:39 AM UTC-8 Cameron D wrote: Another difference between versions 4 and 5 reports that has had me baffled for far too long. For my V4 Seasons Skin I had deleted the sensor fields that I

Re: [weewx-development] Re: V5.0 release candidate available

Another difference between versions 4 and 5 reports that has had me baffled for far too long. For my V4 Seasons Skin I had deleted the sensor fields that I did not have, which led to empty fields in the skin file, such as: sensor_connections = sensor_batteries = sensor_voltages

Re: [weewx-development] Re: V5.0 release candidate available

Hi Tom, on the topic of the missing parameters in V5, do the preferences in the config file [StdWXCalculate] have any bearing on report generation, or they just when archive records are created? So for example if I changed windchill from "prefer_hardware" to "hardware" would it stop trying to

Re: [weewx-development] Re: V5.0 release candidate available

maybe they were only released as deb packages, which is how I installed mine. On Friday 29 December 2023 at 11:20:25 pm UTC+10 Matthias Manhart wrote: > How can i upgrade to rc1-4 ? I made the pip-upgrade 2h ago and i can see > only the release "5.0.0rc1" in my installation. > > Cameron D

Re: [weewx-development] Re: V5.0 release candidate available

Matthias, I had that problem with rc1-1 and it went away with rc1-2. I notice we are now at rc1-4 On Friday 29 December 2023 at 11:09:33 pm UTC+10 Matthias Manhart wrote: > I run weewx 5.0 on a test Raspberry Pi 4 since a few months and made the > upgrades continuously. I'm now on 5.0.0rc1.

Re: [weewx-development] Re: V5.0 release candidate available

I'm not sure. The reason mine are missing is that *I have chosen* them to be missing. I don't want them automatically re-created. With the V4 skin it was happy checking for the existence of each parameter and then ignoring any that weren't there. In my case, I have customised DBs and a

Re: [weewx-development] Re: V5.0 release candidate available

Well, I would not place too much value on my interpretation of the profile, but here is the sorted list of calls that totalled over 1 second. The total within sqlite.py is less than 20 seconds. It spent longer than that just converting to and fro between C and F Ordered by: internal time

Re: [weewx-development] Re: V5.0 release candidate available

wrote: > "'Cameron D' via weewx-development" > writes: > > > My assumption would be that any derivable value that you want to plot > > should be a column in the database. Even if you could speed it up, the > > calculation it would need to be by a factor approach

Re: [weewx-development] Re: V5.0 release candidate available

t_scalar() with the results. I'd have to think about that. It would > definitely be an extension to the xtypes API. > > Thanks for setting us on the right course, Cameron. > > > > > On Thu, Dec 28, 2023 at 3:38 AM 'Cameron D' via weewx-development < > weewx-de...@googlegroups.c

Re: [weewx-development] Re: V5.0 release candidate available

OK, this "ability to generate synthetic types" was the clue to the excessive cpu and time. I started with a copy of my main DB converted to sqlite and trimmed it to 5 minute intervals and ~100k archive records, so it is close to the benchmark db. I did a bit of (my first ever) python

Re: [weewx-development] Re: V5.0 release candidate available

In the cases of my problematic reports, it never prints the diagnostic summary lines at all. But I can tell that the largest gaps, based on file dates is generating each of index.html, then rss.xml, statistics.html and all the images are generated over a similar time in total to each of

Re: [weewx-development] Re: V5.0 release candidate available

Updated to rc1-2 and the problem of weectl looking in the wrong place for user modules seems to be fixed. On Thursday 28 December 2023 at 9:50:25 am UTC+10 Vince Skahan wrote: > The need to create the link 'was' the old bug I mentioned. It was fixed > quite a number of weeks ago, or so it

Re: [weewx-development] Re: V5.0 release candidate available

there much significance to the "version" number in the DB metadata? I see one of my DBs is version 1 and the other is V4? Cameron. On Thursday 28 December 2023 at 10:52:26 am UTC+10 Tom Keffer wrote: On Wed, Dec 27, 2023 at 4:18 PM 'Cameron D' via weewx-development < weewx-de...@

Re: [weewx-development] Re: V5.0 release candidate available

Tom, is there any chance that it is confused by me using mysql storage type of float rather than real? On Thursday 28 December 2023 at 10:12:54 am UTC+10 Tom Keffer wrote: > Back to the benchmarks. I made a small technical mistake. The comparison > now stands at: > > | Hardware

Re: [weewx-development] Re: V5.0 release candidate available

Vince, I think you misinterpreted my observations. Everything *had *been moved to /etc/weewx/bin/user when V5 upgraded, but something *in weectl *was only looking in the old location. I had to create a link at the old location to make weectl work in V5. I suppose I was a bit ambiguous in

Re: [weewx-development] V5.0 release candidate available

2 pm UTC+10 Tom Keffer wrote: >>> >>>> Can you please run the benchmark that Vince linked to earlier: >>>> https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation >>>> >>>> Unfortunately, it runs only under V5 (not v4.10). S

Re: [weewx-development] V5.0 release candidate available

>>> >>> Unfortunately, it runs only under V5 (not v4.10). Still, it will give us >>> an idea whether you're in the right ballpark for runtimes. >>> >>> -tk >>> >>> On Tue, Dec 26, 2023 at 5:38 PM 'Cameron D' via weewx-development

Re: [weewx-development] V5.0 release candidate available

://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation >> >> Unfortunately, it runs only under V5 (not v4.10). Still, it will give us >> an idea whether you're in the right ballpark for runtimes. >> >> -tk >> >> On Tue, Dec 26, 2023 at 5:3

Re: [weewx-development] V5.0 release candidate available

ration > > Unfortunately, it runs only under V5 (not v4.10). Still, it will give us > an idea whether you're in the right ballpark for runtimes. > > -tk > > On Tue, Dec 26, 2023 at 5:38 PM 'Cameron D' via weewx-development < > weewx-de...@googlegroups.com> wrote: > >>

Re: [weewx-development] Re: V5.0 release candidate available

Mine is an upgrade from 4.10.2, and probably all the way from the original v4, at which stage I think is was a clean apt install from V3 and I think that was when I switched to python 3 . My system is up-to-date Deb 12, so Python 3.11.2-1+b1 I can reproduce it by simply removing the symlink

Re: [weewx-development] V5.0 release candidate available

74 seconds clock time to run a seasons report looks like the same problem I am seeing. On Wednesday 27 December 2023 at 11:25:17 am UTC+10 rcst...@gmail.com wrote: Hey All, ... Here’s 4.10 Dec 26 17:00:25 raspi-server-misc weewx[25329] INFO weewx.cheetahgenerator: Generated 8 files for

[weewx-development] Re: V5.0 release candidate available

I ran some more tests with my main DB converted to sqlite, using weectl to run repots using the default Seasons skin. The times remained basically constant at about 4.5 minutes user+sys CPU as I kept deleting old records. As I got below 500k records the times dropped proportionally to the

[weewx-development] Re: V5.0 release candidate available

The wmr300 DB has 3.7million records, and the ecowitt DB has only 530k. The system is a VM running on an i5 - the VM is allocated 8GB ram and 4 cores. It was not stressed at all by weewx V4. The CPU usage is all in the python, not in the mariadb server. While the python code is thrashing

[weewx-development] Re: V5.0 release candidate available

The high cpu usage seems to *relate to the size of the DB*. I tried with my schema and a new sqlite db - no problems I migrated my mysql DB to sqlite and got the same problems, if not worse - elapsed time 18 minutes clock (10 min cpu user, 2 min sys) to run a default Seasons report and it

[weewx-development] Re: V5.0 release candidate available

Back to the reports themselves. I am running the GW1000 system without any reports, and will probably end up running cron jobs to occasionally update the reports. - I created an empty folder and pointed the config file there. - This is a heavily customised Seasons report, because it is

Re: [weewx-development] V5.0 release candidate available

I got basically this same result as Vince, trying to run a report manually. Running as user weewx or root gave same result. I have no venv, and no pip install. A pointer to the cause would be that *renaming /usr/share/weewx/user-2023xxx back to user fixed the problem*. I also tried a symlink

[weewx-development] Re: V5.0 release candidate available

A bit more info on the reports. 1. I disabled any reports from the GW1000 and cpu usage dropped to zero. 2. I set the WMR300 to just use the Seasons skin as-shipped, but there was no improvement. 3. I changed wmr300 config to write to a new empty directory (with suitable

[weewx-development] Re: V5.0 release candidate available

I tried an upgrade from a 4.10.2 simulator system just to see what happened. It looked OK, so I tried on my main system. - Debian 12, using the apt package upgrade. - I have two hardware units - Oregon WMR300 and EcoWitt 1000 (v0.6.0b2), both with custom databases and the gw1000 has

[weewx-development] Re: PostgreSQL support, which branch to target?

Thanks, I understand the issue a bit better now. It looks more like a case of whether the DB system is case-preserving. Sqlite is case-preserving but case-insensitive, whereas pg looks more like case-modifying and case-insensitive (unless identifiers are quoted). Running the mysql server in

[weewx-development] Re: PostgreSQL support, which branch to target?

I don't understand the problem. My MariaDB server is set to *case-sensitive* (file-system and table names). The column names are mainly camel case (using a trimmed down version of the old schema with original names, such as 'inTemp' and 'rain'). I have a second instance with an ecowitt DB where

[weewx-development] Re: Is there a better way of computing/displaying historic solar radiation readings?

The obvious question is what you need to do with the numbers. I have solar panels and I log the values separately so I don't use weewx for that - I got the weather station partly to help understand my PV output. The important long-term values to me are total energy over the day/month/etc. So,

[weewx-development] Re: genSql() error with MySQL

that would make sense. perhaps the sqlite library "helpfully" inserts the space. Not what you requested but probably what you intended. The python concatenation does not insert a space. On Wednesday, 24 August 2022 at 4:32:47 pm UTC+10 kk44...@gmail.com wrote: > I have got an idea: There is

[weewx-development] Re: genSql() error with MySQL

OK, so , as I might have expected, the weewx mysql layer replaces '?' with the '%s' parameter marker. And there is a specific comment about handling timespan tuples (weedb/mysql.py) just before it calls cursor.execute. So, back to what Vince said - you'll need the original error message. The

[weewx-development] Re: genSql() error with MySQL

The problem is you are combining a python formatted string with text for a parameterised query. For whatever bizarre reason, the python sql specs allow different database interfaces to use different parameter indicators. Mysql uses "%s" while sqlite uses "?". For a mysql query you obviously

[weewx-development] Re: WeeWX V4.8.0 available

There are various options for systemd described in the wiki On Thursday, 9 June 2022 at 11:09:28 pm UTC+10 nfbarg...@gmail.com wrote: > Matthew, > > Is there a copy of the Debian systemd unit file publicly available? I am > running Armbian on an

Re: [weewx-development] WeeWX v4.8.0 not starting with new systemd.service file on opensuse Leap 15.3

ive us the > particulars? > > - What operating system? > - Is this a reinstall, or a new install? > - If a reinstall, is this a new systemd file, or the old one? > > On Sun, Apr 24, 2022 at 1:56 AM 'Cameron D' via weewx-development < > weewx-de...@googlegroups.com>

Re: [weewx-development] WeeWX v4.8.0 not starting with new systemd.service file on opensuse Leap 15.3

I have just implemented this in my systemd service file and notice a journal full of LOOP packets. Is that why we also need the line saying... StandardOutput=null On Saturday, 23 April 2022 at 3:21:26 pm UTC+10 Bernhard R. wrote: > Hi matthew, > > confirmed, removing the pidfile option seems

Re: [weewx-development] Sieverts (Ionizing Radiation)

" ionizing_radiation " would be an obvious and unambiguous choice. I'd be reluctant to rely on the uSv/hour value, unless you know the source. The Geiger-Muller tube electronics looks like it is just reporting ionisation events per second, with no measure of the energy. Any "calibration

Re: [weewx-development] another deb11 deprecation warning

It was certainly there in Debian 10, which I noticed as soon as I created a systemd unit file for weewx. The other aspect that can trip up old scripts (usually my home-made ones) is that /run is mounted as tmpfs so subdirectories lose their properties on reboot. The ones under /var/run

Re: [weewx-development] Re: V4.6.0 beta 7

h install. > > Think of unit_system as a shorthand for a bunch of unit group settings. > > NB: this is still in flux. The option may be moved to sit under > [[Defaults]], instead of directly under [StdReport]. > > > > On Mon, Nov 22, 2021 at 6:42 AM 'Cameron D

Re: [weewx-development] Re: V4.6.0 beta 7

not always possible because some things cannot be > overridden. In particular, the templates themselves. > > Thanks for giving it a thorough try! > > > > On Sun, Nov 21, 2021 at 1:08 AM 'Cameron D' via weewx-development < > weewx-de...@googlegroups.com> wrote: > >

[weewx-development] Re: V4.6.0 beta 7

I see for beta6 you were asking for experience on customising Seasons skin. I have a renamed and customised Seasons skin so I thought I try it out. Customisations are fairly trivial - graph size, colours, content of each graph, one extra dataset from mem.py Debian 10, mysql DB system.

Re: [weewx-development] getting /home/weewx/bin into $PATH

On Thursday, 22 April 2021 at 2:34:11 pm UTC+10 Graham Eddy wrote: > maybe i did add the PATH mod myself - setting up systems so often for so > long that no doubt i do some things without thought… > Likewise - $HOME/bin is a standard setup on my systems - I do recall seeing it by default on

Re: [weewx-development] getting /home/weewx/bin into $PATH

> > For that user the correct place would be in their own profile files. >> > > Lean toward agree, 'if' the users would actually do so. History proves > the vast majority do not. This is the underlying issue. We can't make new > users who are disinclined to care about learning anything to

Re: [weewx-development] getting /home/weewx/bin into $PATH

Downsides? in reality, probably none as the likelihood of name clashes are slim. However, it would probably be against the general principals, because those are for all users on a system, while typically only a single user might have permission to access the system. For that user the correct

[weewx-development] Re: Clock drift and loop packet time resolution

Forget this - I was looking at raw packet data instead of returned data. On Sunday, 7 March 2021 at 6:57:47 pm UTC+10 Cameron D wrote: > For those who do not live in North America or Europe, the Oregon WMR300 > offers no correction to clock drift beyond manually prodding the console >

[weewx-development] Clock drift and loop packet time resolution

For those who do not live in North America or Europe, the Oregon WMR300 offers no correction to clock drift beyond manually prodding the console touch-pad. Mine drifts by about 90 seconds per year - not much, but enough to be irritating. Moreover, the loop packet data reports time only to the

[weewx-development] Re: What happens in Weewx around 3AM

The MySQL server may be the trigger but not the actual problem. Some time ago it used to be the case on my system that Weewx would exit whenever the MySQL server was restarted, such as during an upgrade. My recollection was that weewx was changed to try harder to restart under these conditions,