Re: [weewx-development] Format of weewx-variables/datafields for arithmetic (service development)?

[Tom Keffer]

r.
>>>>>>
>>>>>> Thanks!
>>>>>> On Friday, May 3, 2024 at 7:30:12 AM UTC-5 Michael Frotscher wrote:
>>>>>>
>>>>>>> Thanks, guys!
>>>>>>> I see my mistake now. It's not about strings/floats at all, but I
>>>>>>> wrongly assumed that all measurements that have values assigned to them
>>>>>>> (via LOOP or whatever) would already be defined as global variables.
>>>>>>> That's obviously not the case, but assigning the last value to the
>>>>>>> variable via "event.record['pressure']" (as Frantisek pointed out) is
>>>>>>> necessary. I'm still not sure why that needs to be in single quotes,
>>>>>>> indicating a string and not a number.
>>>>>>> I would have expected something like "event.record(pressure)".
>>>>>>>
>>>>>>> I have to admit that I have not found this in any examples out
>>>>>>> there, and not in the documentation for sure.
>>>>>>>
>>>>>>> But it's now doing what I want.
>>>>>>>
>>>>>>> On Friday, May 3, 2024 at 12:16:50 AM UTC-5 František Slimařík wrote:
>>>>>>>
>>>>>>>> I guess you want something like this:
>>>>>>>>
>>>>>>>> if event.record['pressure'] != None:
>>>>>>>>   newpressure = (event.record['pressure'] * 10)
>>>>>>>>   event.record['pb'] = newpressure
>>>>>>>>
>>>>>>>> Dne pátek 3. května 2024 v 3:38:01 UTC+2 uživatel Tom Keffer napsal:
>>>>>>>>
>>>>>>>>> On Thu, May 2, 2024 at 6:32 PM Michael Frotscher <
>>>>>>>>> frot...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Here's the full code of my service:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>def new_archive_packet(self, event):
>>>>>>>>>>
>>>>>>>>>>   if 'pressure' != None:
>>>>>>>>>> newpressure = ('pressure' * 10)
>>>>>>>>>> event.record['pb'] = newpressure
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Strictly interpreted, you first ask whether the string
>>>>>>>>> "'pressure'" is equal to None. It's not, so we proceed to the next
>>>>>>>>> statement. Now you're trying to multiply a string ('pressure') by 10.
>>>>>>>>> Surprisingly, this will succeed, but likely does not give you the 
>>>>>>>>> results
>>>>>>>>> you expect. The variable "newpressure" will actually be set to the 
>>>>>>>>> string
>>>>>>>>> 'pressurepressurepressurepressurepressurepressurepressurepressurepressurepressure'.
>>>>>>>>> That is, the string 'pressure' concatenated 10 times.
>>>>>>>>>
>>>>>>>>> You want a *variable* pressure, not the literal string
>>>>>>>>> 'pressure'. Where will it come from?
>>>>>>>>>
>>>>>>>>> I would suggest taking an online Python course if this is
>>>>>>>>> unfamiliar to you.
>>>>>>>>>
>>>>>>>>> -tk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-development" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx-developm...@googlegroups.com.
>>>>>>
>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-development/8a2868e8-7f39-4038-acd5-c783572e2226n%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/weewx-development/8a2868e8-7f39-4038-acd5-c783572e2226n%40googlegroups.com?utm_medium=email_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/cc1d11ae-3adf-46f5-a65a-c6da05880e73n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/cc1d11ae-3adf-46f5-a65a-c6da05880e73n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/44a5a9c6-491f-4c10-a777-78f310a2a04dn%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/44a5a9c6-491f-4c10-a777-78f310a2a04dn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAbG1qhHAiCHKxupKpJv2POwiwiNechkd2PsdqLVv2zGg%40mail.gmail.com.

Re: [weewx-development] rsync_path

[Tom Keffer]

V5.1 will have a feature that allows any arbitrary rsync command option to
be specified.

See issue #951 .

On Fri, May 3, 2024 at 10:15 PM Clay Jackson  wrote:

> My ISP does not support rsync as  a standard; but does allow users to use
> their own copy.
>
> This requires an option, ""--rsync-path" added to the remote command.
>
> I've created a local copy of rysncupload.py and added that using the
> cmd.extend call.
>
> If the group thinks anyone else would find it useful, I can add a
> configuration option and contribute it.   Would probably take me a couple
> of weeks given my "day job" schedule.
>
> Thoughts?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/28207479-b4cc-4cd5-9d6e-27b84bbd9b18n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAsE43Fdkc0seW1kotLroZCcUgjDP%2BCJ9CKFRYw0nZwjw%40mail.gmail.com.

Re: [weewx-development] Format of weewx-variables/datafields for arithmetic (service development)?

[Tom Keffer]

On Thu, May 2, 2024 at 6:32 PM Michael Frotscher 
wrote:

> Here's the full code of my service:
>
>def new_archive_packet(self, event):
>
>   if 'pressure' != None:
> newpressure = ('pressure' * 10)
> event.record['pb'] = newpressure
>


Strictly interpreted, you first ask whether the string "'pressure'" is
equal to None. It's not, so we proceed to the next statement. Now you're
trying to multiply a string ('pressure') by 10. Surprisingly, this will
succeed, but likely does not give you the results you expect. The variable
"newpressure" will actually be set to the string
'pressurepressurepressurepressurepressurepressurepressurepressurepressurepressure'.
That is, the string 'pressure' concatenated 10 times.

You want a *variable* pressure, not the literal string 'pressure'. Where
will it come from?

I would suggest taking an online Python course if this is unfamiliar to
you.

-tk

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDDunC-ku2AZ%3D1oGT1sMs_eS9NABsMtp9%3DSu5iE1ccpRQ%40mail.gmail.com.

Re: [weewx-development] Format of weewx-variables/datafields for arithmetic (service development)?

[Tom Keffer]

You're going to have to give us more information about where your variable
"pressure" came from.

Other bits, probably not related to your problem.

1. Never assume a variable always contains a valid value. Every instrument
fails.

2. To compare with None use

if pressure is not None:
  foo = do arithmetic
else:
  foo = do nothing

3. Pressure with single quotes ('pressure') is a string. Variables are
unadorned.

4. The variable pressure in my example is undefined because in my little
example I didn't define it. I'm assuming you defined it somewhere, which
brings us back to my first paragraph above.


On Thu, May 2, 2024 at 6:22 PM Michael Frotscher 
wrote:

> Tom,
>
> Thanks for the reply.
> The exact error is: CRITICAL __main__:   ValueError: could not
> convert string to float: 'pressure'
> and weewx crashes when it encounters that.
>
> I have considered that the value might be Null (or None, rather), which is
> why I chose a variable that's fed from my barometer, so should always
> contain a valid value.
> But I have also added a check for that into my code, with the same result:
>
> if 'pressure' != None:
>   foo = do arithmetic
> else:
>   foo = do nothing
>
> FWIW, I'm using the "electricity" service example, and the code in
> question is in the "def new_archive_record(self, event):" section.
>
> But is it correct to refer to the variable as 'pressure'? With the single
> quotes?
> Because if I don't, like you have in your example, I get this:
>
> NameError: name 'pressure' is not defined
>
>
> On Thursday, May 2, 2024 at 7:52:25 PM UTC-5 Tom Keffer wrote:
>
>> When a value comes off the database it's always either a number, or the
>> Python value "None", with very few exceptions. You didn't show us the
>> error, but the issue is likely to be that "pressure" has the value "None",
>> which signals bad or missing data.
>>
>> Be sure to read the document *Notes for developers
>> <https://www.weewx.com/docs/5.0/devnotes/>*. It includes some
>> information about None.
>>
>> So, your example becomes,
>>
>> *foo = pressure * 10 if pressure is not None else None*
>>
>> Then, of course, you have to remember that "foo" could be None. This is a
>> very common pattern in WeeWX.
>>
>> Then again, perhaps the error is completely different and it's not due to
>> the value None. If you're still having trouble, show us the error.
>>
>> -tk
>>
>>
>> On Thu, May 2, 2024 at 5:44 PM Michael Frotscher 
>> wrote:
>>
>>> All,
>>> I'm trying to write a simple service that creates a new derived
>>> measurement based on values already in the weewx database. Ultimately, I
>>> want to get a running total of the rainfall, but I'm not there yet. Also,
>>> not a Python developer by trade, but I'm trying.
>>>
>>> My issue is: when I try to do arithmetic, the variables have, of course,
>>> to be numbers, be it int or float. I'll take the 'pressure' measurement as
>>> an example. Looking into the database, it seems to be a float, but it could
>>> also be a string.
>>> But trying to use it that way, or trying to convert the string into a
>>> float, always gives me errors (could not convert string to float). I've
>>> tried stripping invisible whitespaces from the string (if it is one), but
>>> the error remains.
>>> Here's the relevant code, one of the many I've tried.
>>>
>>> So how do I get a weewx-variable into usable form? I'm probably missing
>>> something pretty basic here.
>>> Thanks!
>>> BTW, is there a way to test-run a service file? Like have the python
>>> code run in the weewx-environment?
>>>
>>> foo = (float('pressure') * 10)
>>>
>>> or
>>>
>>> pressure_trim = 'pressure'.strip()
>>> foo = (float(pressure_trim) * 10)
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/f08bf769-d2b2-4840-b0a3-2feeef33ea67n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/f08bf769-d2b2-4840-b0a3-2feeef33ea67n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the G

Re: [weewx-development] Format of weewx-variables/datafields for arithmetic (service development)?

[Tom Keffer]

When a value comes off the database it's always either a number, or the
Python value "None", with very few exceptions. You didn't show us the
error, but the issue is likely to be that "pressure" has the value "None",
which signals bad or missing data.

Be sure to read the document *Notes for developers
*. It includes some information
about None.

So, your example becomes,

*foo = pressure * 10 if pressure is not None else None*

Then, of course, you have to remember that "foo" could be None. This is a
very common pattern in WeeWX.

Then again, perhaps the error is completely different and it's not due to
the value None. If you're still having trouble, show us the error.

-tk


On Thu, May 2, 2024 at 5:44 PM Michael Frotscher 
wrote:

> All,
> I'm trying to write a simple service that creates a new derived
> measurement based on values already in the weewx database. Ultimately, I
> want to get a running total of the rainfall, but I'm not there yet. Also,
> not a Python developer by trade, but I'm trying.
>
> My issue is: when I try to do arithmetic, the variables have, of course,
> to be numbers, be it int or float. I'll take the 'pressure' measurement as
> an example. Looking into the database, it seems to be a float, but it could
> also be a string.
> But trying to use it that way, or trying to convert the string into a
> float, always gives me errors (could not convert string to float). I've
> tried stripping invisible whitespaces from the string (if it is one), but
> the error remains.
> Here's the relevant code, one of the many I've tried.
>
> So how do I get a weewx-variable into usable form? I'm probably missing
> something pretty basic here.
> Thanks!
> BTW, is there a way to test-run a service file? Like have the python code
> run in the weewx-environment?
>
> foo = (float('pressure') * 10)
>
> or
>
> pressure_trim = 'pressure'.strip()
> foo = (float(pressure_trim) * 10)
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/f08bf769-d2b2-4840-b0a3-2feeef33ea67n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDAqQiV9Yf0MRvM_3RwhDWWBJKCFUsm-Wh%2BZ99OWfpesQ%40mail.gmail.com.

Re: [weewx-development] Change order that the engine instantiates the DBBinder and the driver

[Tom Keffer]

Sure! That sounds pretty reasonable.

On Thu, May 2, 2024 at 8:35 AM bell...@gmail.com 
wrote:

> Background:
> I am experimenting with a driver/service pair that performs a
> ‘lightweight’ (Not sure a reliable synchronization can be light weight)
> synchronization of sqlite DBs over MQTT v5. When the driver is
> instantiated, it retrieves the ‘lastGoodStamp’ from the database. It then
> makes a MQTT request to the primary DB for all records since that
> timestamp. By the time genStartupRecords is called, the catchup data should
> (hopefully) be available. But, the DBBinder has not been instantiated yet.
> I can get around this using ‘weewx.manager.open_manager’.
>
> I’m wondering if you would be open to a pull request that just moves
> ‘self.db_binder = weewx.manager.DBBinder(config_dict)’ prior to ‘
> self.setupStation(config_dict)? Then when the driver retrieves the
> db_manager, it would be cached and available to the rest of the WeeWX
> pipeline/infrastructure.
>
> Thanks. rich
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/5e16890b-b3c3-41b2-aa79-f441dda52625n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECnwuQJuWOx0DE_72AtBKMSD5hvBfH7QJS6EwtmVJ%2BccA%40mail.gmail.com.

Re: [weewx-development] I wrote to new Search List Extensions (SLE) but the script is repeated 18 times

[Tom Keffer]

I'm sorry, but that is way too much custom code for me to debug.

If you don't have a good debugger (I can recommend PyCharm), just put in
print statements that print out the stack each time your extension is
invoked. It will give you some ideas about what is activating it.

On Wed, Mar 20, 2024 at 6:56 AM Gianluca Contessa 
wrote:

> the sh files that I get, I run them via crontab every few hours
>
> Il giorno mercoledì 20 marzo 2024 alle 13:37:23 UTC+1 Tom Keffer ha
> scritto:
>
>> I don't know what I'm looking at. Which messages are from your extension?
>> The HTTP GET?
>>
>> If so, we would have to see how it is used to offer an opinion. How is it
>> referenced in your templates? Which templates is it used in?
>>
>>
>>
>> On Wed, Mar 20, 2024 at 5:08 AM Gianluca Contessa 
>> wrote:
>>
>>> I wrote to new Search List Extensions (SLE). Everything works fine, but
>>> I noticed from the weewx log that the script is repeated 18 times, and not
>>> just once as it should be. in fact, weewx delays more than a minute to
>>> generate the files. I don't understand where is the issue.
>>>
>>>
>>> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> https://www.santodelgiorno.it:443 "GET / HTTP/1.1" 200 12219
>>>
>>> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> Starting new HTTPS connection (1): www.santodelgiorno.it:443
>>>
>>> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> https://www.santodelgiorno.it:443 "GET /san-giovanni-nepomuceno/
>>> HTTP/1.1" 200 17870
>>>
>>> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> Starting new HTTPS connection (1): my.meteoblue.com:443
>>>
>>> Mar 20 10:25:44 raspberrypi weewxd[11529]: INFO weewx.restx: MQTT:
>>> Published record 2024-03-20 10:25:44 CET (1710926744)
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> https://my.meteoblue.com:443 "GET
>>> /images/meteogram?temperature_units=C_units=kmh_units=mm=it=40.6249=15.1035=138=Europe%2FRome=jhMJTOUVRNvs25m4=it_name=Quadrivio_units=kmh=019aad2a754640de4dcc7cef14e5a258
>>> HTTP/1.1" 200 82323
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
>>> STREAM b'IHDR' 16 13
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
>>> STREAM b'PLTE' 41 768
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
>>> STREAM b'tRNS' 821 256
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
>>> STREAM b'pHYs' 1089 9
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
>>> STREAM b'IDAT' 1110 65536
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: INFO user.richiesta_token:
>>> Immagine salvata con successo in
>>> /var/www/html/weewx/previsione/previsione.png
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> Starting new HTTPS connection (1): www.3bmeteo.com:443
>>>
>>> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> https://www.3bmeteo.com:443 "GET /meteo/quadrivio HTTP/1.1" 200 None
>>>
>>> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> Starting new HTTPS connection (1): graph.facebook.com:443
>>>
>>> Mar 20 10:25:46 raspberrypi weewxd[11529]: INFO weewx.restx: MQTT:
>>> Published record 2024-03-20 10:25:46 CET (1710926746)
>>>
>>> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> https://graph.facebook.com:443 "GET
>>> /v19.0/3479353805541966/accounts?access_token=EAATInc3craQBO4TVblpp4l4j6ZB8l6XRLLoWILxwMZAZBGLvfpZARk84dzoFJeI9yuuD5SWu3sv84MeCLe35oXKet3aASFE9ZBwtIZBupstk0RdmQbjZCtvUZCW2F8RZBrfOgB9nTLDjz6MBtwbW6SCxUvZA1lR0z6qDxDri2k4aQA4O5EbcqGLhkUhUekDeTZB7PQZD
>>> HTTP/1.1" 200 505
>>>
>>> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> Starting new HTTPS connection (1): www.santodelgiorno.it:443
>>>
>>> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> https://www.santodelgiorno.it:443 "GET / HTTP/1.1" 200 12219
>>>
>>> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
>>> Starting new HTTPS connection (1):

Re: [weewx-development] Unable to instantiate generator (again)

[Tom Keffer]

Since I responded to Luke, I have a much better understanding of how this
happens. See the section on Pillow
<https://github.com/weewx/weewx/wiki/pip-troubleshooting#pillow> in the
wiki article *pip troubleshooting*.

On Wed, Mar 20, 2024 at 5:08 AM Vince Skahan  wrote:

> Tom - this is an old thread but I thought I'd mention that I just ran into
> this on a clean 5.0.2 pip installation on a couple newly imaged pi running
> the latest 32-bit raspi os dated march-15-2024.   Same error happens on a
> pi4 and pi5 running that os.
>
> 2024-03-19T15:25:16.177588-07:00 pi5 weewxd[3821]: ERROR
> weewx.reportengine: Unable to instantiate generator
> 'weewx.imagegenerator.ImageGenerator'
> 2024-03-19T15:25:16.177710-07:00 pi5 weewxd[3821]: ERROR
> weewx.reportengine:   libopenjp2.so.7: cannot open shared
> object file: No such file or directory
>
> Running "sudo apt install -y libopenjp2-7" as per the thread above fixes
> the problem still with current raspios.  Cool.
>
> To quantify the os setup,  "dpkg --print-architecture" reports armhf on
> the pi5 which reflects what the kernel is really doing.  The 'arch' command
> returns a bogus aarch64 result when running the raspios 32-bit os, which is
> a known though confusing thing.
>
> On Monday, May 29, 2023 at 6:58:44 AM UTC-7 Lucas Heijst wrote:
>
>> Thanks for your answer Tom
>>
>> I skipped the step "pip3 install wheel" because I thought this was not
>> needed anymore
>>
>> sudo apt update
>> sudo apt -y install gcc python3-dev python3-pip
>> (skipped step: pip3 install wheel)
>> pip3 install weewx --user
>>
>> Op maandag 29 mei 2023 om 10:39:40 UTC-3 schreef Tom Keffer:
>>
>>> It's a good question, and I'm not sure I know the full answer.
>>>
>>> Generally, if one installs Pillow, it installs by using a wheel, which
>>> includes the necessary binary files (look in
>>> ~/.local/lib/python3.7/site-packages/Pillow.libs). However, if a wheel is
>>> not available for your platform (and it may not be for buster light), then
>>> pip will compile from source. That requires that the underlying libraries,
>>> including openjpeg, have already been installed.
>>>
>>> That's my best guess.
>>>
>>> -tk
>>>
>>> On Mon, May 29, 2023 at 6:19 AM Lucas Heijst  wrote:
>>>
>>>>
>>>> You guessed right, Tom, installing libopenjp2-7 solved the problem.
>>>>
>>>> Question: why was this module not installed by the weewx install?
>>>> Op maandag 29 mei 2023 om 10:03:49 UTC-3 schreef Tom Keffer:
>>>>
>>>>> I would guess that you're missing the underlying JPEG library
>>>>>
>>>>>
>>>>> *sudo apt install libopenjp2-7*
>>>>>
>>>>> -tk
>>>>>
>>>>>
>>>>> On Mon, May 29, 2023 at 5:56 AM Lucas Heijst 
>>>>> wrote:
>>>>>
>>>>>> Started with a fresh install of debian buster light on my rpi model 2
>>>>>> Followed by a fresh install of weewx via:
>>>>>> pip3 install weewx --user
>>>>>> /root/.local/bin/weectl station create --no-prompt
>>>>>>
>>>>>> I got the following error:
>>>>>> Traceback (most recent call last):
>>>>>>   File
>>>>>> "/root/.local/lib/python3.7/site-packages/weewx/reportengine.py", line 
>>>>>> 179,
>>>>>> in run
>>>>>> obj = weeutil.weeutil.get_object(generator)(
>>>>>>   File "/root/.local/lib/python3.7/site-packages/weeutil/weeutil.py",
>>>>>> line 1352, in get_object
>>>>>> mod = __import__(module)
>>>>>>   File
>>>>>> "/root/.local/lib/python3.7/site-packages/weewx/imagegenerator.py", line
>>>>>> 14, in 
>>>>>> import weeplot.genplot
>>>>>>   File "/root/.local/lib/python3.7/site-packages/weeplot/genplot.py",
>>>>>> line 13, in 
>>>>>> from PIL import Image, ImageDraw, ImageFont
>>>>>>   File "/root/.local/lib/python3.7/site-packages/PIL/Image.py", line
>>>>>> 103, in 
>>>>>> from . import _imaging as core
>>>>>> ImportError: libopenjp2.so.7: cannot open shared object file: No such
>>>>>> file or directory
>>>>>>
>>>>>> pip3 install weewx 

Re: [weewx-development] I wrote to new Search List Extensions (SLE) but the script is repeated 18 times

[Tom Keffer]

I don't know what I'm looking at. Which messages are from your extension?
The HTTP GET?

If so, we would have to see how it is used to offer an opinion. How is it
referenced in your templates? Which templates is it used in?



On Wed, Mar 20, 2024 at 5:08 AM Gianluca Contessa 
wrote:

> I wrote to new Search List Extensions (SLE). Everything works fine, but I
> noticed from the weewx log that the script is repeated 18 times, and not
> just once as it should be. in fact, weewx delays more than a minute to
> generate the files. I don't understand where is the issue.
>
>
> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://www.santodelgiorno.it:443 "GET / HTTP/1.1" 200 12219
>
> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): www.santodelgiorno.it:443
>
> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://www.santodelgiorno.it:443 "GET /san-giovanni-nepomuceno/
> HTTP/1.1" 200 17870
>
> Mar 20 10:25:43 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): my.meteoblue.com:443
>
> Mar 20 10:25:44 raspberrypi weewxd[11529]: INFO weewx.restx: MQTT:
> Published record 2024-03-20 10:25:44 CET (1710926744)
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://my.meteoblue.com:443 "GET
> /images/meteogram?temperature_units=C_units=kmh_units=mm=it=40.6249=15.1035=138=Europe%2FRome=jhMJTOUVRNvs25m4=it_name=Quadrivio_units=kmh=019aad2a754640de4dcc7cef14e5a258
> HTTP/1.1" 200 82323
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'IHDR' 16 13
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'PLTE' 41 768
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'tRNS' 821 256
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'pHYs' 1089 9
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'IDAT' 1110 65536
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: INFO user.richiesta_token:
> Immagine salvata con successo in
> /var/www/html/weewx/previsione/previsione.png
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): www.3bmeteo.com:443
>
> Mar 20 10:25:45 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://www.3bmeteo.com:443 "GET /meteo/quadrivio HTTP/1.1" 200 None
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): graph.facebook.com:443
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: INFO weewx.restx: MQTT:
> Published record 2024-03-20 10:25:46 CET (1710926746)
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://graph.facebook.com:443 "GET
> /v19.0/3479353805541966/accounts?access_token=EAATInc3craQBO4TVblpp4l4j6ZB8l6XRLLoWILxwMZAZBGLvfpZARk84dzoFJeI9yuuD5SWu3sv84MeCLe35oXKet3aASFE9ZBwtIZBupstk0RdmQbjZCtvUZCW2F8RZBrfOgB9nTLDjz6MBtwbW6SCxUvZA1lR0z6qDxDri2k4aQA4O5EbcqGLhkUhUekDeTZB7PQZD
> HTTP/1.1" 200 505
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): www.santodelgiorno.it:443
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://www.santodelgiorno.it:443 "GET / HTTP/1.1" 200 12219
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): www.santodelgiorno.it:443
>
> Mar 20 10:25:46 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://www.santodelgiorno.it:443 "GET /san-giovanni-nepomuceno/
> HTTP/1.1" 200 17870
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): my.meteoblue.com:443
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> https://my.meteoblue.com:443 "GET
> /images/meteogram?temperature_units=C_units=kmh_units=mm=it=40.6249=15.1035=138=Europe%2FRome=jhMJTOUVRNvs25m4=it_name=Quadrivio_units=kmh=019aad2a754640de4dcc7cef14e5a258
> HTTP/1.1" 200 82323
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'IHDR' 16 13
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'PLTE' 41 768
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'tRNS' 821 256
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'pHYs' 1089 9
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG PIL.PngImagePlugin:
> STREAM b'IDAT' 1110 65536
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: INFO user.richiesta_token:
> Immagine salvata con successo in
> /var/www/html/weewx/previsione/previsione.png
>
> Mar 20 10:25:47 raspberrypi weewxd[11529]: DEBUG urllib3.connectionpool:
> Starting new HTTPS connection (1): www.3bmeteo.com:443
>
> Mar 20 10:25:47 

Re: [weewx-development] DST start/stop in WeeWX

[Tom Keffer]

https://www.newyorker.com/cartoons/daily-cartoon/daily-cartoon-monday-march-11th-daylight-savings

On Mon, Mar 11, 2024 at 7:50 AM Tom Keffer  wrote:

> The plot is always time_length long. For example, for a day plot, this
> defaults to 27 hours. The DST transition only affects the labeling.
>
> Take this example from yesterday:
> [image: image.png]
> The total width is 27 realtime hours  --- that doesn't change with DST.
> However, the advent of DST causes the major increments to the right of the
> DST transition to shift a little bit to the left --- if you will, we
> reached them quicker thanks to the "spring forward."
>
> The density of data points across the graph remains the same --- one every
> 5 minutes --- but the labeling shifts around a bit.
>
> As for the tallies: the March NOAA report starts with 1 March 2024 00:00,
> exclusive, and ends with 1 April 2024 00:00, inclusive. Those are *local
> times*. Because of DST, the table actually contains 1 hour less data than
> if DST had not happened.
>
> Hope that helps.
>
> On Mon, Mar 11, 2024 at 6:37 AM Joel Bion  wrote:
>
>> In my country, we changed to DST at 2AM this past Sunday early morning.
>>
>> I’m assuming that in the database, the timestamps are all UTC.
>>
>> It’s the display of the information that I am wondering about, because
>> the graphs show local times. I live in the Pacific Time Zone in North
>> America. Right now, my server displays on my WeeWX home page that I am in
>> UTC -7, which is correct. Until 2am my time Sunday, I was in UTC-8.
>>
>> So, at 4AM Sunday (yes I was awake), my graph for the day showed 4 hours
>> of data in my day. Yet, because of the time change, there were only 3
>> realtime hours in the day. Again, 4 hours of data was displayed in that
>> day. Yet, that day only had 3 hours.
>>
>> My guess is, when I am in “standard time” and ask weewx to graph me
>> historic data it assumes my decade+ of data is all in standard time,
>> ignoring the date and basing off a chosen date-independent “start of
>> epoch?”, or some such?  And then when I am in daylight savings time, do
>> something similar during Summer time? so, if I was in a cloudburst in the
>> last hour of the day before time-change day, during daylight savings time,
>> that data is shown in Sunday’s graph, during standard time it’s shown in
>> Saturday’s?
>>
>> And what about “tallies” in the tables, like “rain each day of the
>> month”? How do they choose what day events the last hour of the day before
>> a time change are bucketed to?
>>
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-development+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/35200E9A-9AB7-4B33-985F-AB80CB35D890%40gmail.com
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDi%3D7uO62tnZ8deaq0VpvnCj9Robmsd2_PfG_UNjVg%2BEQ%40mail.gmail.com.

Re: [weewx-development] DST start/stop in WeeWX

[Tom Keffer]

The plot is always time_length long. For example, for a day plot, this
defaults to 27 hours. The DST transition only affects the labeling.

Take this example from yesterday:
[image: image.png]
The total width is 27 realtime hours  --- that doesn't change with DST.
However, the advent of DST causes the major increments to the right of the
DST transition to shift a little bit to the left --- if you will, we
reached them quicker thanks to the "spring forward."

The density of data points across the graph remains the same --- one every
5 minutes --- but the labeling shifts around a bit.

As for the tallies: the March NOAA report starts with 1 March 2024 00:00,
exclusive, and ends with 1 April 2024 00:00, inclusive. Those are *local
times*. Because of DST, the table actually contains 1 hour less data than
if DST had not happened.

Hope that helps.

On Mon, Mar 11, 2024 at 6:37 AM Joel Bion  wrote:

> In my country, we changed to DST at 2AM this past Sunday early morning.
>
> I’m assuming that in the database, the timestamps are all UTC.
>
> It’s the display of the information that I am wondering about, because the
> graphs show local times. I live in the Pacific Time Zone in North America.
> Right now, my server displays on my WeeWX home page that I am in UTC -7,
> which is correct. Until 2am my time Sunday, I was in UTC-8.
>
> So, at 4AM Sunday (yes I was awake), my graph for the day showed 4 hours
> of data in my day. Yet, because of the time change, there were only 3
> realtime hours in the day. Again, 4 hours of data was displayed in that
> day. Yet, that day only had 3 hours.
>
> My guess is, when I am in “standard time” and ask weewx to graph me
> historic data it assumes my decade+ of data is all in standard time,
> ignoring the date and basing off a chosen date-independent “start of
> epoch?”, or some such?  And then when I am in daylight savings time, do
> something similar during Summer time? so, if I was in a cloudburst in the
> last hour of the day before time-change day, during daylight savings time,
> that data is shown in Sunday’s graph, during standard time it’s shown in
> Saturday’s?
>
> And what about “tallies” in the tables, like “rain each day of the month”?
> How do they choose what day events the last hour of the day before a time
> change are bucketed to?
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/35200E9A-9AB7-4B33-985F-AB80CB35D890%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBPCUkjwceVZ8d-%3DxKEL4a_huFXq_BBkumpGhm1DoEUbg%40mail.gmail.com.

Re: [weewx-development] Re: Ramblings from an XType developer

[Tom Keffer]

Rich,

Going back to your post in this thread, one thing I forgot to mention: You
and I talked about how the XTypes system is not very rigorous on when a
function should or should not raise weewx.CannotCalculate.  HEAD has
tightened this up and now has a few changes that prevent get_aggregate()
and get_series() from raising weewx.CannotCalculate. Instead, they return
None.

However, get_scalar() can still raise CannotCalculate.

The XTypes wiki <https://github.com/weewx/weewx/wiki/xtypes> has been
revised to clarify this

-tk

On Mon, Mar 4, 2024 at 8:53 AM bell...@gmail.com 
wrote:

> Tom,
> Thanks for the example. I did the same thing, just not as elegant. When I
> have time, I will try this out.
> I also did it for ‘not_null’ since it is called when ‘has_data’ is called.
> For most people, probably not necessary, but I had a sensor offline
> recently, so the default get_aggregate of ‘not_null’ had to iterate until a
> time when the sensor was online.
> Again, thanks!
> rich
> On Monday 4 March 2024 at 10:41:39 UTC-5 Tom Keffer wrote:
>
>> I've added an optimized version of get_aggregate(). This version
>> recognizes the monotonic relationship between temperature and vapor
>> pressure, so it can use the daily summaries to find the min, max, mintime,
>> and maxtime of vapor pressure. It would not work for the average value.
>>
>> See
>> https://github.com/weewx/weewx/wiki/XTypes-performance#optimized-version-of-get_aggregate
>>
>> On Sat, Mar 2, 2024 at 6:52 PM John Kline  wrote:
>>
>>> It looks like I specialized get_aggregate() for my weewx-purple
>>> extension in July of 2020.  AQI isn’t in the database, but pm2_5 is, which
>>> makes things efficient.
>>>
>>> I don’t know if get_aggregate was actually called in v4 (since I update
>>> my skin pages on every loop record, the values initially generated in
>>> reports aren’t interesting).
>>>
>>> I do know that reporting got slower in v5.
>>>
>>> And this simple change dramatically speeded things up:
>>> - weewx.xtypes.xtypes.append(AQI())
>>> + weewx.xtypes.xtypes.insert(0, AQI())
>>> I’m glad I finally paid attention to this subject.  Thanks, Tom and
>>> others.
>>>
>>> On Mar 2, 2024, at 5:16 PM, Tom Keffer  wrote:
>>>
>>> 
>>> On Sat, Mar 2, 2024 at 9:38 AM Karen K  wrote:
>>>
>>>>
>>>> First is the ArchiveTable class. It calls the general get_aggregate
>>>> function and thus also your XType. If you want ArchiveTable handle
>>>> get_series aggregations for your XType, you have to insert your XType after
>>>> ArchiveTable. But if you implemented aggregations and do not want
>>>> XTypeTable handle those aggregations, then you need to insert your XTypes
>>>> before XTypeTable. That is, between ArchiveTable and XTypeTable.
>>>>
>>>
>>> I guess I'm not seeing that. If ArchiveTable.get_series() calls
>>> xtypes.get_aggregate(), it will run down the list of xtypes. It won't use
>>> the one in ArchiveTable because it doesn't know about your new type. It
>>> keeps going until it gets to your xtype extension.
>>>
>>>
>>>> If your users want to speed up something and include your XTypes into
>>>> the database, they surely want the DailySummaries class handle
>>>> aggregations. But that does not mean that you want XTypeTable handle the
>>>> aggegations of your XType in the other case. You may have provided your own
>>>> implementation of the aggregations. That means you have to insert your
>>>> XType after DailySummaries but before XTypeTable.
>>>>
>>>
>>> If the XType is in the database, it will be handled like any other type.
>>> There is no longer anything special about it. There is no reason for
>>> XTypeTable to come into play. I also don't know why you'd want your own
>>> implementation of get_aggregate() if the type is in the database.
>>>
>>> So I guess if you want your XType to behave in an intuitive way and you
>>>> have implemented the aggregations, then you will have to insert it after
>>>> ArchiveTable and DailySummaries, but before XTypeTable.
>>>>
>>>
>>> Respectively disagree. New XTypes can be appended or prepended to the
>>> list --- the only time it matters is a straight optimization of
>>> get_aggregate(). Then it must appear at the beginning of the list so it
>>> appears before XTypeTable.get_aggregate().
>>>
>>> See the new wiki article on xtypes perf

Re: [weewx-development] Re: Ramblings from an XType developer

[Tom Keffer]

I've added an optimized version of get_aggregate(). This version recognizes
the monotonic relationship between temperature and vapor pressure, so it
can use the daily summaries to find the min, max, mintime, and maxtime of
vapor pressure. It would not work for the average value.

See
https://github.com/weewx/weewx/wiki/XTypes-performance#optimized-version-of-get_aggregate

On Sat, Mar 2, 2024 at 6:52 PM John Kline  wrote:

> It looks like I specialized get_aggregate() for my weewx-purple extension
> in July of 2020.  AQI isn’t in the database, but pm2_5 is, which makes
> things efficient.
>
> I don’t know if get_aggregate was actually called in v4 (since I update my
> skin pages on every loop record, the values initially generated in reports
> aren’t interesting).
>
> I do know that reporting got slower in v5.
>
> And this simple change dramatically speeded things up:
> - weewx.xtypes.xtypes.append(AQI())
> + weewx.xtypes.xtypes.insert(0, AQI())
> I’m glad I finally paid attention to this subject.  Thanks, Tom and others.
>
> On Mar 2, 2024, at 5:16 PM, Tom Keffer  wrote:
>
> 
> On Sat, Mar 2, 2024 at 9:38 AM Karen K  wrote:
>
>>
>> First is the ArchiveTable class. It calls the general get_aggregate
>> function and thus also your XType. If you want ArchiveTable handle
>> get_series aggregations for your XType, you have to insert your XType after
>> ArchiveTable. But if you implemented aggregations and do not want
>> XTypeTable handle those aggregations, then you need to insert your XTypes
>> before XTypeTable. That is, between ArchiveTable and XTypeTable.
>>
>
> I guess I'm not seeing that. If ArchiveTable.get_series() calls
> xtypes.get_aggregate(), it will run down the list of xtypes. It won't use
> the one in ArchiveTable because it doesn't know about your new type. It
> keeps going until it gets to your xtype extension.
>
>
>> If your users want to speed up something and include your XTypes into the
>> database, they surely want the DailySummaries class handle aggregations.
>> But that does not mean that you want XTypeTable handle the aggegations of
>> your XType in the other case. You may have provided your own implementation
>> of the aggregations. That means you have to insert your XType after
>> DailySummaries but before XTypeTable.
>>
>
> If the XType is in the database, it will be handled like any other type.
> There is no longer anything special about it. There is no reason for
> XTypeTable to come into play. I also don't know why you'd want your own
> implementation of get_aggregate() if the type is in the database.
>
> So I guess if you want your XType to behave in an intuitive way and you
>> have implemented the aggregations, then you will have to insert it after
>> ArchiveTable and DailySummaries, but before XTypeTable.
>>
>
> Respectively disagree. New XTypes can be appended or prepended to the list
> --- the only time it matters is a straight optimization of get_aggregate().
> Then it must appear at the beginning of the list so it appears before
> XTypeTable.get_aggregate().
>
> See the new wiki article on xtypes performance
> <https://github.com/weewx/weewx/wiki/XTypes-performance>. It explains
> some of this.
>
> -tk
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/CAPq0zEA8h_DYLP6%2BrTB4VeEo46%2B9ySXsfasExohbptVuk8peSw%40mail.gmail.com
> <https://groups.google.com/d/msgid/weewx-development/CAPq0zEA8h_DYLP6%2BrTB4VeEo46%2B9ySXsfasExohbptVuk8peSw%40mail.gmail.com?utm_medium=email_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDdDcw_6Sd8T4SkG%3DS89i3W3NnfvLTZVAoqt%3DcKU0MEXw%40mail.gmail.com.

Re: [weewx-development] Re: Ramblings from an XType developer

[Tom Keffer]

On Sat, Mar 2, 2024 at 9:38 AM Karen K  wrote:

>
> First is the ArchiveTable class. It calls the general get_aggregate
> function and thus also your XType. If you want ArchiveTable handle
> get_series aggregations for your XType, you have to insert your XType after
> ArchiveTable. But if you implemented aggregations and do not want
> XTypeTable handle those aggregations, then you need to insert your XTypes
> before XTypeTable. That is, between ArchiveTable and XTypeTable.
>

I guess I'm not seeing that. If ArchiveTable.get_series() calls
xtypes.get_aggregate(), it will run down the list of xtypes. It won't use
the one in ArchiveTable because it doesn't know about your new type. It
keeps going until it gets to your xtype extension.


> If your users want to speed up something and include your XTypes into the
> database, they surely want the DailySummaries class handle aggregations.
> But that does not mean that you want XTypeTable handle the aggegations of
> your XType in the other case. You may have provided your own implementation
> of the aggregations. That means you have to insert your XType after
> DailySummaries but before XTypeTable.
>

If the XType is in the database, it will be handled like any other type.
There is no longer anything special about it. There is no reason for
XTypeTable to come into play. I also don't know why you'd want your own
implementation of get_aggregate() if the type is in the database.

So I guess if you want your XType to behave in an intuitive way and you
> have implemented the aggregations, then you will have to insert it after
> ArchiveTable and DailySummaries, but before XTypeTable.
>

Respectively disagree. New XTypes can be appended or prepended to the list
--- the only time it matters is a straight optimization of get_aggregate().
Then it must appear at the beginning of the list so it appears before
XTypeTable.get_aggregate().

See the new wiki article on xtypes performance
. It explains some
of this.

-tk

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEA8h_DYLP6%2BrTB4VeEo46%2B9ySXsfasExohbptVuk8peSw%40mail.gmail.com.

Re: [weewx-development] Re: Ramblings from an XType developer

[Tom Keffer]

1. In most cases, an xtype offers a new type or a new aggregation, in which
case it doesn't matter if you prepend or append. This is the case of the
examples in the xtype document. However, if an optimization is offered,
that is, a specialized version of get_aggregate() then, yes, it should be
prepended. I will make the wiki change.

2. I am not aware of a case where it's necessary to insert an xtype
between the others. Let me know if I missed something.

3. As I have said repeatedly, V4.x *does less*. It does not "work fine" ---
it doesn't work at all. For example, you can't do an aggregate of a
synthetic type in V4.x.

On Fri, Mar 1, 2024 at 11:49 PM Karen K  wrote:

> Tom Keffer schrieb am Samstag, 2. März 2024 um 02:00:54 UTC+1:
>
> For an aggregate, for your xtype to be used, it must appear *before* 
> XTypeTable
> ...
>
>
> In the Wiki about XTypes <https://github.com/weewx/weewx/wiki/xtypes> all
> the examples *append* the XType to the list. Not a single example does it
> otherwise. And there are a lot of examples. I must confess I did not
> realize that one sentence about prepending the XType there. By now I
> thought you wanted users to always *append* their XTypes. And there is no
> word that it may be necessary to insert the XType between others. May be
> that confused developers.
>
> I wonder why WeeWX 4.X worked fine without get_aggregate() in XTypeTable.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/1c5b42cc-07c8-4eb8-b83d-af6b077a4b64n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/1c5b42cc-07c8-4eb8-b83d-af6b077a4b64n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAWXPATWmFy0iakJDtcAORaiRJWbLqEAG_ersXEH%3DX1aw%40mail.gmail.com.

Re: [weewx-development] Re: Ramblings from an XType developer

[Tom Keffer]

In practice, this is not a problem.

The tag $current.myxtype checks the database first before using the xtypes
system.

For an aggregate, for your xtype to be used, it must appear *before*
 XTypeTable *and* it must provide get_aggregate() *and* it must provide the
requested type of aggregation. So, for a tag such as $day.myxtype.min, if
you prepend your xtype, it will only be used if you provide get_aggregate()
and provide an implementation for aggregate type 'min'.


On Thu, Feb 29, 2024 at 4:56 PM bell...@gmail.com 
wrote:

> Hmmm, another vote for the XTypeTable always being the last XType in the
> list; serving as a ‘backstop’ if there are no more specific implementations.
> I hadn’t thought about the ArchiveTable XType. If person A persists my
> XType (adds it to the db), I need to make sure my XType is after
> ArchiveTable but before XTypeTable.
> I suppose there is a use case for overriding ArchiveTableXType (prepending
> it, but at this time I don’t see it….
> My head hurts! Thanks for the code snippet.
>
> On Thursday 29 February 2024 at 15:18:41 UTC-5 Karen K wrote:
>
>> I thought about observation types that are calculated within an XType,
>> but then saved to the database. After that summaries could be retrieved
>> from the database instead of calculated again. So I guess neither inserting
>> the XType at the beginning nor appending it at the end is the best
>> solution. For the weewx-GTS 
>> extension I now decided to insert it before XTypeTable, but after
>> DailySummaries and ArchiveTable like this:
>>
>> # Register the class
>> archive_seen = False
>> summaries_seen = False
>> for idx,xtype in enumerate(weewx.xtypes.xtypes):
>> if (isinstance(xtype,weewx.xtypes.XTypeTable) or
>> (archive_seen and
>> summaries_seen)):
>> weewx.xtypes.xtypes.insert(idx,self.GTSextension)
>> break
>> if isinstance(xtype,weewx.xtypes.ArchiveTable):
>> archive_seen = True
>> elif isinstance(xtype,weewx.xtypes.DailySummaries):
>> summaries_seen = True
>> else:
>> weewx.xtypes.xtypes.append(self.GTSextension)
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/2ffcddf2-5695-4d24-bb49-c04de6b87467n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECqpOx6c7QyvdosZxpP%3DcrXeUGEh7B0qLjTk3Qw0ZNM%2Bg%40mail.gmail.com.

Re: [weewx-development] Programmatic way to know when end of webpages generation is complete?

[Tom Keffer]

Report generation is done in a separate thread, so there is no event
created when the reports finish.

However, what you can do is create a new report generator
. Right
now, there are 5 report generators: CheetahGenerator, ImageGenerator,
CopyGenerator, FtpGenerator, and RsyncGenerator, generally run in that
order.

Create a new generator that does what you want, following the pattern of an
existing generator (say, FtpGenerator). Then create a report which runs it.
Look at the "FTP" report, which does something similar. Put your "report"
at the end of all the other reports, and it will be run last.

That's a brief description. Let me know if you have any questions.

On Wed, Feb 28, 2024 at 6:17 AM jpb...@gmail.com  wrote:

> Hi -
>
> There are a couple of programmatic tasks I would like to perform the
> moment weewx has finished its every 5-minute 'round' of updating its
> webpages, graphics, etc. Essentially, I am wanting an 'event' to trigger
> when this generation is complete, so then I can execute some tasks at that
> moment.
>
> Does such an event exist?
>
> Thanks!
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/4528e321-2455-4abc-a80f-0c3b142e1d51n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBCcBDKSm1zm%3Du-Kk%2BGifWJFcxq%2BKaUURJfsigH5jNO1A%40mail.gmail.com.

Re: [weewx-development] Re: Custom install path weewx v5

[Tom Keffer]

It's actually very simple:

*weectl station create /home/test*


See the weectl station create documentation
.

On Fri, Feb 23, 2024 at 12:06 PM Copain  wrote:

> Here my results of further install tests.
>
> Install of weewx-venv to /home/test works as follows:
> python3 -m venv /home/test/weewx-venv
> source /home/test/weewx-venv/bin/activate
> python3 -m pip install weewx
>
> The provision of a new station with weectl station create did NOT work, as
> I found no way to add my custom path.
> Depending on the logged on user, weewx-data is either generated in
> /home/USER or /root.
> In a way, this approach does not seem to be consistently.
>
> Did I overlook something to generate the station using a custom path?
> Would it make sense to add a 'path option' to weewx utilities?
>
> Copain schrieb am Freitag, 23. Februar 2024 um 17:30:53 UTC+1:
>
>> Many thanks to all of you for the continuous development of weewx.
>>
>> I am trying to install weewx v5.0.2 to /home/test.
>> This does not seem to work, as the installer by default wants to use
>> /home/USER/...
>>
>> How can I pip install weewx-venv and weewx-data to a custom path,
>> independent of the logged on system user?
>> Where can I change the default path?
>>
>> Thank you.
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/48799331-310b-4304-a2da-4b6a17e1413dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAdHRXtOK0pxghS7-FvE%3DH%2BOCbVckO_7BX7p_hq9%3D%3DRag%40mail.gmail.com.

Re: [weewx-development] Some thought about the case of more than one data source

[Tom Keffer]

This is a good point. Gary is going to give it a rethink.

Thanks for pointing this out, Karen.

On Wed, Feb 21, 2024 at 10:22 PM Karen K  wrote:

> What I understood about the "update" parameter is, that it is intended to
> fill in observation types that are *None* before.
>
> What about updating a reading that - for example - set the maximum value
> in the daily summary, and the updated value is lower than before?
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/4b4c6f1a-c2f8-45b9-9b47-e1031b6662b9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDzBXoCKfQz_97KM8092CWO98g7Njjh2bwK2yuq-H_hMg%40mail.gmail.com.

Re: [weewx-development] Pillow (or PIL) import issue with fresh 5.0.2 install

[Tom Keffer]

The wheel pillow-10.2.0-cp311-cp311-linux_armv7l.whl is not listed on pypi (
list <https://pypi.org/project/pillow/#files>). The processor "armv7" is a
32-bit processor, so I guess you're running a pretty old machine? If so,
I'm not surprised that Pillow does not offer a prebuilt wheel.

So, I'm guessing that it was built on your machine, which would explain why
the truetype library is not included until you installed it with apt.

I've included it in the troubleshooting guide:
https://github.com/weewx/weewx/wiki/pip-troubleshooting#pillow-runtime-error


On Thu, Feb 22, 2024 at 1:45 PM Chris Richmond  wrote:

> So... (from pyvenv):
> (weewx) bcm2709_/home/crichmon> pip3 cache list
> Cache contents:
>
>  - CT3-3.3.3-cp311-cp311-linux_armv7l.whl (195 kB)
>  - args-0.1.0-py3-none-any.whl (3.3 kB)
>  - clint-0.5.1-py3-none-any.whl (34 kB)
>  - ephem-4.1.5-cp311-cp311-linux_armv7l.whl (1.7 MB)
>  - pillow-10.2.0-cp311-cp311-linux_armv7l.whl (1.0 MB)
>
> My xterm history doesn't go back far enough to show the log when the
> pillow build failed.  It sounds like that
> should have included getting libfreetype6 in the process.
>
> As a reminder; this OS version or more likely python version doesn't
> support adding non-OS supported python
> modules (again, error not in xterm history), so anything the OS doesn't
> support natively HAS to be in a pyvenv.
>
> I'm guessing the only way to figure this out is to start from scratch.
> I'm up for it if you think it's required.
>
> And, FWIW, I tried this (outside pyvenv):
>
> bcm2709_/home/crichmon> pip3 cache list
> Cache contents:
>
>  - CT3-3.3.3-cp311-cp311-linux_armv7l.whl (195 kB)
>  - args-0.1.0-py3-none-any.whl (3.3 kB)
>  - clint-0.5.1-py3-none-any.whl (34 kB)
>  - ephem-4.1.5-cp311-cp311-linux_armv7l.whl (1.7 MB)
>  - pillow-10.2.0-cp311-cp311-linux_armv7l.whl (1.0 MB)
> bcm2709_/home/crichmon> sudo tcsh
> [sudo] password for crichmon:
> root@bcm2709> cd
> bcm2709_/root# pip3 cache list
> No locally built wheels cached.
>
> Here's how pip3 fails outside the pyvenv:  (guessing this isn't news)
>
> bcm2709_/home/crichmon> sudo pip3 install Adafruit_Blinka
> error: externally-managed-environment
>
> Ã This environment is externally managed
> â°â> To install Python packages system-wide, try apt install
> python3-xyz, where xyz is the package you are trying to
> install.
>
> If you wish to install a non-Debian-packaged Python package,
> create a virtual environment using python3 -m venv path/to/venv.
> Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
> sure you have python3-full installed.
>
> If you wish to install a non-Debian packaged Python application,
> it may be easiest to use pipx install xyz, which will manage a
> virtual environment for you. Make sure you have pipx installed.
>
> See /usr/share/doc/python3.11/README.venv for more information.
>
> note: If you believe this is a mistake, please contact your Python
> installation or OS distribution provider. You can override this, at the
> risk of breaking your Python installation or OS, by passing
> --break-system-packages.
> hint: See PEP 668 for the detailed specification.
>
> Thx, Chris
>
> On Thursday, February 22, 2024 at 2:24:24 PM UTC-7 Tom Keffer wrote:
>
>> I'm a little mystified why libfreetype6 wasn't included in the Pillow
>> wheel.
>>
>> Could you check which wheel you used? You can see what's in the cache with
>>
>> *pip3 cache list*
>>
>> -tk
>>
>> On Thu, Feb 22, 2024 at 12:56 PM  wrote:
>>
>>> Excellent guess!  So…  I rarely use pip, so those bits took a few tries…
>>>
>>>
>>>
>>> Anyhow;
>>>
>>> bcm2709_/root# apt install python3-pip -y
>>>
>>> python3-pip is already the newest version (23.0.1+dfsg-1).
>>>
>>>
>>>
>>> bcm2709_/root# apt install libfreetype6-dev
>>>
>>> The following NEW packages will be installed:
>>>
>>>   libbrotli-dev libfreetype-dev libfreetype6-dev libpng-dev libpng-tools
>>>
>>>
>>>
>>> bcm2709_/root# exit
>>>
>>> bcm2709_/home/crichmon> source weewx/bin/activate.csh
>>>
>>> (weewx) bcm2709_/home/crichmon> python3 -m pip remove pillow
>>>
>>> ERROR: unknown command "remove"
>>>
>>> (weewx) bcm2709_/home/crichmon> python3 -m pip uninstall pillow
>>>
>>> Found existing installation: pillow 10.2.0
>>>
>>> Uninstalling pillow-10.2.0:
>>>
>>>   Would remove:
>>>
>>>

Re: [weewx-development] Pillow (or PIL) import issue with fresh 5.0.2 install

[Tom Keffer]

I'm a little mystified why libfreetype6 wasn't included in the Pillow wheel.

Could you check which wheel you used? You can see what's in the cache with

*pip3 cache list*

-tk

On Thu, Feb 22, 2024 at 12:56 PM  wrote:

> Excellent guess!  So…  I rarely use pip, so those bits took a few tries…
>
>
>
> Anyhow;
>
> bcm2709_/root# apt install python3-pip -y
>
> python3-pip is already the newest version (23.0.1+dfsg-1).
>
>
>
> bcm2709_/root# apt install libfreetype6-dev
>
> The following NEW packages will be installed:
>
>   libbrotli-dev libfreetype-dev libfreetype6-dev libpng-dev libpng-tools
>
>
>
> bcm2709_/root# exit
>
> bcm2709_/home/crichmon> source weewx/bin/activate.csh
>
> (weewx) bcm2709_/home/crichmon> python3 -m pip remove pillow
>
> ERROR: unknown command "remove"
>
> (weewx) bcm2709_/home/crichmon> python3 -m pip uninstall pillow
>
> Found existing installation: pillow 10.2.0
>
> Uninstalling pillow-10.2.0:
>
>   Would remove:
>
> /usr/local/py.venv/weewx/lib/python3.11/site-packages/PIL/*
>
>
> /usr/local/py.venv/weewx/lib/python3.11/site-packages/pillow-10.2.0.dist-info/*
>
> Proceed (Y/n)?
>
>   Successfully uninstalled pillow-10.2.0
>
> (weewx) bcm2709_/home/crichmon> python3 -m pip install pillow
>
> Collecting pillow
>
>   Using cached pillow-10.2.0-cp311-cp311-linux_armv7l.whl
>
> Installing collected packages: pillow
>
> Successfully installed pillow-10.2.0
>
> (weewx) bcm2709_/home/crichmon> python3 -m pip uninstall pillow
>
> Found existing installation: pillow 10.2.0
>
> Uninstalling pillow-10.2.0:
>
>   Would remove:
>
> /usr/local/py.venv/weewx/lib/python3.11/site-packages/PIL/*
>
>
> /usr/local/py.venv/weewx/lib/python3.11/site-packages/pillow-10.2.0.dist-info/*
>
> Proceed (Y/n)?
>
>   Successfully uninstalled pillow-10.2.0
>
>
>
> (weewx) bcm2709_/home/crichmon> find .cache/pip/wheels/ -type f -ls
>
> (weewx) bcm2709_/home/crichmon> rm -r .cache/pip/wheels/40
>
> (weewx) bcm2709_/home/crichmon> python3 -m pip install pillow
>
>   
>
> Successfully installed pillow-10.2.0
>
>
>
> Restarted weewx and this was the previously broken part of the log:
>
> 2024-02-22T13:40:15.411903-07:00 bcm2709 crichmon[7216]: INFO
> weewx.manager: Added record 2024-02-22 13:40:00 MST (1708634400) to
> database 'weewx.sdb'
>
> 2024-02-22T13:40:15.458556-07:00 bcm2709 crichmon[7216]: INFO
> weewx.manager: Added record 2024-02-22 13:40:00 MST (1708634400) to daily
> summary in 'weewx.sdb'
>
> 2024-02-22T13:40:15.607406-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.reportengine: Running reports for latest time in the database.
>
> 2024-02-22T13:40:15.607850-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.reportengine: Running report 'SeasonsReport'
>
> 2024-02-22T13:40:15.648974-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.reportengine: Found configuration file
> /home/crichmon/weewx-data/skins/Seasons/skin.conf for report 'SeasonsReport'
>
> 2024-02-22T13:40:15.678583-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.reportengine: Running generators for report 'SeasonsReport' in
> directory '/home/crichmon/weewx-data/skins/Seasons'
>
> 2024-02-22T13:40:15.841151-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.cheetahgenerator: Using search list
> ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Current',
> 'weewx.cheetahgenerator.DisplayOptions', 'weewx.cheetahgenerator.Extras',
> 'weewx.cheetahgenerator.Gettext', 'weewx.cheetahgenerator.JSONHelpers',
> 'weewx.cheetahgenerator.PlotInfo', 'weewx.cheetahgenerator.SkinInfo',
> 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Stats',
> 'weewx.cheetahgenerator.UnitInfo']
>
> 2024-02-22T13:40:15.857779-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.manager: Daily summary version is 4.0
>
> 2024-02-22T13:40:19.310712-07:00 bcm2709 crichmon[7216]: INFO
> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.49
> seconds
>
> 2024-02-22T13:40:19.405933-07:00 bcm2709 crichmon[7216]: DEBUG
> weewx.manager: Daily summary version is 4.0
>
> 2024-02-22T13:40:23.482435-07:00 bcm2709 crichmon[7216]: INFO
> weewx.imagegenerator: Generated 60 images for report SeasonsReport in 4.09
> seconds
>
> 2024-02-22T13:40:23.494414-07:00 bcm2709 crichmon[7216]: INFO
> weewx.reportengine: Copied 5 files to /home/crichmon/weewx-data/public_html
>
>
>
> And all the .png files showed up in ~/weewx-data/public_html with the rest
> of the .html’s.
>
>
>
> So, I checked several of my RPi’s, mostly running RPi-OS Bullseye, and it
> was a mix of what I found with:
>
> apt search libfreetype6
>

Re: [weewx-development] Pillow (or PIL) import issue with fresh 5.0.2 install

[Tom Keffer]

For whatever reason, your copy of Pillow was compiled without freetype
(that's what the "ft" stands for in _imagingft).

Try updating pip, then remove pillow, then reinstall.

If that doesn't work, try installing the freetype library first:

*sudo apt install libfreetype6-dev*

Then remove pillow, then reinstall.

If one of these works, let me know and I'll add it to our pip
troubleshooting guide
.

-tk

On Thu, Feb 22, 2024 at 9:54 AM Chris Richmond  wrote:

> Finally trying out weewx 5, and running into python issues.  This is on a
> pi3 running
> Devuan daedalus (Debian bookworm sans systemd).  I'd previously set up my
> own
> stuff to run in a pyvenv, so that basic stuff worked OK.  Following the
> weewx install steps failed because I forgot about activating
> with activate.csh since I run tcsh.
> Next, pillow failed to build because libjpeg headers were missing.  Fixed
> with:
> apt install libjpeg-dev
> Also, the instructions here:
> https://pillow.readthedocs.io/en/latest/installation.html
> warned about having PIL and pillow installed, so I removed PIL from the
> non-pyvenv and started over (bcm2709_/root# apt remove python3-pil)
> Not 100% sure now, but had to install wheel once sourced into the pyvenv:
> (weewx) crichmon@bcm2709> python3 -m pip install wheel
> Proceeded with: (weewx) crichmon@bcm2709> python3 -m pip install weewx
> which ended with:
> Successfully built Pillow
> Installing collected packages: pyserial, ephem, six, pyusb, PyMySQL,
> Pillow, CT3, configobj, weewx
> Successfully installed CT3-3.3.3 Pillow-10.2.0 PyMySQL-1.1.0
> configobj-5.0.8 ephem-4.1.5 pyserial-3.5 pyusb-1.2.1 six-1.16.0 weewx-5.0.2
> Created a Simulator station
> Created the init.d script and weewxd was off to the races until it came
> time to make graphs.  Got this is the logs:
> File
> "/usr/local/py.venv/weewx/lib/python3.11/site-packages/PIL/ImageFont.py",
> line 52, in 
>  from PIL import _imagingft as core
> I'd modified that import line from: "from . import" to "from PIL import"
> with no change.
> From the pyvenv command line, this works:
> bcm2709_/home/crichmon> source weewx/bin/activate.csh
> (weewx) crichmon@bcm2709> python3
> Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from PIL import ImageFont
> >>> ImageFont
>  '/usr/local/py.venv/weewx/lib/python3.11/site-packages/PIL/ImageFont.py'>
> >>>
> Bug with newer python version or something I'm missing?
> syslog attached.
> Thx, Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/bd9fc7d0-9bc6-4d0b-96c2-042674d04632n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBNDy65v%3DNqB_8DsgM%2Bm%2BG-3DdS1EPY_rZAZF8qYJNTxw%40mail.gmail.com.

Re: [weewx-development] Some thought about the case of more than one data source

[Tom Keffer]

The method addRecord() now has an "update
"
parameter, making it possible to update old records. To appear in V5.1.


On Wed, Feb 21, 2024 at 10:26 AM Karen K  wrote:

> I read the roadmap.md file and I saw the item about supporting more than
> one driver at the same time.
>
> So there are some thoughts.
>
> Some people not only have one single weather station but different sources
> of data to process. That could be:
>
>- additional hardware (measurement devices)
>- measurements from other locations in the neighborhood
>- data received from governmental services to use for comparison
>purposes like this
>.
>
> Some of those data sources deliver their readings with some delay but
> include a timestamp. Or they provide an update to a previously sent reading
> after performing quality checks. In all those cases data do not arrive
> within the archive interval they belong to. This involves updating database
> records and summaries.
>
> I have got that case about governmental data and I actually work around
> the limitations by
>
>- not using addRecord() for writing into the database
>- using the binding for reading the database only
>- writing to the database like this
>
> 
>.
>
> The disadvantage of that solution is, that the DaySummaryManager cannot be
> used.
>
> Updating a database record is not only a question of importing. It is also
> required if you want to use data that arrive with some delay and include a
> timestamp of their own.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/12bc601f-220d-4cfd-87c5-e4e956882a1bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDg_BpL6kUE6JRixD-saO_aZV_Aip5rjj17poRevJc01A%40mail.gmail.com.

Re: [weewx-development] HTTP 200 vs 400 from stations api

[Tom Keffer]

I'm happy with what we have because it also works on older versions of
WeeWX. If we start probing the exception, that would only work on new code.

On Mon, Feb 19, 2024 at 11:37 AM bell...@gmail.com 
wrote:

>
> Mumble, grumble… fp seems to added in python 3.12. Or at least that is the
> place it is documented.
> If you wanted to pursue, I could test on other versions.
> But what you ain’t broke….
> On Monday 19 February 2024 at 14:33:43 UTC-5 bell...@gmail.com wrote:
>
>> Oh the arguments/debates we had on what response code to use. It is a
>> grey area…
>> My example code does read the response body on that exception. It appears
>> you can either use the read() method on the exception or the fp attribute.
>> From
>> https://docs.python.org/3/library/urllib.error.html#module-urllib.error
>>
>> *exception *urllib.error.HTTPError(*url*, *code*, *msg*, *hdrs*, *fp*)
>> <https://docs.python.org/3/library/urllib.error.html#urllib.error.HTTPError>
>>
>> Though being an exception (a subclass of URLError
>> <https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError>),
>> an HTTPError
>> <https://docs.python.org/3/library/urllib.error.html#urllib.error.HTTPError> 
>> can
>> also function as a non-exceptional file-like return value (the same thing
>> that urlopen()
>> <https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen>
>>  returns).
>> This is useful when handling exotic HTTP errors, such as requests for
>> authentication.
>>
>> You have something working, its not worth changing.
>>
>> rich
>>
>> Ps. Flask looks pretty cool. Got the stations api running in dev mode in
>> minutes.
>> On Monday 19 February 2024 at 14:21:32 UTC-5 Tom Keffer wrote:
>>
>>> The problem is that urllib.request.urlopen() raises an exception if it
>>> gets an HTTP error code 400. That makes the response body totally
>>> unavailable.
>>>
>>> I studied the HTTP error codes and decided that code 200 means that the
>>> HTTP transmission was successful. It does not necessarily mean that the
>>> application-level transaction was successful, i.e., the resource (in this
>>> case, a station registration) was created. So, a 200 code can be sent back
>>> even if the registration failed.
>>>
>>> By contrast, 400 means that the client malformed the request badly
>>> enough that the server is unable to process it. Presumably, at all.
>>>
>>> So, I switched from 400 to 200 so that we can send some details to the
>>> client on why a registration could not be processed.
>>>
>>> But, the documentation I read was never crystal clear on what to do when
>>> a client passed on invalid data.
>>>
>>> Caveat: I am not an expert on this! Everything above comes from an hour
>>> or two of research on the topic --- the sum total of my expertise!
>>>
>>> Thanks for taking a look at this, Rich!
>>>
>>> -tk
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Feb 19, 2024 at 11:00 AM bell...@gmail.com 
>>> wrote:
>>>
>>>> I was in the process of updating restx.py to log the returned error
>>>> message. If you want the stations api to return a 400, you could do
>>>> something like this.
>>>>
>>>> def test_post():
>>>> url = 'http://localhost:5000/api/v2/stations/'
>>>>
>>>> body = {
>>>> 'station_url': 'example.com',
>>>> }
>>>> json_body = json.dumps(body)
>>>>
>>>> request = urllib.request.Request(url)
>>>> request.add_header('Content-Type', 'application/json')
>>>>
>>>> try:
>>>> response = urllib.request.urlopen(request,
>>>> data=json_body.encode('utf-8'))
>>>> except urllib.error.HTTPError as http_error:
>>>> print(http_error)
>>>> raw_data = http_error.read()
>>>> print(raw_data)
>>>> error_msg = raw_data.decode()
>>>> print(error_msg)
>>>>
>>>> I’m still trying to understand any edge cases/nuances, so the 200 is
>>>> probably a bit ‘safer’.
>>>> rich
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-development" group.
>>>> To unsubscribe from this

Re: [weewx-development] HTTP 200 vs 400 from stations api

[Tom Keffer]

The problem is that urllib.request.urlopen() raises an exception if it gets
an HTTP error code 400. That makes the response body totally unavailable.

I studied the HTTP error codes and decided that code 200 means that the
HTTP transmission was successful. It does not necessarily mean that the
application-level transaction was successful, i.e., the resource (in this
case, a station registration) was created. So, a 200 code can be sent back
even if the registration failed.

By contrast, 400 means that the client malformed the request badly enough
that the server is unable to process it. Presumably, at all.

So, I switched from 400 to 200 so that we can send some details to the
client on why a registration could not be processed.

But, the documentation I read was never crystal clear on what to do when a
client passed on invalid data.

Caveat: I am not an expert on this! Everything above comes from an hour or
two of research on the topic --- the sum total of my expertise!

Thanks for taking a look at this, Rich!

-tk








On Mon, Feb 19, 2024 at 11:00 AM bell...@gmail.com 
wrote:

> I was in the process of updating restx.py to log the returned error
> message. If you want the stations api to return a 400, you could do
> something like this.
>
> def test_post():
> url = 'http://localhost:5000/api/v2/stations/'
>
> body = {
> 'station_url': 'example.com',
> }
> json_body = json.dumps(body)
>
> request = urllib.request.Request(url)
> request.add_header('Content-Type', 'application/json')
>
> try:
> response = urllib.request.urlopen(request,
> data=json_body.encode('utf-8'))
> except urllib.error.HTTPError as http_error:
> print(http_error)
> raw_data = http_error.read()
> print(raw_data)
> error_msg = raw_data.decode()
> print(error_msg)
>
> I’m still trying to understand any edge cases/nuances, so the 200 is
> probably a bit ‘safer’.
> rich
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/45b3404d-7d3a-4e64-82bc-8cf97ff16959n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAMZicJBwaexdT0Y3gzY0DcaQOj9BfxbC%2B%3DK3C148qYdQ%40mail.gmail.com.

Re: [weewx-development] Re: Important release: V5.0.1 !

[Tom Keffer]

As noted in the comment you included, the version stamp in a config file is
the version that created it, not the version that is currently running.

For the latter, use "weectl -v", or look in the log.

-tk

On Mon, Feb 5, 2024 at 10:18 PM Steven Sheeley 
wrote:

> Hi all,
>
>  I'm sorry if I'm missing something here. I just did what the above
> instructions directed for a pip install (Which my system is)
>
>
>  rayvenhaus  wendydev  ~  # Activate the WeeWX virtual environment
>  rayvenhaus  wendydev  ~  source ~/weewx-venv/bin/activate
>  rayvenhaus  wendydev  ~  weewx-venv  # Upgrade the WeeWX code
>  rayvenhaus  wendydev  ~  weewx-venv  python3 -m pip install weewx
> --upgrade
> Looking in indexes: https://pypi.org/simple,
> https://www.piwheels.org/simple
> Requirement already satisfied: weewx in
> ./weewx-venv/lib/python3.9/site-packages (5.0.0)
> Collecting weewx
>   Downloading
> https://www.piwheels.org/simple/weewx/weewx-5.0.1-py3-none-any.whl (1.4
> MB)
>  || 1.4 MB 136 kB/s
> Requirement already satisfied: ephem<5.0,>=4.1 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (4.1.5)
> Requirement already satisfied: Pillow>=5.2 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (10.1.0)
> Requirement already satisfied: pyusb<2.0.0,>=1.0.2 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (1.2.1)
> Requirement already satisfied: PyMySQL<2.0,>=1.0 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (1.1.0)
> Requirement already satisfied: configobj<6.0,>=5.0 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (5.0.8)
> Requirement already satisfied: pyserial<4.0,>=3.4 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (3.5)
> Requirement already satisfied: CT3<4.0,>=3.1 in
> ./weewx-venv/lib/python3.9/site-packages (from weewx) (3.3.3)
> Requirement already satisfied: six in
> ./weewx-venv/lib/python3.9/site-packages (from configobj<6.0,>=5.0->weewx)
> (1.16.0)
> Installing collected packages: weewx
>   Attempting uninstall: weewx
> Found existing installation: weewx 5.0.0
> Uninstalling weewx-5.0.0:
>   Successfully uninstalled weewx-5.0.0
> Successfully installed weewx-5.0.1
>
> Then, I stopped and started Weewx, waited several minutes and verified
> that all was running well. I then checked the webpage, and the system
> reported that I was still running v5.0.0.  Looking at the weewx,conf file,
> I see
>
> 
> # This configuration file was created by ...
> version = 5.0.0rc2
> 
>
> Why didn't the upgrade change my version to 5.0.1?  Did I miss something?
> On Monday, February 5, 2024 at 11:37:59 PM UTC+11 Tom Keffer wrote:
>
>> This is a very important bug fix!
>>
>> If you upgraded from v4.x to v5.0 using a package installer, then tried
>> to install an extension, then tried to uninstall it, you could delete
>> system files!
>>
>> V5.0.1 fixes this, along with some other bugs.
>>
>> See the Upgrade Guide <https://www.weewx.com/docs/5.0/upgrade/> for
>> upgrade instructions.
>>
>> -Tom & Matthew
>>
>>
>> Change log:
>>
>> 5.0.1 02/04/2024¶ <https://www.weewx.com/docs/5.0/changes/#501-02042024>
>>
>> Include backwards compatible reference to weewx.UnknownType.
>>
>> Fix problem with installing extensions into installations that used V4 
>> config files that were installed by a package installer.
>>
>> Fix problem with weectl device when using drivers that were installed using 
>> the extension installer. Fixes issue #918.
>>
>> Fix problem that prevented daily summaries from being rebuilt if they had 
>> been modified by using weectl database drop-columns.
>>
>> Allow the use of the tilde (~) prefix with --config options.
>>
>> Fix problem that prevented debug statements from being logged.
>>
>> Minor corrections to the Norwegian translations. Thanks to user Aslak! PR 
>> #919.
>>
>> Change Chinese language code to zh. Fixes issue #912.
>>
>> Fix bug in redhat/suse scriptlet that incorrectly substituted {weewx} 
>> instead of weewx in the udev rules file.
>>
>> In the redhat/suse installers, use /var/lib/weewx as HOME for user weewx.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/ms

[weewx-development] Important release: V5.0.1 !

[Tom Keffer]

This is a very important bug fix!

If you upgraded from v4.x to v5.0 using a package installer, then tried to
install an extension, then tried to uninstall it, you could delete system
files!

V5.0.1 fixes this, along with some other bugs.

See the Upgrade Guide  for upgrade
instructions.

-Tom & Matthew


Change log:

5.0.1 02/04/2024¶ 

Include backwards compatible reference to weewx.UnknownType.

Fix problem with installing extensions into installations that used V4
config files that were installed by a package installer.

Fix problem with weectl device when using drivers that were installed
using the extension installer. Fixes issue #918.

Fix problem that prevented daily summaries from being rebuilt if they
had been modified by using weectl database drop-columns.

Allow the use of the tilde (~) prefix with --config options.

Fix problem that prevented debug statements from being logged.

Minor corrections to the Norwegian translations. Thanks to user Aslak! PR #919.

Change Chinese language code to zh. Fixes issue #912.

Fix bug in redhat/suse scriptlet that incorrectly substituted {weewx}
instead of weewx in the udev rules file.

In the redhat/suse installers, use /var/lib/weewx as HOME for user weewx.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAeBahmwEtg0T%2B6VBwr7ABnNaAj7Ff2FMMwoO67vtznPg%40mail.gmail.com.

Re: [weewx-development] Loop Packets and Accumulators

[Tom Keffer]

You could run weewxd from the command line and see exactly what LOOP
packets are being emitted, as well as the resultant archive records.

On Sun, Feb 4, 2024 at 10:54 AM Billy Jackson  wrote:

> Thanks Tom,
>
> Only one packet (obs_st type) per 60 seconds.  One rapid_wind every three
> seconds but that doesn't include rain.  At this point I'm fairly certain
> it's the device
>
> Billy
>
> On Sun, Feb 4, 2024 at 10:20 AM Tom Keffer  wrote:
>
>> Putting aside the iffy way that the Tempest calculates rain...
>>
>> The LOOP packet should include rain since the last LOOP packet. If
>> the Tempest is actually emitting rain over the last 60 seconds and there is
>> more than one LOOP packet in that 60 seconds, then the emitted value will
>> be too high.
>>
>> -tk
>>
>> On Sat, Feb 3, 2024 at 7:21 PM Billy Jackson 
>> wrote:
>>
>>> Hi Y'all,
>>>
>>> I wanted to validate an assumption I have about how loop packets work
>>> with Weewx.  The reason being I'm working out a rain issue and it's tough
>>> to sort if it's the HW or if I'm doing something odd
>>>
>>> For reference, this is for my weatherflow tempest driver
>>> https://github.com/livysdad27/tempestWS.
>>>
>>> 1.  The driver opens and uses a WebSocket from the weatherflow Websocket
>>> endpoint.
>>> 2.  It sends rapid_wind messages every three seconds that I use to
>>> generate a partial loop packet.
>>> 3.  It sends a obs_st message every minute that I use to generate most
>>> of the other data including rain.
>>> 4.  Rain, via the API is coming in as "mm recorded since last 60 seconds"
>>> 5.  These are emitted via genLoopPacket()
>>>
>>> The issue is a report that I'm getting from a user who is seeing 4 - 5X
>>> the rain value as other nearby weather stations.  I suspect it's a hardware
>>> issue but want to make sure I have some assumptions correct before going
>>> down that route.
>>>
>>> I THINK what happens is...
>>> 1.  Every 300 seconds (archive_interval) the loop packets are
>>> accumulated into a record to write into the DB for.
>>>   a.  The current five minute readings
>>>   b.  The daily accumulation is updated.
>>>
>>> I'm likely getting something wrong and want to make sure I understand
>>> before pushing on the hardware as an issue.
>>>
>>> Many thanks.
>>>
>>> Billy
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-development+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/4df9c828-57c5-4ac9-ba4e-f870317dcb47n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/4df9c828-57c5-4ac9-ba4e-f870317dcb47n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>
> --
> Billy Jackson
> livysda...@gmail.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEB_QQw2V%3DEwfPaQDOeudNL6UPXYxrFcxwsY3kgi%2B6D7jg%40mail.gmail.com.

Re: [weewx-development] Loop Packets and Accumulators

[Tom Keffer]

Putting aside the iffy way that the Tempest calculates rain...

The LOOP packet should include rain since the last LOOP packet. If
the Tempest is actually emitting rain over the last 60 seconds and there is
more than one LOOP packet in that 60 seconds, then the emitted value will
be too high.

-tk

On Sat, Feb 3, 2024 at 7:21 PM Billy Jackson  wrote:

> Hi Y'all,
>
> I wanted to validate an assumption I have about how loop packets work with
> Weewx.  The reason being I'm working out a rain issue and it's tough to
> sort if it's the HW or if I'm doing something odd
>
> For reference, this is for my weatherflow tempest driver
> https://github.com/livysdad27/tempestWS.
>
> 1.  The driver opens and uses a WebSocket from the weatherflow Websocket
> endpoint.
> 2.  It sends rapid_wind messages every three seconds that I use to
> generate a partial loop packet.
> 3.  It sends a obs_st message every minute that I use to generate most of
> the other data including rain.
> 4.  Rain, via the API is coming in as "mm recorded since last 60 seconds"
> 5.  These are emitted via genLoopPacket()
>
> The issue is a report that I'm getting from a user who is seeing 4 - 5X
> the rain value as other nearby weather stations.  I suspect it's a hardware
> issue but want to make sure I have some assumptions correct before going
> down that route.
>
> I THINK what happens is...
> 1.  Every 300 seconds (archive_interval) the loop packets are accumulated
> into a record to write into the DB for.
>   a.  The current five minute readings
>   b.  The daily accumulation is updated.
>
> I'm likely getting something wrong and want to make sure I understand
> before pushing on the hardware as an issue.
>
> Many thanks.
>
> Billy
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/4df9c828-57c5-4ac9-ba4e-f870317dcb47n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDTDOq11pdjTVBWKtfMWUgZOmdWtSK8Bwkc4YJ%2BZ1vFsw%40mail.gmail.com.

Re: [weewx-development] How to package a driver for V5

[Tom Keffer]

It's the same.

On Wed, Jan 24, 2024 at 6:08 PM Susan Mackay  wrote:

> I have nearly completed fixing an error in the HP1000 driver and I've also
> checked it for compatibility with V5.
>
> I can see the instructions on how to package an extension but nt how to
> package a driver.
>
> Is it the same as for V4 (same directories, structure etc) or is it
> different?
>
> Susan
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/69603331-5993-47bf-9cc9-e13990b4c10fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBiMBUU%2B%3DgrgnYk%3DxEQ%3DN9GwPwkwfvVx737nVQMzcHraA%40mail.gmail.com.

[weewx-development] READ THIS!

[Tom Keffer]

We have discovered a potentially serious bug. The specific situation is as
follows:

   - A V4.x configuration file;
   - Package installer;
   - Install an extension;
   - Uninstall the extension.

Under these circumstances, the extension uninstaller could remove system
files!

If you are using a V4.x configuration file, please do not install any
extensions until we get a fix out.

Apologies.

-tk

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEC0hzPJ6ViEustFzdLW84%2BnTCGLzfb35pB%3DA3B_UfSuOQ%40mail.gmail.com.

Re: [weewx-development] Availability of v5.0.0rc3

[Tom Keffer]

The action weectl station upgrade *does not upgrade the configuration file
unless you tell it to.* See the docs

.

For apt upgrade, make sure you replace the apt repository. In particular,
use "apt" instead of "apt-test".


*echo "deb [arch=all] https://weewx.com/ apt/python3
buster main" | sudo tee /etc/apt/sources.list.d/weewx.list*


On Sun, Jan 14, 2024 at 3:36 PM John Smith 
wrote:

>
>
> On Mon, 15 Jan 2024 at 06:27, Greg  wrote:
>
>> Maybe put see
>> https://weewx.com/docs/5.0/utilities/weectl-station/#upgrade-an-existing-station
>> in the weewx.conf file under this section as an information comment?
>> # Do not modify this. It is used when installing and updating weewx.
>> version = 5.0.0b6
>>
>> If something like that was there I would have read it  to me I would
>> think it's would mirror the version I am running.
>> Keep up the good work.
>>
>
> I was testing rc2, and weectl station upgrade didn't update the weewx.conf
> file to the new version number, and you might want to bump the version
> number because apt install weewx doesn't work, similar reason it thinks
> 5.0.0 is less than rc2
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/CAGTinV4dL0rfd6fLyZ6r4Qm1tE%2Bkt-0bDm9wCmMWf_dOuZP%3DSQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEB%2BU-xkVF_BG%3DTovfDnfgWAcSyhBswETQXWOcZiQWXQpA%40mail.gmail.com.

[weewx-development] V5.0.0 available

[Tom Keffer]

A year in the making, V5 is finally available! Some minor new features, but
most of the work has been to keep up with the changing world of Python ---
lots of things have become obsolete (distutils), while others have just
changed.

A few changes to highlight:

   - WeeWX can now be installed using pip. The old setup.py method is gone.
   - Support for the latest versions of Python.
   - Improved documentation.
   - The utilities have been pulled together under one master utility,
   weectl, making it easier to find the functionality you need.
   - Management of multiple stations is easier and more modular.

Complete change list .

Quickstart install and upgrade instructions
.

Be sure to read the Upgrade Guide .

- Tom, Matt, and Gary.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDyOXhODJ_XaCw%3Dcxm9L3B2t3k70Y3L2fkZQwYSyb71Og%40mail.gmail.com.

Re: [weewx-development] Availability of v5.0.0rc3

[Tom Keffer]

On Sat, Jan 13, 2024 at 6:43 PM Greg  wrote:

> I am using the pip install and my config files in /opt/weex/weewx-data
> I am running v5.0.0.0rc3 but in the weewx.conf it says:
>
> # Do not modify this. It is used when installing and updating weewx.
> version = 5.0.0b6
>
> Does this change when upgrading like the comment says?
>

Yes, but you must explicitly upgrade the config file. It does not happen
automatically. See weectl station upgrade

.

For V5, there were no significant changes to the configuration file, so
there's no need to upgrade.

-tk

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAeda%3D8K7TjNwSsyM-7SahxjG%2Bd_Hg97r%2BEgfj8PJqFDQ%40mail.gmail.com.

[weewx-development] Availability of v5.0.0rc3

[Tom Keffer]

One last look at V5 before it goes out the door!

V5 can be installed using either pip, or the usual package installers.

*Pip*

For pip installs, please delete your old virtual environment, then install
from scratch by following the pip install instructions
. While upgrading should work,
we are particularly interested in the experience of a new install,
including setting up a daemon and udev files. Make sure to follow the new
instructions that use a daemon setup script.


*Debian*

For Debian package installs, modify /etc/apt/sources.list as follows:

*echo "deb [arch=all] https://weewx.com/apt-test/python3
 buster main" | sudo tee
/etc/apt/sources.list.d/weewx.list*

Note the "apt-test". This tells apt to look there for the beta release,
instead of the normal repository. You may want to change it back when
you're done.

Otherwise, follow the Debian install instructions
.


*Red Hat*

For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo

*[weewx]
name=weewx
baseurl=http://weewx.com/yum-test/weewx/el9

enabled=1*

Note the "yum-test". This tells yum to look there for the beta release,
instead of the normal repository. Again, you may want to change things back
when you're done.

Otherwise, follow the Redhat install instructions
.

*What we're looking for*

1. We are interested in your upgrade experience. Did the installers get
your configuration file right? Skins? Is the daemon configured correctly?

2. With this release, we have included udev rules to set the correct
permissions for devices. Did it work? If not, did unplugging then
replugging the device work?

3. How about logging? Is it going to your system logger? Do the labels look
reasonable?

*Thanks!*

-Tom & Matt

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECPG4PmhBH0RKYOULr8RdSrwMZ-sM%3DURzbtqUmEvC1Uug%40mail.gmail.com.

[weewx-development] Re: Private message regarding: [weewx-user] Weewx stopped posting data

[Tom Keffer]

It dumps directly to the database. See the documentation on wee_device
--dump
<https://www.weewx.com/docs/4.10/hardware.htm#vantage_dumping_the_logger_memory>
.

On Wed, Jan 3, 2024 at 9:29 AM emur...@gmail.com 
wrote:

> Running those commands using sudo solved the issue. Once I did that, weewx
> started posting data again. Question - where does wee_device --dump put
> those files? Many thanks for your help!
>
> On Monday, January 1, 2024 at 7:01:53 PM UTC-5 Tom Keffer wrote:
>
> Set debug=1, restart weewx, let it run until it crashes or the first
> reporting cycle, whichever happens first, post the log.
>
> On Mon, Jan 1, 2024 at 1:42 PM emur...@gmail.com 
> wrote:
>
> I have a vantage pro 2 station. My OS is Linux Mint. I had changed the
> batteries in the console. When I did so, I also unplugged the power to it.
> Upon replacing the batteries and rebooting the console, weewx not longer
> posts anything to Index HTML or NOAA. I tried removing power including
> batteries and rebooting again but to no avail.
>
> Next, I purged weewx and then reinstalled. Still nothing. Yet, when I run
> weewxd in terminal it posts the loops there with all my station data.
>
> Here is the log:
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Loading
> station type Vantage (weewx.drivers.vantage)
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: StdConvert
> target unit is 0x1
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.wxservices:
> StdWXCalculate will use data binding wx_binding
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Archive will
> use data binding wx_binding
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Record
> generation will be attempted in 'hardware'
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: The archive
> interval in the configuration file (300) does not match the station
> hardware interval (1800).
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Using archive
> interval of 1800 seconds (specified by hardware)
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.restx:
> StationRegistry: Registration not requested.
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.restx: Wunderground:
> Posting not enabled.
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.restx: PWSweather:
> Posting not enabled.
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.restx: CWOP: Posting
> not enabled.
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.restx: WOW: Posting
> not enabled.
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.restx: AWEKAS:
> Posting not enabled.
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: 'pyephem' not
> detected, extended almanac data is not available
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO __main__: Starting up weewx
> version 4.10.2
> Jan  1 16:33:14 Mel-computer weewx[16417] INFO weewx.engine: Clock error
> is -2.41 seconds (positive is fast)
> Jan  1 16:33:14 Mel-computer weewx[16417] INFO weewx.engine: Using binding
> 'wx_binding' to database 'weewx.sdb'
> Jan  1 16:33:14 Mel-computer weewx[16417] INFO weewx.manager: Starting
> backfill of daily summaries
> Jan  1 16:33:14 Mel-computer weewx[16417] INFO weewx.manager: Daily
> summaries up to date
> Jan  1 16:33:16 Mel-computer weewx[16417] INFO weewx.engine: Starting main
> packet loop.
> Jan  1 16:33:17 Mel-computer weewx[16417] INFO weewx.engine: Main loop
> exiting. Shutting engine down.
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: Caught
> OSError: [Errno 5] Input/output error
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  Traceback (most recent call last):
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  File "/usr/share/weewx/weewxd", line 154, in main
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  engine.run()
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  File "/usr/share/weewx/weewx/engine.py", line 210, in run
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  File "/usr/share/weewx/weewx/engine.py", line 245, in dispatchEvent
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  callback(event)
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  File "/usr/share/weewx/weewx/engine.py", line 799, in new_loop_packet
> Jan  1 16:33:17 Mel-computer weewx[16417] CRITICAL __main__: 
>  print("LOOP:  ",
> Jan  1 16:33:17 Mel-compu

[weewx-development] Re: Private message regarding: [weewx-user] Weewx stopped posting data

[Tom Keffer]

 72, in main
> device.configure(config_dict)
>   File "/usr/share/weewx/weewx/drivers/__init__.py", line 67, in configure
> self.do_options(options, parser, config_dict, not options.noprompt)
>   File "/usr/share/weewx/weewx/drivers/vantage.py", line 2155, in
> do_options
> station = Vantage(**config_dict[DRIVER_NAME])
>   File "/usr/share/weewx/weewx/drivers/vantage.py", line 531, in __init__
> self.port.openPort()
>   File "/usr/share/weewx/weewx/drivers/vantage.py", line 318, in openPort
> self.serial_port = serial.Serial(self.port, self.baudrate,
> timeout=self.timeout)
>   File "/usr/lib/python3/dist-packages/serial/serialutil.py", line 240, in
> __init__
> self.open()
>   File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 268,
> in open
> raise SerialException(msg.errno, "could not open port {}:
> {}".format(self._port, msg))
> serial.serialutil.SerialException: [Errno 13] could not open port
> /dev/ttyUSB0: [Errno 13] Permission denied: '/dev/ttyUSB0'
> Error in sys.excepthook:
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153,
> in apport_excepthook
> with os.fdopen(os.open(pr_filename,
> FileNotFoundError: [Errno 2] No such file or directory:
> '/var/crash/_usr_share_weewx_wee_device.1000.crash'
>
> Original exception was:
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 265,
> in open
> self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY |
> os.O_NONBLOCK)
> PermissionError: [Errno 13] Permission denied: '/dev/ttyUSB0'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/share/weewx/wee_device", line 76, in 
> main()
>   File "/usr/share/weewx/wee_device", line 72, in main
> device.configure(config_dict)
>   File "/usr/share/weewx/weewx/drivers/__init__.py", line 67, in configure
> self.do_options(options, parser, config_dict, not options.noprompt)
>   File "/usr/share/weewx/weewx/drivers/vantage.py", line 2155, in
> do_options
> station = Vantage(**config_dict[DRIVER_NAME])
>   File "/usr/share/weewx/weewx/drivers/vantage.py", line 531, in __init__
> self.port.openPort()
>   File "/usr/share/weewx/weewx/drivers/vantage.py", line 318, in openPort
> self.serial_port = serial.Serial(self.port, self.baudrate,
> timeout=self.timeout)
>   File "/usr/lib/python3/dist-packages/serial/serialutil.py", line 240, in
> __init__
> self.open()
>   File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 268,
> in open
> raise SerialException(msg.errno, "could not open port {}:
> {}".format(self._port, msg))
> serial.serialutil.SerialException: [Errno 13] could not open port
> /dev/ttyUSB0: [Errno 13] Permission denied: '/dev/ttyUSB0'
>
> I restarted weewx and still nothing.
>
> On Monday, January 1, 2024 at 7:01:53 PM UTC-5 Tom Keffer wrote:
>
> Set debug=1, restart weewx, let it run until it crashes or the first
> reporting cycle, whichever happens first, post the log.
>
> On Mon, Jan 1, 2024 at 1:42 PM emur...@gmail.com 
> wrote:
>
> I have a vantage pro 2 station. My OS is Linux Mint. I had changed the
> batteries in the console. When I did so, I also unplugged the power to it.
> Upon replacing the batteries and rebooting the console, weewx not longer
> posts anything to Index HTML or NOAA. I tried removing power including
> batteries and rebooting again but to no avail.
>
> Next, I purged weewx and then reinstalled. Still nothing. Yet, when I run
> weewxd in terminal it posts the loops there with all my station data.
>
> Here is the log:
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Loading
> station type Vantage (weewx.drivers.vantage)
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: StdConvert
> target unit is 0x1
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.wxservices:
> StdWXCalculate will use data binding wx_binding
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Archive will
> use data binding wx_binding
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Record
> generation will be attempted in 'hardware'
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: The archive
> interval in the configuration file (300) does not match the station
> hardware interval (1800).
> Jan  1 16:33:13 Mel-computer weewx[16417] INFO weewx.engine: Using archive
> interval of 1800 seconds (speci

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

On Sat, Dec 30, 2023 at 12:51 AM 'Cameron D' via weewx-development <
weewx-development@googlegroups.com> wrote:

> Another difference between versions 4 and 5 reports that has had me
> baffled for far too long.
>
> For my V4 Seasons Skin I had deleted the sensor fields that I did not
> have, which led to empty fields in the skin file, such as:
>
> sensor_connections =
> sensor_batteries =
> sensor_voltages =
> telemetry_plot_groups =
>

By setting these to nothing, you're actually setting them to an empty
string. SQL is interpreting this as an observation type with an empty
string as a name.

If you really need to eliminate these options, add a comma:

sensor_connections = ,
etc.


This will turn the option into an empty list, instead of an empty string.

Or, just comment the lines out.

Or, just leave them alone.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDussOdDdfnZLCppcjqvue%2BMDpej4LHFSOkCPk9SxR%2BJQ%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

Ryan, I just tried this on my database: dropped ET, added it back on, did
calc-missing, and the values were set to null, not zero. Did you do
something different? Are your radiation values null? Or zero?

On Fri, Dec 29, 2023 at 5:19 PM Tom Keffer  wrote:

> Frankly, I'm surprised (and more than a little relieved!) that setting a
> small tranche value worked.
>
> Very useful. Thanks, Ryan, for all your work on this!
>
> I'm satisfied that the longer report times we've been seeing are due to
> graphs of derived data. ET, in particular, can be very database and compute
> intensive.
>
> Yes, you need radiation to calculate ET. You should set ET to null (not
> zero) if it can't be calculated. Calc-missing should have done this.
>
> -tk
>
> On Fri, Dec 29, 2023 at 4:44 PM  wrote:
>
>> Hey Tom,
>>
>>
>>
>> That worked (on my MBP and Raspi, though the latter was substantially
>> slower (MBP total processing time was 475.95 seconds, raspi was a comical
>> 7999.35))! I was reading on sqlite3 locking processes for writing, and it
>> seemed oddly complex, but I’m sure there are reasons for it.
>>
>>
>>
>> Doing a quick comparison of report generation
>>
>> MBP (M2 Pro):
>>
>> Default Seasons, DB missing ET column: initial 14.39s, update run 9.29s.
>>
>> Default Seasons, DB with ET, calc-missing run: initial 5.54s, update run
>> 0.44s.
>>
>>
>>
>> Raspi (Pi 4, 4GB):
>>
>> Default Seasons, DB missing ET column: initial 2m26s, update run 1m23s.
>>
>> Default Seasons, DB with ET, calc-missing run: initial 1m6s, update run
>> 3.9s.
>>
>>
>>
>> Started back up weewx after all the work and testing, after it grabbed
>> the backlog of records (about 2.5 hours worth), the results speak for
>> themselves:
>>
>>
>>
>> Dec 29 16:20:22 raspi-server-misc weewxd[23387]: INFO
>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.87
>> seconds
>>
>> Dec 29 16:20:23 raspi-server-misc weewxd[23387]: INFO
>> weewx.imagegenerator: Generated 13 images for report SeasonsReport in 0.73
>> seconds
>>
>>
>>
>> Where previously
>>
>>
>>
>> Dec 29 12:45:31 raspi-server-misc weewxd[13706]: INFO
>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 10.84
>> seconds
>>
>> Dec 29 12:46:43 raspi-server-misc weewxd[13706]: INFO
>> weewx.imagegenerator: Generated 12 images for report SeasonsReport in 72.09
>> seconds
>>
>>
>>
>> This is basically on par with 4.x numbers.
>>
>>
>>
>> (version 4.10.2 run)
>>
>> Dec 24 00:05:23 raspi-server-misc weewx[10332] INFO
>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.78
>> seconds
>>
>> Dec 24 00:05:24 raspi-server-misc weewx[10332] INFO weewx.imagegenerator:
>> Generated 12 images for report SeasonsReport in 0.68 seconds
>>
>>
>>
>> The interesting piece, and this is starting to give me déjà vu, is ET is
>> still 0.0. I seem to recall I had this issue when I first started with
>> weewx and the goal was to prevent generating the graph that was always 0…
>>
>>
>>
>> select dateTime, datetime(dateTime, 'unixepoch','localtime'), ET from
>> archive order by dateTime desc limit 20;
>>
>> 1703895600|2023-12-29 16:20:00|0.0
>>
>> 1703895300|2023-12-29 16:15:00|0.0
>>
>> 1703895000|2023-12-29 16:10:00|0.0
>>
>> 1703894700|2023-12-29 16:05:00|0.0
>>
>> 1703894400|2023-12-29 16:00:00|0.0
>>
>> 1703894100|2023-12-29 15:55:00|0.0
>>
>> 1703893800|2023-12-29 15:50:00|0.0
>>
>> 1703893500|2023-12-29 15:45:00|0.0
>>
>> 1703893200|2023-12-29 15:40:00|0.0
>>
>> 1703892900|2023-12-29 15:35:00|0.0
>>
>> 1703892600|2023-12-29 15:30:00|0.0
>>
>> 1703892300|2023-12-29 15:25:00|0.0
>>
>> 1703892000|2023-12-29 15:20:00|0.0
>>
>> 1703891700|2023-12-29 15:15:00|0.0
>>
>> 1703891400|2023-12-29 15:10:00|0.0
>>
>> 1703891100|2023-12-29 15:05:00|0.0
>>
>> 1703890800|2023-12-29 15:00:00|0.0
>>
>> 1703890500|2023-12-29 14:55:00|0.0
>>
>> 1703890200|2023-12-29 14:50:00|0.0
>>
>> 1703889900|2023-12-29 14:45:00|0.0
>>
>>
>>
>> Likely because the Vantage Vue doesn’t provide ET, or Radiation (which I
>> believe is required to calculate ET).
>>
>>
>>
>> It appears calc-missing calculated basically 0 ET for old values (likely
>> wh

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

93443e-05
>
> 1596512160|2020-08-03 20:36:00|1.40241618388119e-05
>
> 1596512040|2020-08-03 20:34:00|1.40227686497999e-05
>
> 1596511920|2020-08-03 20:32:00|1.40698529257927e-05
>
> 1596511800|2020-08-03 20:30:00|1.40682932226393e-05
>
> 1596511680|2020-08-03 20:28:00|1.41120992729917e-05
>
> 1596511560|2020-08-03 20:26:00|1.41120992729917e-05
>
> 1596511440|2020-08-03 20:24:00|1.41092006622079e-05
>
> 1596511320|2020-08-03 20:22:00|1.41092006622079e-05
>
> 1596511200|2020-08-03 20:20:00|1.41077338780795e-05
>
>
>
> Everything newer than that is 0.0.
>
>
>
> Looking at the data, it seems weathercat must have been “calculating”
> maxSolarRad (based on one of the entries I’m seeing) and that’s leading to
> the errant ET calculations. There’s also some stupid levels of precision on
> some of the values (out to like 8 decimal places).
>
>
>
> Since I’ve never had a station that will generate ET, or solar rad for
> that matter, should I just blank those columns with something like
>
>
>
> Update archive set maxSolarRad=0.0, ET=0.0 where maxSolarRad<>0;
>
>
>
> ?
>
>
>
> Then maybe regenerate the daily summaries?
>
>
>
> Thanks Tom. I (think?) we’ve highlighted both some possibly unexpected
> behavior from weewx when a column is missing (same issue Cameron is/was
> seeing), and how my data is not as tidy as I would like…
>
>
>
> -Ryan Stasel
>
>
>
> p.s. Here’s what one of the bad records looks like
>
> dateTime
>
> 1596511200
>
> usUnits
>
> 1
>
> interval
>
> 2
>
> altimeter
>
> 30.05136968
>
> appTemp
>
> 65.54286608
>
> appTemp1
>
> barometer
>
> 30.03188507
>
> batteryStatus1
>
> batteryStatus2
>
> batteryStatus3
>
> batteryStatus4
>
> batteryStatus5
>
> batteryStatus6
>
> batteryStatus7
>
> batteryStatus8
>
> cloudbase
>
> 1429.432901
>
> co
>
> co2
>
> consBatteryVoltage
>
> dewpoint
>
> 58.55
>
> dewpoint1
>
> extraHumid1
>
> extraHumid2
>
> extraHumid3
>
> extraHumid4
>
> extraHumid5
>
> extraHumid6
>
> extraHumid7
>
> extraHumid8
>
> extraTemp1
>
> extraTemp2
>
> extraTemp3
>
> extraTemp4
>
> extraTemp5
>
> extraTemp6
>
> extraTemp7
>
> extraTemp8
>
> forecast
>
> hail
>
> hailBatteryStatus
>
> hailRate
>
> heatindex
>
> 62.798
>
> heatindex1
>
> heatingTemp
>
> heatingVoltage
>
> humidex
>
> 69.66087234
>
> humidex1
>
> inDewpoint
>
> 50.13910232
>
> inHumidity
>
> 41
>
> inTemp
>
> 75.506
>
> leafTemp1
>
> leafTemp2
>
> leafWet1
>
> leafWet2
>
> lightning_distance
>
> lightning_disturber_count
>
> lightning_energy
>
> lightning_noise_count
>
> lightning_strike_count
>
> luminosity
>
> maxSolarRad
>
> 0.29728566
>
> nh3
>
> no2
>
> noise
>
> o3
>
> outHumidity
>
> 86
>
> outTemp
>
> 62.798
>
> pb
>
> pm10_0
>
> pm1_0
>
> pm2_5
>
> pressure
>
> 29.55013196
>
> radiation
>
> 0
>
> rain
>
> 0
>
> rainBatteryStatus
>
> rainRate
>
> 0
>
> referenceVoltage
>
> rxCheckPercent
>
> signal1
>
> signal2
>
> signal3
>
> signal4
>
> signal5
>
> signal6
>
> signal7
>
> signal8
>
> snow
>
> snowBatteryStatus
>
> snowDepth
>
> snowMoisture
>
> snowRate
>
> so2
>
> soilMoist1
>
> soilMoist2
>
> soilMoist3
>
> soilMoist4
>
> soilTemp1
>
> soilTemp2
>
> soilTemp3
>
> soilTemp4
>
> supplyVoltage
>
> txBatteryStatus
>
> UV
>
> 0
>
> uvBatteryStatus
>
> windBatteryStatus
>
> windchill
>
> 62.798
>
> windDir
>
> windGust
>
> 0
>
> windGustDir
>
> windrun
>
> 0
>
> windSpeed
>
> 0
>
> ET
>
> 1.41077E-05
>
>
>
> *From:* Tom Keffer tkef...@gmail.com
> *Sent:* Friday, December 29, 2023 12:56 PM
> *To:* rcsta...@gmail.com
> *Cc:* Vince Skahan vinceska...@gmail.com; weewx-development
> weewx-development@googlegroups.com
> *Subject:* Re: [weewx-development] Re: V5.0 release candidate available
>
>
>
> I tried this on my own largish database by dropping ET, adding it back in,
> then using calc-missing. No problems.
>
>
>
> It's not quite as big (380MB; 1.7M rows) as yours, but has many more rows
> than your "hang up" rows.
>
>
>
> Long shot thing to try: using smaller 

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

I tried this on my own largish database by dropping ET, adding it back in,
then using calc-missing. No problems.

It's not quite as big (380MB; 1.7M rows) as yours, but has many more rows
than your "hang up" rows.

Long shot thing to try: using smaller "tranches." For example,

*weectl database calc-missing --tranche=1*


This would use a one day tranche, instead of the default of 10 days. This
reduces memory requirements and keeps the transaction sizes smaller.

But, I think there is something more fundamental going on. The calc-missing
action uses two connections to the database --- they may be interacting in
subtle ways.

-tk



On Fri, Dec 29, 2023 at 9:48 AM  wrote:

> Hey Tom,
>
>
>
> Got new weewx install on my MBP (just to speed things up), copied over
> weewx.conf (making adjustments for venv install) and my db and can say
> calc-missing hangs at a different spot on here. This time hanging at record
> 49000.
>
>
>
> Oh a whim, I added –dry-run, and it processed all records without error.
> On M2 Pro, that took 181.99 seconds for nearly 15 years of data (2.55M
> rows).
>
>
>
> Are we hitting some DB write lock race/contention? (I say, as I re-read
> what I previously had copied from errors and see “sqlite3.OperationalError:
> database is locked”. Clearly reading IS fundamental. =P
>
>
>
> And to be clear, on Raspi I had stopped weewx service, and on Macbook Pro,
> I never started weewxd to begin with. So this would seemingly be entirely
> weectl locking the db.
>
>
>
> Thanks!
>
>
>
> -Ryan Stasel
>
>
>
> *From:* rcsta...@gmail.com 
> *Sent:* Friday, December 29, 2023 8:33 AM
> *To:* 'Tom Keffer' 
> *Cc:* 'Vince Skahan' ; 'weewx-development' <
> weewx-development@googlegroups.com>
> *Subject:* RE: [weewx-development] Re: V5.0 release candidate available
>
>
>
> Hey Tom,
>
>
>
> Thanks. Below is what I tried… sadly I hit a few roadblocks.
>
>
>
> I stopped weewx on the machine, and checked my weewx.conf, and changed the
> ET value from “hardware” to “prefer_hardware”. I believe this was due to
> previous issue, but I cannot find the thread… =(
>
> Saved that change.
>
>
>
> Backed up the db prior to the add, then ran
>
> Weectl database add-column ET
>
> Confirmed to add as REAL
>
> Weectl database calc-missing
>
> Confirmed. Process hung at/around record 68000. No CPU usage. Guessing
> there’s data missing there? No output in logs indicating the issue.
>
> Processing record: 68000; Last record: 2010-02-17 15:58:00 PST (1266451080)
>
> Exited with ctrl-C, got
>
> Traceback (most recent call last):
>
>   File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn
>
> return fn(*args, **kwargs)
>
>   File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute
>
> return sqlite3.Cursor.execute(self, *args, **kwargs)
>
> sqlite3.OperationalError: database is locked
>
>
>
> During handling of the above exception, another exception occurred:
>
>
>
> Traceback (most recent call last):
>
>   File "/usr/share/weewx/weectl.py", line 74, in 
>
> main()
>
>   File "/usr/share/weewx/weectl.py", line 66, in main
>
> namespace.func(namespace)
>
>   File "/usr/share/weewx/weectllib/__init__.py", line 92, in dispatch
>
> namespace.action_func(config_dict, namespace)
>
>   File "/usr/share/weewx/weectllib/database_cmd.py", line 395, in
> calc_missing
>
> no_confirm=namespace.yes)
>
>   File "/usr/share/weewx/weectllib/database_actions.py", line 480, in
> calc_missing
>
> calc_missing_obj.run()
>
>   File "/usr/share/weewx/weecfg/database.py", line 433, in run
>
> wxcalculate.do_calculations(record)
>
>   File "/usr/share/weewx/weewx/wxservices.py", line 136, in do_calculations
>
> val = weewx.xtypes.get_scalar(obs_type, data_dict, self.db_manager)
>
>   File "/usr/share/weewx/weewx/xtypes.py", line 86, in get_scalar
>
> return xtype.get_scalar(obs_type, record, db_manager, **option_dict)
>
>   File "/usr/share/weewx/weewx/wxxtypes.py", line 305, in get_scalar
>
> % db_manager.table_name, (start_ts, end_ts))
>
>   File "/usr/share/weewx/weewx/manager.py", line 579, in getSql
>
> _cursor.execute(sql, sqlargs)
>
>   File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn
>
> return fn(*args, **kwargs)
>
>
>
> Started process again, this time just targeting last couple years of data.
>
>
>
> Weectl database calc-missing –from=2020-01-01
>
>
>

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

; there, and ). Maybe this explains the behavior? Is my best bet a
>>>> "weectl database reconfigure" to bring things back to default, or just
>>>> re-add via "weectl database add-column ET"?
>>>>
>>>> Would love some help!
>>>>
>>>> Thanks!
>>>> -Ryan Stasel
>>>>
>>>> Here's what I get from listing columns in archive:
>>>> pragma table_info(archive);
>>>> 0|dateTime|INTEGER|1||1
>>>> 1|usUnits|INTEGER|1||0
>>>> 2|interval|INTEGER|1||0
>>>> 3|altimeter|REAL|0||0
>>>> 4|appTemp|REAL|0||0
>>>> 5|appTemp1|REAL|0||0
>>>> 6|barometer|REAL|0||0
>>>> 7|batteryStatus1|REAL|0||0
>>>> 8|batteryStatus2|REAL|0||0
>>>> 9|batteryStatus3|REAL|0||0
>>>> 10|batteryStatus4|REAL|0||0
>>>> 11|batteryStatus5|REAL|0||0
>>>> 12|batteryStatus6|REAL|0||0
>>>> 13|batteryStatus7|REAL|0||0
>>>> 14|batteryStatus8|REAL|0||0
>>>> 15|cloudbase|REAL|0||0
>>>> 16|co|REAL|0||0
>>>> 17|co2|REAL|0||0
>>>> 18|consBatteryVoltage|REAL|0||0
>>>> 19|dewpoint|REAL|0||0
>>>> 20|dewpoint1|REAL|0||0
>>>> 21|extraHumid1|REAL|0||0
>>>> 22|extraHumid2|REAL|0||0
>>>> 23|extraHumid3|REAL|0||0
>>>> 24|extraHumid4|REAL|0||0
>>>> 25|extraHumid5|REAL|0||0
>>>> 26|extraHumid6|REAL|0||0
>>>> 27|extraHumid7|REAL|0||0
>>>> 28|extraHumid8|REAL|0||0
>>>> 29|extraTemp1|REAL|0||0
>>>> 30|extraTemp2|REAL|0||0
>>>> 31|extraTemp3|REAL|0||0
>>>> 32|extraTemp4|REAL|0||0
>>>> 33|extraTemp5|REAL|0||0
>>>> 34|extraTemp6|REAL|0||0
>>>> 35|extraTemp7|REAL|0||0
>>>> 36|extraTemp8|REAL|0||0
>>>> 37|forecast|REAL|0||0
>>>> 38|hail|REAL|0||0
>>>> 39|hailBatteryStatus|REAL|0||0
>>>> 40|hailRate|REAL|0||0
>>>> 41|heatindex|REAL|0||0
>>>> 42|heatindex1|REAL|0||0
>>>> 43|heatingTemp|REAL|0||0
>>>> 44|heatingVoltage|REAL|0||0
>>>> 45|humidex|REAL|0||0
>>>> 46|humidex1|REAL|0||0
>>>> 47|inDewpoint|REAL|0||0
>>>> 48|inHumidity|REAL|0||0
>>>> 49|inTemp|REAL|0||0
>>>> 50|leafTemp1|REAL|0||0
>>>> 51|leafTemp2|REAL|0||0
>>>> 52|leafWet1|REAL|0||0
>>>> 53|leafWet2|REAL|0||0
>>>> 54|lightning_distance|REAL|0||0
>>>> 55|lightning_disturber_count|REAL|0||0
>>>> 56|lightning_energy|REAL|0||0
>>>> 57|lightning_noise_count|REAL|0||0
>>>> 58|lightning_strike_count|REAL|0||0
>>>> 59|luminosity|REAL|0||0
>>>> 60|maxSolarRad|REAL|0||0
>>>> 61|nh3|REAL|0||0
>>>> 62|no2|REAL|0||0
>>>> 63|noise|REAL|0||0
>>>> 64|o3|REAL|0||0
>>>> 65|outHumidity|REAL|0||0
>>>> 66|outTemp|REAL|0||0
>>>> 67|pb|REAL|0||0
>>>> 68|pm10_0|REAL|0||0
>>>> 69|pm1_0|REAL|0||0
>>>> 70|pm2_5|REAL|0||0
>>>> 71|pressure|REAL|0||0
>>>> 72|radiation|REAL|0||0
>>>> 73|rain|REAL|0||0
>>>> 74|rainBatteryStatus|REAL|0||0
>>>> 75|rainRate|REAL|0||0
>>>> 76|referenceVoltage|REAL|0||0
>>>> 77|rxCheckPercent|REAL|0||0
>>>> 78|signal1|REAL|0||0
>>>> 79|signal2|REAL|0||0
>>>> 80|signal3|REAL|0||0
>>>> 81|signal4|REAL|0||0
>>>> 82|signal5|REAL|0||0
>>>> 83|signal6|REAL|0||0
>>>> 84|signal7|REAL|0||0
>>>> 85|signal8|REAL|0||0
>>>> 86|snow|REAL|0||0
>>>> 87|snowBatteryStatus|REAL|0||0
>>>> 88|snowDepth|REAL|0||0
>>>> 89|snowMoisture|REAL|0||0
>>>> 90|snowRate|REAL|0||0
>>>> 91|so2|REAL|0||0
>>>> 92|soilMoist1|REAL|0||0
>>>> 93|soilMoist2|REAL|0||0
>>>> 94|soilMoist3|REAL|0||0
>>>> 95|soilMoist4|REAL|0||0
>>>> 96|soilTemp1|REAL|0||0
>>>> 97|soilTemp2|REAL|0||0
>>>> 98|soilTemp3|REAL|0||0
>>>> 99|soilTemp4|REAL|0||0
>>>> 100|supplyVoltage|REAL|0||0
>>>> 101|txBatteryStatus|REAL|0||0
>>>> 102|UV|REAL|0||0
>>>> 103|uvBatteryStatus|REAL|0||0
>>>> 104|windBatteryStatus|REAL|0||0
>>>> 105|windchill|REAL|0||0
>>>> 106|windDir|REAL|0||0
>>>> 107|windGust|REAL|0||0
>>>> 108|windGustDir|REAL|0||0
>>>> 109|windrun|REAL|0||0
>&g

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

ryVoltage|REAL|0||0
>> 19|dewpoint|REAL|0||0
>> 20|dewpoint1|REAL|0||0
>> 21|extraHumid1|REAL|0||0
>> 22|extraHumid2|REAL|0||0
>> 23|extraHumid3|REAL|0||0
>> 24|extraHumid4|REAL|0||0
>> 25|extraHumid5|REAL|0||0
>> 26|extraHumid6|REAL|0||0
>> 27|extraHumid7|REAL|0||0
>> 28|extraHumid8|REAL|0||0
>> 29|extraTemp1|REAL|0||0
>> 30|extraTemp2|REAL|0||0
>> 31|extraTemp3|REAL|0||0
>> 32|extraTemp4|REAL|0||0
>> 33|extraTemp5|REAL|0||0
>> 34|extraTemp6|REAL|0||0
>> 35|extraTemp7|REAL|0||0
>> 36|extraTemp8|REAL|0||0
>> 37|forecast|REAL|0||0
>> 38|hail|REAL|0||0
>> 39|hailBatteryStatus|REAL|0||0
>> 40|hailRate|REAL|0||0
>> 41|heatindex|REAL|0||0
>> 42|heatindex1|REAL|0||0
>> 43|heatingTemp|REAL|0||0
>> 44|heatingVoltage|REAL|0||0
>> 45|humidex|REAL|0||0
>> 46|humidex1|REAL|0||0
>> 47|inDewpoint|REAL|0||0
>> 48|inHumidity|REAL|0||0
>> 49|inTemp|REAL|0||0
>> 50|leafTemp1|REAL|0||0
>> 51|leafTemp2|REAL|0||0
>> 52|leafWet1|REAL|0||0
>> 53|leafWet2|REAL|0||0
>> 54|lightning_distance|REAL|0||0
>> 55|lightning_disturber_count|REAL|0||0
>> 56|lightning_energy|REAL|0||0
>> 57|lightning_noise_count|REAL|0||0
>> 58|lightning_strike_count|REAL|0||0
>> 59|luminosity|REAL|0||0
>> 60|maxSolarRad|REAL|0||0
>> 61|nh3|REAL|0||0
>> 62|no2|REAL|0||0
>> 63|noise|REAL|0||0
>> 64|o3|REAL|0||0
>> 65|outHumidity|REAL|0||0
>> 66|outTemp|REAL|0||0
>> 67|pb|REAL|0||0
>> 68|pm10_0|REAL|0||0
>> 69|pm1_0|REAL|0||0
>> 70|pm2_5|REAL|0||0
>> 71|pressure|REAL|0||0
>> 72|radiation|REAL|0||0
>> 73|rain|REAL|0||0
>> 74|rainBatteryStatus|REAL|0||0
>> 75|rainRate|REAL|0||0
>> 76|referenceVoltage|REAL|0||0
>> 77|rxCheckPercent|REAL|0||0
>> 78|signal1|REAL|0||0
>> 79|signal2|REAL|0||0
>> 80|signal3|REAL|0||0
>> 81|signal4|REAL|0||0
>> 82|signal5|REAL|0||0
>> 83|signal6|REAL|0||0
>> 84|signal7|REAL|0||0
>> 85|signal8|REAL|0||0
>> 86|snow|REAL|0||0
>> 87|snowBatteryStatus|REAL|0||0
>> 88|snowDepth|REAL|0||0
>> 89|snowMoisture|REAL|0||0
>> 90|snowRate|REAL|0||0
>> 91|so2|REAL|0||0
>> 92|soilMoist1|REAL|0||0
>> 93|soilMoist2|REAL|0||0
>> 94|soilMoist3|REAL|0||0
>> 95|soilMoist4|REAL|0||0
>> 96|soilTemp1|REAL|0||0
>> 97|soilTemp2|REAL|0||0
>> 98|soilTemp3|REAL|0||0
>> 99|soilTemp4|REAL|0||0
>> 100|supplyVoltage|REAL|0||0
>> 101|txBatteryStatus|REAL|0||0
>> 102|UV|REAL|0||0
>> 103|uvBatteryStatus|REAL|0||0
>> 104|windBatteryStatus|REAL|0||0
>> 105|windchill|REAL|0||0
>> 106|windDir|REAL|0||0
>> 107|windGust|REAL|0||0
>> 108|windGustDir|REAL|0||0
>> 109|windrun|REAL|0||0
>> 110|windSpeed|REAL|0||0
>>
>>
>> On Thu, Dec 28, 2023 at 9:33 AM Vince Skahan  wrote:
>>
>>> On Thursday, December 28, 2023 at 5:26:09 AM UTC-8 Tom Keffer wrote:
>>>
>>> Alternatively, one could write a specialized algorithm for windchill.
>>> The sensible thing to do would be to read a year's worth of temperature and
>>> wind speed, all in one go --- one database access. Then use the results to
>>> calculate the year's worth of windchill. The downside is that it's not
>>> general at all: it would only know how to calculate windchill. The upside
>>> is that the existing xtypes API can be used right now to do this. You'd
>>> have to write extensions for all of your missing synthesized types.
>>>
>>>
>>> I see two things here.  One is extension(s) to handle the missing
>>> synthesized types in archive periods moving forward.  The second is some
>>> kind of standalone utility to 'one time' catch up a legacy db with those
>>> items that you wished it would have calculated in the ancient past.  Have
>>> the standalone utility to get the legacy pain over with 'once' so you don't
>>> have to feel that pain every 5 minutes moving forward
>>>
>>> Kinda like how rebuild-daily or calc-missing work (?)
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/d07cd082-dd2c-42bf-bef0-a051241b5388n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/d07cd082-dd2c-42bf-bef0-a051241b5388n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>>
>> --
>> -Ryan Stasel
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/16d8970b-5359-4726-9b54-ee571a8c41bcn%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/16d8970b-5359-4726-9b54-ee571a8c41bcn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEA%2BU9vJNqB7p6yqUR83NHO8_1gszwrfj2-sbOtYi_ZwVg%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

If you don't have an ET column in the database, but request a plot of one
in [ImageGenerator], then the image generator will calculate ET in software
using data in the database. That's likely to be expensive.

Running "weectl station reconfigure" will not add it back to the database.
You need "weectl database add-column", followed by "weectl
database calc-missing".

Keep track of what you're doing, including report run times. It will be
very interesting to see if it makes a big difference.

-tk

On Thu, Dec 28, 2023 at 8:11 PM Ryan Stasel  wrote:

> Testing this some more, and based on suggestion from Cameron, I have made
> a copy of the default Seasons template, and enabled in weewx.conf.
>
> Going through and removing pieces I don't have (ET, UV, etc) from
> skin.conf, [DisplayOptions], got my generation from 2m27s to 2m20s (no
> appreciable change) and subsequent runs of 1m45s. So it doesn't appear to
> be anything in DisplayOptions.
>
> Going a step further and commenting out pieces I don't have from
> [ImageGenerator] got me down to 1m7s. with subsequent runs of 3s.
>
> This was gathered running "time weectl report run SeasonsTest", and
> removing the output after each run.
>
> Going through and toggling specific ImageGenerator stanzas, issue appears
> to be ET. My station doesn't provide ET, so the column is blank... but it's
> there because I have a Vantage (weewx assumes vantagepro2, or the loop
> packets include just a null value, super unclear here).
>
> Looking at my DB, I don't seem to have an ET column (maybe I dropped it at
> some point in the past... vague recollection of there being bad data in
> there, and ). Maybe this explains the behavior? Is my best bet a
> "weectl database reconfigure" to bring things back to default, or just
> re-add via "weectl database add-column ET"?
>
> Would love some help!
>
> Thanks!
> -Ryan Stasel
>
> Here's what I get from listing columns in archive:
> pragma table_info(archive);
> 0|dateTime|INTEGER|1||1
> 1|usUnits|INTEGER|1||0
> 2|interval|INTEGER|1||0
> 3|altimeter|REAL|0||0
> 4|appTemp|REAL|0||0
> 5|appTemp1|REAL|0||0
> 6|barometer|REAL|0||0
> 7|batteryStatus1|REAL|0||0
> 8|batteryStatus2|REAL|0||0
> 9|batteryStatus3|REAL|0||0
> 10|batteryStatus4|REAL|0||0
> 11|batteryStatus5|REAL|0||0
> 12|batteryStatus6|REAL|0||0
> 13|batteryStatus7|REAL|0||0
> 14|batteryStatus8|REAL|0||0
> 15|cloudbase|REAL|0||0
> 16|co|REAL|0||0
> 17|co2|REAL|0||0
> 18|consBatteryVoltage|REAL|0||0
> 19|dewpoint|REAL|0||0
> 20|dewpoint1|REAL|0||0
> 21|extraHumid1|REAL|0||0
> 22|extraHumid2|REAL|0||0
> 23|extraHumid3|REAL|0||0
> 24|extraHumid4|REAL|0||0
> 25|extraHumid5|REAL|0||0
> 26|extraHumid6|REAL|0||0
> 27|extraHumid7|REAL|0||0
> 28|extraHumid8|REAL|0||0
> 29|extraTemp1|REAL|0||0
> 30|extraTemp2|REAL|0||0
> 31|extraTemp3|REAL|0||0
> 32|extraTemp4|REAL|0||0
> 33|extraTemp5|REAL|0||0
> 34|extraTemp6|REAL|0||0
> 35|extraTemp7|REAL|0||0
> 36|extraTemp8|REAL|0||0
> 37|forecast|REAL|0||0
> 38|hail|REAL|0||0
> 39|hailBatteryStatus|REAL|0||0
> 40|hailRate|REAL|0||0
> 41|heatindex|REAL|0||0
> 42|heatindex1|REAL|0||0
> 43|heatingTemp|REAL|0||0
> 44|heatingVoltage|REAL|0||0
> 45|humidex|REAL|0||0
> 46|humidex1|REAL|0||0
> 47|inDewpoint|REAL|0||0
> 48|inHumidity|REAL|0||0
> 49|inTemp|REAL|0||0
> 50|leafTemp1|REAL|0||0
> 51|leafTemp2|REAL|0||0
> 52|leafWet1|REAL|0||0
> 53|leafWet2|REAL|0||0
> 54|lightning_distance|REAL|0||0
> 55|lightning_disturber_count|REAL|0||0
> 56|lightning_energy|REAL|0||0
> 57|lightning_noise_count|REAL|0||0
> 58|lightning_strike_count|REAL|0||0
> 59|luminosity|REAL|0||0
> 60|maxSolarRad|REAL|0||0
> 61|nh3|REAL|0||0
> 62|no2|REAL|0||0
> 63|noise|REAL|0||0
> 64|o3|REAL|0||0
> 65|outHumidity|REAL|0||0
> 66|outTemp|REAL|0||0
> 67|pb|REAL|0||0
> 68|pm10_0|REAL|0||0
> 69|pm1_0|REAL|0||0
> 70|pm2_5|REAL|0||0
> 71|pressure|REAL|0||0
> 72|radiation|REAL|0||0
> 73|rain|REAL|0||0
> 74|rainBatteryStatus|REAL|0||0
> 75|rainRate|REAL|0||0
> 76|referenceVoltage|REAL|0||0
> 77|rxCheckPercent|REAL|0||0
> 78|signal1|REAL|0||0
> 79|signal2|REAL|0||0
> 80|signal3|REAL|0||0
> 81|signal4|REAL|0||0
> 82|signal5|REAL|0||0
> 83|signal6|REAL|0||0
> 84|signal7|REAL|0||0
> 85|signal8|REAL|0||0
> 86|snow|REAL|0||0
> 87|snowBatteryStatus|REAL|0||0
> 88|snowDepth|REAL|0||0
> 89|snowMoisture|REAL|0||0
> 90|snowRate|REAL|0||0
> 91|so2|REAL|0||0
> 92|soilMoist1|REAL|0||0
> 93|soilMoist2|REAL|0||0
> 94|soilMoist3|REAL|0||0
> 95|soilMoist4|REAL|0||0
> 96|soilTemp1|REAL|0||0
> 97|soilTemp2|REAL|0||0
> 98|soilTemp3|REAL|0||0
> 99|soilTemp4|REAL|0||0
>

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

>
> I wonder if this could be organized into
>
>   a database accessor to get multiple columns, that results in a stream
>   of N values
>
>   functions that accept N values to compute something else
>   (e.g. windchill from temp/wind).
>
>   a function that takes a computing function and a stream
>

This is pretty much what we have now.

dbmanager is the first;
xtypes.get_scalar() is the second;
XTypeTable.get_series() is the third.

What's missing is the ability to specify the N observation types. However,
one could just assume them all. That is, XTypeTable.get_series() would just
ask for all the data for a year, then call get_scaler(). Of course, that
might require a lot of memory. You could relax that requirement a bit by
doing the calculation in tranches --- perhaps a month's worth of data at a
time --- then stitch together the pieces.

Cameron: your comment that a savings of 100x would be needed for this to be
useful is about right. But, database accesses are *very* expensive. I think
you'd be surprised by the savings of reducing the number of db calls from
100k to one.

-tk

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECGVW%3D52pP6zp7f2qNM-13Op%2BvM8QqwLP9FVx7a4Rhsdg%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

Very useful data, Cameron.

Right now, if you want one year's worth of say, windchill, and it's not in
the database, it is calculated one record at a time. For a 5 minute archive
interval, that's 100k+ database accesses. It's a general algorithm, but
it's slow.

Alternatively, one could write a specialized algorithm for windchill. The
sensible thing to do would be to read a year's worth of temperature and
wind speed, all in one go --- one database access. Then use the results to
calculate the year's worth of windchill. The downside is that it's not
general at all: it would only know how to calculate windchill. The upside
is that the existing xtypes API can be used right now to do this. You'd
have to write extensions for all of your missing synthesized types.

How to write a more general algorithm that uses only a single database
access? It might be possible to set up a mechanism where an xtype specifies
which types it needs from the database first. Then get_series() calls
get_scalar() with the results. I'd have to think about that. It would
definitely be an extension to the xtypes API.

Thanks for setting us on the right course, Cameron.




On Thu, Dec 28, 2023 at 3:38 AM 'Cameron D' via weewx-development <
weewx-development@googlegroups.com> wrote:

> OK, this "ability to generate synthetic types"  was the clue to the
> excessive cpu and time.
> I started with a copy of my main DB converted to sqlite and trimmed it to
> 5 minute intervals and ~100k archive records, so it is close to the
> benchmark db.
>
> I did a bit of (my first ever) python profiling and compared the benchmark
> times with the same reports on my DB.
>
>- benchmark: The busiest weewx function *xtypes.py:128(get_aggregate)*
>was called ~16000 times for a total duration of 0.4 seconds.
>- my DB: the top function was: *xtypes.py:76(get_scalar)*, called 6
>milllion times, for a duration of 130 seconds, then
>*xtypes.py:27(get_scalar)*, was called 35 milllion times, total time
>84 seconds.
>
> Further down I could see lots of time spent in routines with names based
> on *windchill *or *heatindex*.
> From the start, I had decided I didn't want to see either of these
> parameters, so I had removed them from my DB.  But now Seasons is forcing a
> years' worth of their recalculation every report.  Hence durations of up to
> several minutes.
>
> I gradually trimmed down the skin.conf, and only by removing every trace
> of windchill, heatindex and tempfeel could I get the execution times down
> from 59 seconds to about 10 seconds - if I left a single reference
> remaining then it hung around the 56-59 seconds mark.
>
> The other, more minor, but still quite significant factor, is that all the
> other items listed like extratemp1, extratemp2, etc have been removed from
> my db. Referencing them in charts seems to take more cpu to *not *plot
> them than to have data and have them plotted.  By removing all the chart
> references to non-existent columns from my skin I got the elapsed time down
> from ~10 seconds to 2.3s.
>
> It is still refusing to print the times, either to console or the system
> log. (but I can see the debug output).  The times I quoted were user+system
> cpu time consumed, as reported by "time command..."
>
>
>
> On Thursday 28 December 2023 at 10:12:54 am UTC+10 Tom Keffer wrote:
>
>
> So, V5 image generation is slower, but then it's also doing more. V5 is
> able to generate plots of pure synthetic types, not just types in the
> database. That requires consulting the database to see whether it can do
> the type.
>
> -tk
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEB86SZ_NYQgzkPXfUOWrZmAVsYPvEgpzdwxmwpVL4COmQ%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

The database version number only comes into play if you change your archive
interval. Higher versions "weight" the result by the length of the interval.

One of my databases, now 18 years old, is still at V1.

On Wed, Dec 27, 2023 at 6:12 PM 'Cameron D' via weewx-development <
weewx-development@googlegroups.com> wrote:

> Tom,
> my question was not from the MySQL side, but because you said the weewx
> code consults the database to see if it can "do that type".
> In any case, I since realised tha I am seeing the same problem with the DB
> converted to sqlite, so that cannot be a cause.
>
> On a side question, is there much significance to the "version" number in
> the DB metadata?  I see one of my DBs is version 1 and the other is V4?
>
> Cameron.
>
> On Thursday 28 December 2023 at 10:52:26 am UTC+10 Tom Keffer wrote:
>
> On Wed, Dec 27, 2023 at 4:18 PM 'Cameron D' via weewx-development <
> weewx-de...@googlegroups.com> wrote:
>
> Tom,
> is there any chance that it is confused by me using  mysql storage type of
> float rather than real?
>
>
> I'm not a MySQL expert, but I doubt it.
>
> WeeWX does lots of little queries, which MySQL doesn't do as well as
> SQLite. It's possible that V5 is doing something that aggravates that.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/db5da043-0192-417a-ab27-838a82ebb47bn%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/db5da043-0192-417a-ab27-838a82ebb47bn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECNJU3p9LTQTcgnxrsZ0EE4pACKWqh_0AYcmYSDN-r15Q%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

On Wed, Dec 27, 2023 at 4:18 PM 'Cameron D' via weewx-development <
weewx-development@googlegroups.com> wrote:

> Tom,
> is there any chance that it is confused by me using  mysql storage type of
> float rather than real?
>

I'm not a MySQL expert, but I doubt it.

WeeWX does lots of little queries, which MySQL doesn't do as well as
SQLite. It's possible that V5 is doing something that aggravates that.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDdK_EBZQ4Qx-kx5--PJUy0N-9ame3Yur-KQcWSCBOcvg%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

Back to the benchmarks. I made a small technical mistake. The comparison
now stands at:

| Hardware  | WeeWX  | Python | Files (21) | Images (68) |
|---|||---:|:|
| MacBook Air, M1 2020  | 4.10.2 | 3.7.16 |  0.69s |   0.78s |
| MacBook Air, M1 2020  | 5.0.0b13   | 3.7.16 |  0.71s |   0.97s |
| MacBook Air, M1 2020  | 5.0.0rc1   | 3.7.16 |  0.69s |   0.93s |


So, V5 image generation is slower, but then it's also doing more. V5 is
able to generate plots of pure synthetic types, not just types in the
database. That requires consulting the database to see whether it can do
the type.

-tk

On Wed, Dec 27, 2023 at 3:50 PM Vince Skahan  wrote:

> The need to create the link 'was' the old bug I mentioned.  It was fixed
> quite a number of weeks ago, or so it seemed at the time that is.
>
> I cannot duplicate what you ran into, and everything works as expected
> with no errors seen here.
>
>- started with a clean deb system
>- installed v4 dpkg via the procedure for that
>- added one extension via wee_extension
>- restarted weewx v4 - the extension works fine with no errors seen
>- upgraded to v5 rc1 per that procedure, taking the default answers
>when prompted
>- restarted weewx v5 - the extension still works fine with no errors
>seen
>- and the old user tree under /usr/share/weewx was renamed to be
>user.mmddhhmmss as expected
>
> You're going to have to precisely document step-by-step exactly how you
> made something break in order for somebody to be able to work the issue if
> you're still having it.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/b755bb2d-b64f-4f78-812e-dbae0224f1f9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECTbuY%2BnyiANU5%2BH9c42NfNuYRfJ47qQESvbHZGcB4Zrg%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

I can confirm that V5 does have a performance problem with image
generation. Here's how the various versions compare (using the file and
image generation benchmark <https://github.com/weewx/weewx-benchmark>).

| Hardware| WeeWX| Python | Files (21) | Images (68)
|-|--||---:|
| MacBook Air, M1 2020| 4.10.2   | 3.7.16 |  0.69s |   0.78s
| MacBook Air, M1 2020| 5.0.0b13 | 3.7.16 |  0.71s |   0.97s
| MacBook Air, M1 2020| 5.0.0rc1 | 3.7.16 |  0.74s |   1.28s


On Tue, Dec 26, 2023 at 8:31 PM 'Cameron D' via weewx-development <
weewx-development@googlegroups.com> wrote:

> I copied the benchmark DB and the conf file onto my main weewx
> installation, edited the conf file accordingly, and got the same benchmark
> results, about 1.5 seconds each.
>
> So I can only assume there is something in my customisations that V5 is
> having trouble  with.
>
> On Wednesday 27 December 2023 at 12:25:22 pm UTC+10 Cameron D wrote:
>
>> Benchmark results: 1.4s for files and 1.5s for images
>>
>> Added to Wiki.
>>
>> On Wednesday 27 December 2023 at 12:02:02 pm UTC+10 Tom Keffer wrote:
>>
>>> Can you please run the benchmark that Vince linked to earlier:
>>> https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation
>>>
>>> Unfortunately, it runs only under V5 (not v4.10). Still, it will give us
>>> an idea whether you're in the right ballpark for runtimes.
>>>
>>> -tk
>>>
>>> On Tue, Dec 26, 2023 at 5:38 PM 'Cameron D' via weewx-development <
>>> weewx-de...@googlegroups.com> wrote:
>>>
>>>> 74 seconds clock time to run a seasons report looks like the same
>>>> problem I am seeing.
>>>>
>>>>
>>>> On Wednesday 27 December 2023 at 11:25:17 am UTC+10 rcst...@gmail.com
>>>> wrote:
>>>>
>>>> Hey All,
>>>>
>>>> ...
>>>>
>>>> Here’s 4.10
>>>>
>>>> Dec 26 17:00:25 raspi-server-misc weewx[25329] INFO
>>>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.79
>>>> seconds
>>>>
>>>>
>>>>
>>>> *...*
>>>>
>>>> Here’s 5.0rc1
>>>>
>>>> Dec 26 17:21:47 raspi-server-misc weewxd[716]: INFO
>>>> weewx.imagegenerator: Generated 12 images for report SeasonsReport in 74.31
>>>> seconds
>>>>
>>>> --
>>>>
>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to weewx-developm...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-development/a8336c10-46c9-41f7-96ce-a6b248cc4431n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/weewx-development/a8336c10-46c9-41f7-96ce-a6b248cc4431n%40googlegroups.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/122b7ed6-0a04-4947-8f51-9a603abcb688n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/122b7ed6-0a04-4947-8f51-9a603abcb688n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDchmGYwrfBa%2B7BEQR3Vfe6zN%2BcR1m9uGuPUn9yhJU5NA%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

Can you please run the benchmark that Vince linked to earlier:
https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation

Unfortunately, it runs only under V5 (not v4.10). Still, it will give us an
idea whether you're in the right ballpark for runtimes.

-tk

On Tue, Dec 26, 2023 at 5:38 PM 'Cameron D' via weewx-development <
weewx-development@googlegroups.com> wrote:

> 74 seconds clock time to run a seasons report looks like the same problem
> I am seeing.
>
>
> On Wednesday 27 December 2023 at 11:25:17 am UTC+10 rcst...@gmail.com
> wrote:
>
> Hey All,
>
> ...
>
> Here’s 4.10
>
> Dec 26 17:00:25 raspi-server-misc weewx[25329] INFO
> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.79
> seconds
>
>
>
> *...*
>
> Here’s 5.0rc1
>
> Dec 26 17:21:47 raspi-server-misc weewxd[716]: INFO weewx.imagegenerator:
> Generated 12 images for report SeasonsReport in 74.31 seconds
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/a8336c10-46c9-41f7-96ce-a6b248cc4431n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDg6Dk9J%3D6DPer%2BPo8%2Bso3Myr4YAzpqx2kO1mMs0cNp1A%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

Yes, there is a packaged version, so, yes, you do a package upgrade. NOTE:
the release candidate is in a temporary apt repository, as outlined in the
first message of this thread.

On Tue, Dec 26, 2023 at 3:11 PM  wrote:

> Hey All,
>
>
>
> I’m going to give this a go, but want to confirm based on this:
> https://weewx.com/docs/5.0/upgrade/
>
>
>
> I’m just running on a raspi, and installed via apt, so I should just be
> able to do as documented here and upgrade this way? I swear a few months
> back there wasn’t going to be a packaged version of 5, and the emphasis was
> pushing toward using pip… that changed along the way? (sorry for being
> oblivious to that change, I try to at least glance at all the emails via
> this list, I must have just missed it).
>
>
>
> Thanks!
>
>
>
> *From:* weewx-development@googlegroups.com <
> weewx-development@googlegroups.com> *On Behalf Of *Tom Keffer
> *Sent:* Thursday, December 21, 2023 3:07 PM
> *To:* weewx-development 
> *Subject:* [weewx-development] V5.0 release candidate available
>
>
>
> Celebrate the solstice (coming up in 4 hours) and the start of Winter with
> a WeeWX release candidate!
>
>
>
> *Changes since b17*
>
> We gave up on a standalone logger and have gone back to whatever your
> system uses. This is more robust to permission problems, but may make it
> harder to get the logs back out. See the wiki article *View logs
> <https://github.com/weewx/weewx/wiki/view-logs>* for tips on how to
> convince your system to let you have a look.
>
>
>
> As always, you can add a [Logging] section to weewx.conf and set up your
> own rotating log system. See the article *WeeWX V4 and logging
> <https://github.com/weewx/weewx/wiki/WeeWX-v4-and-logging>* for how to do
> this.
>
>
>
> *Pip*
>
>
>
> For pip installs, please delete your old virtual environment, then install
> from scratch by following the pip install instructions
> <https://weewx.com/docs/5.0/quickstarts/pip/#install-in-a-virtual-environment>.
> While upgrading should work, we are particularly interested in the
> experience of a new install, including setting up a daemon and udev files.
> Make sure to follow the new instructions that use a daemon setup script.
>
>
>
>
>
> *Debian*
>
>
>
> For Debian package installs, modify /etc/apt/sources.list as follows:
>
> *echo "deb [arch=all] https://weewx.com/apt-test/python3 
> <https://weewx.com/apt-test/python3> buster main" | sudo tee 
> /etc/apt/sources.list.d/weewx.list*
>
> Note the "apt-test". This tells apt to look there for the beta release,
> instead of the normal repository. You may want to change it back when
> you're done.
>
>
>
>
>
> *Red Hat*
>
>
>
> For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo
>
> *[weewx]*
>
> *name=weewx*
>
> *baseurl=http://weewx.com/yum-test/weewx/el9 
> <http://weewx.com/yum-test/weewx/el9>*
>
> *enabled=1*
>
> Note the "yum-test". This tells yum to look there for the beta release,
> instead of the normal repository. Again, you may want to change things back
> when you're done.
>
>
>
>
>
> *What we're looking for*
>
>
>
> 1. We are interested in your upgrade experience. Did the installers get
> your configuration file right? Skins? Is the daemon configured correctly?
>
>
>
> 2. With this release, we have included udev rules to set the correct
> permissions for devices. Did it work? If not, did unplugging then
> replugging the device work?
>
>
>
> 3. How about logging? Is it going to your system logger? Do the labels
> look reasonable?
>
>
>
> *Thanks!*
>
>
>
> -Tom & Matt
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/CAPq0zECFSEWf9immM3fFroWU79GcYyT65ibe8w1hWd-XJ92s1g%40mail.gmail.com
> <https://groups.google.com/d/msgid/weewx-development/CAPq0zECFSEWf9immM3fFroWU79GcYyT65ibe8w1hWd-XJ92s1g%40mail.gmail.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBkSBKKss8spyJj1P85hk8v%2BTeKojNo6OAq6%2BRYAHtqxQ%40mail.gmail.com.

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

So, it sounds like you can't reproduce it either...?

Because it looks like a hard crash of the Python interpreter, it is likely
it was caused by a bad cache, or something like that.

On Tue, Dec 26, 2023 at 1:46 PM Vince Skahan  wrote:

> What I did was install a clean beta via pip, then overwrite extension.py
> down buried in the venv from the later one you'd uploaded to github.  I
> 'think' that I hand-edited in one patch as you'd previously updated
> multiple things after the beta was released, but my memory is a little hazy
> there.
>
> In thinking about it after the fact, I wish I had stopped weewx and
> deleted all the __pycache__ trees in the venv, then restarted weewx and
> retried the same command  to see if the issue went away.  A clean install
> worked.  A later try where I was more careful hand-editing the diff in
> worked.
>
> On Tuesday, December 26, 2023 at 1:23:34 PM UTC-8 Tom Keffer wrote:
>
>> I am trying to reproduce the result that Vince and Cameron have reported,
>> but am unable to do so. More details would be helpful.
>>
>> Did you both do an upgrade from v4.10.2 to v5.0.0rc1?
>>
>> Did you have any extensions installed?
>>
>> Vince, I know you were running "weectl report run" using Python 3.11,
>> but, Cameron, what were you running?
>>
>> Cameron, do you have a stack trace?
>>
>> It's hard to debug because the snippet that Vince has does not show the
>> type of the underlying exception.
>>
>> -tk
>>
>> On Tue, Dec 26, 2023 at 9:02 AM Vince Skahan  wrote:
>>
>>> If you're running a throwaway vm, try running the benchmarks at
>>> https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation
>>> and see what your numbers look like.  You should see numbers well under 3
>>> seconds on that kind of gear.
>>>
>>> On Tuesday, December 26, 2023 at 6:52:42 AM UTC-8 Cameron D wrote:
>>>
>>>> I ran some more tests with my main DB converted to sqlite, using weectl
>>>> to run repots using the default Seasons skin.
>>>>
>>>> The times remained basically constant  at about  4.5 minutes user+sys
>>>> CPU as I kept deleting old records.
>>>> As I got below 500k records the times dropped proportionally to the
>>>> number of archive records, extrapolating back to about 20 seconds for no
>>>> records (which is still a crazy long time.
>>>>
>>>> In addition the various gaps between the file creation times also
>>>> dropped in proportion.
>>>>
>>>> The bit I wrote in the previous post about memory use did not apply to
>>>> this set of tests, as the value just stuck steadily to 120MB. I think I
>>>> have lost track of some of the tests I was doing.
>>>>
>>>> On Tuesday 26 December 2023 at 6:04:05 pm UTC+10 Cameron D wrote:
>>>>
>>>>> The wmr300 DB has 3.7million records, and the ecowitt DB has only
>>>>> 530k.  The system is a VM running on an i5 - the VM is allocated 8GB ram
>>>>> and 4 cores.  It was not stressed at all by weewx V4.
>>>>>
>>>>> The CPU usage is all in the python, not in the mariadb server.
>>>>>
>>>>> While the python code is thrashing around achieving nothing, the
>>>>> memory allocation for the report script is oscillating from under 200MB to
>>>>> 780M - no swapping, the machine still has 4GB free.
>>>>>
>>>>> On Tuesday 26 December 2023 at 4:11:59 pm UTC+10 Vince Skahan wrote:
>>>>>
>>>>>> Cameron several of us have run v5 with very large db of over 10 years
>>>>>> data on pi4 or lesser boxes without such issue, so a bit more data from 
>>>>>> you
>>>>>> would be helpful. How big a size are you running ? On what hardware ?
>>>>>>
>>>>>> --
>>>
>> You received this message because you are subscribed to the Google Groups
>>> "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/c5270d4d-0008-4b23-ac1b-5ab4647e2e8bn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/c5270d4d-0008-4b23-ac1b-5ab4647e2e8bn%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message becaus

Re: [weewx-development] Re: V5.0 release candidate available

[Tom Keffer]

I am trying to reproduce the result that Vince and Cameron have reported,
but am unable to do so. More details would be helpful.

Did you both do an upgrade from v4.10.2 to v5.0.0rc1?

Did you have any extensions installed?

Vince, I know you were running "weectl report run" using Python 3.11, but,
Cameron, what were you running?

Cameron, do you have a stack trace?

It's hard to debug because the snippet that Vince has does not show the
type of the underlying exception.

-tk

On Tue, Dec 26, 2023 at 9:02 AM Vince Skahan  wrote:

> If you're running a throwaway vm, try running the benchmarks at
> https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation
> and see what your numbers look like.  You should see numbers well under 3
> seconds on that kind of gear.
>
> On Tuesday, December 26, 2023 at 6:52:42 AM UTC-8 Cameron D wrote:
>
>> I ran some more tests with my main DB converted to sqlite, using weectl
>> to run repots using the default Seasons skin.
>>
>> The times remained basically constant  at about  4.5 minutes user+sys CPU
>> as I kept deleting old records.
>> As I got below 500k records the times dropped proportionally to the
>> number of archive records, extrapolating back to about 20 seconds for no
>> records (which is still a crazy long time.
>>
>> In addition the various gaps between the file creation times also dropped
>> in proportion.
>>
>> The bit I wrote in the previous post about memory use did not apply to
>> this set of tests, as the value just stuck steadily to 120MB. I think I
>> have lost track of some of the tests I was doing.
>>
>> On Tuesday 26 December 2023 at 6:04:05 pm UTC+10 Cameron D wrote:
>>
>>> The wmr300 DB has 3.7million records, and the ecowitt DB has only 530k.
>>> The system is a VM running on an i5 - the VM is allocated 8GB ram and 4
>>> cores.  It was not stressed at all by weewx V4.
>>>
>>> The CPU usage is all in the python, not in the mariadb server.
>>>
>>> While the python code is thrashing around achieving nothing, the memory
>>> allocation for the report script is oscillating from under 200MB to 780M -
>>> no swapping, the machine still has 4GB free.
>>>
>>> On Tuesday 26 December 2023 at 4:11:59 pm UTC+10 Vince Skahan wrote:
>>>
 Cameron several of us have run v5 with very large db of over 10 years
 data on pi4 or lesser boxes without such issue, so a bit more data from you
 would be helpful. How big a size are you running ? On what hardware ?

 --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/c5270d4d-0008-4b23-ac1b-5ab4647e2e8bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zED9xuYZdxBHwcrjY019d%3DQBVf7cLbED-hogm1SEu0qYHQ%40mail.gmail.com.

Re: [weewx-development] Use tags in Python extension code?

[Tom Keffer]

In order to do this, a lot of infrastructure has to be in place. Take a
look at the test file test_daily.py
,
function testTags

(line 206).

It sets up a TimeBinder object (line 222). Once you have that, you can do
things like

print(tagStats.trend.barometer)

For many of these aggregates, you can use the xtypes
 system, which is a lot simpler
to access. For example, to get the day's rain

val_tuple = xtypes.get_aggregate('rain', timespan, 'sum', db_manager)

where timespan is the start/stop times over which the aggregation is to be
taken, and db_manager is an open database manager object. It returns a
ValueTuple .

If you're not a Python programmer, it's going to be tough. :-(

-tk

On Sun, Dec 17, 2023 at 3:38 PM nfbarg...@gmail.com 
wrote:

> This likely has everything to do with my lack of Python acuity than WeeWX.
>
> When obtaining values for a user extension that does not have an
> associated skin so I am not using the Cheetah Generator, how might I call
> into the tag system to get the data such as 'trend.barometer'?  in
> cheetahgenerator.py I see the Stats class that looks like it should be what
> should be called but I don't see it subclassed anywhere but is part of
> 'default_search_list'.
>
> Primarily I am interested in 'trend.barometer' and 'day.rain'.  The rest I
> am getting from the latest archive record.  I've not found any extensions
> that use the tag system like this that I can be inspired by so I am asking
> the experts.
>
> TIA
>
> - Nate
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/b4a1b9d6-2e1e-4132-8731-477c2d6018f8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAsOmrfyb5p0T%3Ds0vxae4MQNdchLHkFh%3D_YUue7U8AOhw%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

On Fri, Dec 22, 2023 at 5:46 PM Vince Skahan  wrote:

> It worked with the --config switch specified ?


Yes. I tried macOS, Debian 12, Ubuntu 23.04, with a mix of Python 3.7 and
3.11 --- all worked with the --config switch.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAr7uQbv_Dcvsz2UVAKGL-g352yAmJoFaFFpz8t8epvEg%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

Hmm, "weectl report run" works perfectly on my system, using Python 3.11.

It's not clear from your stack trace what raised the exception. Can you dig
deeper?

-tk

On Fri, Dec 22, 2023 at 5:30 PM Vince Skahan  wrote:

> I hand-edited the diff into
> weewx-venv/lib/python3.11/site-packages/weecfg/extension.py which worked
> for installing the extensions, but I get a similar message trying to run
> reports manually
>
> You might try this on a current fully patched up version to see if it
> works with all the updates you made after release to pypi just in case...
>
> (weewx-venv) vagrant@deb12:~/weewx-data$ weectl report run
> --config=simulator.conf
> Using configuration file simulator.conf
> All enabled reports will be run.
> Generating as of last timestamp in the database.
> Traceback (most recent call last):
>   File "/home/vagrant/weewx-venv/bin/weectl", line 8, in 
> sys.exit(main())
>  ^^
>   File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py",
> line 66, in main
> namespace.func(namespace)
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py",
> line 96, in dispatch
> namespace.action_func(config_dict, namespace)
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/report_cmd.py",
> line 92, in run_reports
> weectllib.report_actions.run_reports(config_dict,
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/report_actions.py",
> line 84, in run_reports
> engine = weewx.engine.DummyEngine(config_dict)
>  ^
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weewx/engine.py",
> line 89, in __init__
> self.loadServices(config_dict)
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weewx/engine.py",
> line 157, in loadServices
> obj = weeutil.weeutil.get_object(svc)(self, config_dict)
>   ^^^
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weeutil/weeutil.py",
> line 1404, in get_object
> module = importlib.import_module(module_name)
>  
>   File "/usr/lib/python3.11/importlib/__init__.py", line 126, in
> import_module
> return _bootstrap._gcd_import(name[level:], package, level)
>
>   File "", line 1206, in _gcd_import
>   File "", line 1178, in _find_and_load
>   File "", line 1128, in
> _find_and_load_unlocked
>   File "", line 241, in
> _call_with_frames_removed
>   File "", line 1206, in _gcd_import
>   File "", line 1178, in _find_and_load
>   File "", line 1142, in
> _find_and_load_unlocked
>
>
> On Friday, December 22, 2023 at 4:51:49 PM UTC-8 Tom Keffer wrote:
>
>> Thanks for spotting that, Vince!
>>
>> Fixed in commit 256cac5
>> <https://github.com/weewx/weewx/commit/256cac54931bf03f11f1153917c1bfc43fcc34a1>
>> .
>>
>> On Fri, Dec 22, 2023 at 4:31 PM Vince Skahan  wrote:
>>
>>> Tom - your xaggs extension isn't installing.  I see the identical issue
>>> with one of my custom extensions too.
>>>
>>> (weewx-venv) vagrant@deb12:~/adds$ weectl extension install
>>> --config=/home/vagrant/weewx-data/simulator.conf weewx-xaggs-master/
>>> Using configuration file /home/vagrant/weewx-data/simulator.conf
>>> Request to install 'weewx-xaggs-master/'.
>>> Traceback (most recent call last):
>>>   File "/home/vagrant/weewx-venv/bin/weectl", line 8, in 
>>> sys.exit(main())
>>>  ^^
>>>   File
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py", line 66,
>>> in main
>>> namespace.func(namespace)
>>>   File
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py",
>>> line 96, in dispatch
>>> namespace.action_func(config_dict, namespace)
>>>   File
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/extension_cmd.py",
>>> line 116, in install_extension
>>> ext.install_extension(namespace.source)
>>>   File
>>> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py",
>>> line 125, in install_extension
>>> extension_name = self.install_from_dir(extension_path)
>>>  

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

Thanks for spotting that, Vince!

Fixed in commit 256cac5

.

On Fri, Dec 22, 2023 at 4:31 PM Vince Skahan  wrote:

> Tom - your xaggs extension isn't installing.  I see the identical issue
> with one of my custom extensions too.
>
> (weewx-venv) vagrant@deb12:~/adds$ weectl extension install
> --config=/home/vagrant/weewx-data/simulator.conf weewx-xaggs-master/
> Using configuration file /home/vagrant/weewx-data/simulator.conf
> Request to install 'weewx-xaggs-master/'.
> Traceback (most recent call last):
>   File "/home/vagrant/weewx-venv/bin/weectl", line 8, in 
> sys.exit(main())
>  ^^
>   File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py",
> line 66, in main
> namespace.func(namespace)
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py",
> line 96, in dispatch
> namespace.action_func(config_dict, namespace)
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/extension_cmd.py",
> line 116, in install_extension
> ext.install_extension(namespace.source)
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py",
> line 125, in install_extension
> extension_name = self.install_from_dir(extension_path)
>  ^
>   File
> "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py",
> line 201, in install_from_dir
> save_config |= self._inject_config(installer['config'], extension_name)
>~^^
> KeyError: 'config'
>
> (weewx-venv) vagrant@deb12:~/adds$ pip3 list
> PackageVersion
> -- --
> certifi2023.11.17
> charset-normalizer 3.3.2
> configobj  5.0.8
> CT33.3.3
> ephem  4.1.5
> idna   3.6
> paho-mqtt  1.6.1
> Pillow 10.1.0
> pip23.0.1
> pyephem9.99
> PyMySQL1.1.0
> pyserial   3.5
> pyusb  1.2.1
> requests   2.31.0
> setuptools 66.1.1
> six1.16.0
> urllib32.1.0
> weewx  5.0.0rc1
> wheel  0.42.0
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/73f123da-7608-46d5-add0-a2b9b870d1ffn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECTEGMYxhtD3yJJ5X00d_1ZBPWPj32%3DON3mtrmyBdXovw%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

After a bit more investigation, it appears that some systems install
python3-wheel when they install python3-pip, some don't. Hence the
confusing results.


On Fri, Dec 22, 2023 at 2:33 PM Tom Keffer  wrote:

> Hmmm, that didn't work. If you list "wheel" as a dependency, pip isn't
> smart enough to install it first before attempting the others. So, you get
> the same error. I'm reluctant to create yet another step in the install
> process.
>
>
>
> On Fri, Dec 22, 2023 at 2:24 PM Greg  wrote:
>
>> Yes having install wheel first removes all my errors. I think it should
>> be installed by default to save any questions from people like me who don't
>> really know how this all works and what depends on what. :)
>>
>> Because I was already running the beta V5 I just renamed my venv and
>> recreated it so my weewx data would work and also if it didn't work I could
>> rename it back and all would be good. (in theory) I have installed in
>> /opt/weewx
>>
>> All looks good so far.
>>
>> https://weather.ubeaut.work/
>>
>> Thanks for the support.
>>
>>
>> On Saturday 23 December 2023 at 06:53:11 UTC+11 Vince Skahan wrote:
>>
>>> Works for me on deb12 vagrant and pi5 deb12 based raspios
>>>
>>> On Friday, December 22, 2023 at 11:42:48 AM UTC-8 Tom Keffer wrote:
>>>
>>>> I was able to install both paho-mqtt and ephem (NB: these days, it's
>>>> just "ephem", not "pyephem") without warnings on Debian 12 by installing
>>>> "wheel" first:
>>>>
>>>> *python3 -m pip install wheel*
>>>> *python3 -m pip install ephem*
>>>> *python3 -m pip install paho-mqtt*
>>>>
>>>> Does this work for others? If so, we can put "wheel" in the
>>>> requirements list for pip installs.
>>>>
>>>> -tk
>>>>
>>>>
>>>> On Fri, Dec 22, 2023 at 11:19 AM Greg  wrote:
>>>>
>>>>> Oops...I meant python version is 3.11 not pip. Pip is at the latest
>>>>> version.
>>>>> So I guess I install this way  pip 23.1 will enforce this behaviour
>>>>> change. A possible replacement is to enable the '--use-pep517'?
>>>>> I am running debian 12 by the way. Anyone else seen these messages?
>>>>>
>>>>> On Saturday 23 December 2023 at 00:09:15 UTC+11 Tom Keffer wrote:
>>>>>
>>>>>> Thanks, all! Keep them coming.
>>>>>>
>>>>>> 1. The syslog comment is annoying, but harmless. We've changed the
>>>>>> weewx unit service file to specify StandardError of journal+console. 
>>>>>> Commit
>>>>>> 940eff4
>>>>>> <https://github.com/weewx/weewx/commit/940eff463d9af28932fbd3e1464c5e3858567d53>
>>>>>> .
>>>>>>
>>>>>> 2. Missing Vantage records. I've noticed variants of this problem
>>>>>> when it takes a long time to download records from the console. My best 
>>>>>> is
>>>>>> that while the console is busy emitting historic records, it neglects to
>>>>>> create new ones. Hence, you miss a few records. I don't think it has
>>>>>> anything to do with V5, as the driver code hasn't changed much.
>>>>>>
>>>>>> 3. Greg, I would guess that your problems are due to using a very old
>>>>>> version of pip. You're at 3.11, but the current version is 23.3. See the
>>>>>> wiki article *Troubleshooting pip installs
>>>>>> <https://github.com/weewx/weewx/wiki/pip-troubleshooting>* and see
>>>>>> if that helps.
>>>>>>
>>>>>> -tk
>>>>>>
>>>>>> On Fri, Dec 22, 2023 at 12:45 AM Greg  wrote:
>>>>>>
>>>>>>> I installed weewx using the pip method and followed the instructions.
>>>>>>> I did a pip list before installing to get the list of what other
>>>>>>> dependencies I am using.
>>>>>>>  When I installed paho-mqtt and pyephem I got these errors:
>>>>>>>
>>>>>>> Installing collected packages: paho-mqtt
>>>>>>>   DEPRECATION: paho-mqtt is being installed using the legacy
>>>>>>> 'setup.py install' method, because it does not have a 'pyproject.toml' 
>>>>>>> and
>>>>>>> the 'wheel' package is not instal

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

Hmmm, that didn't work. If you list "wheel" as a dependency, pip isn't
smart enough to install it first before attempting the others. So, you get
the same error. I'm reluctant to create yet another step in the install
process.



On Fri, Dec 22, 2023 at 2:24 PM Greg  wrote:

> Yes having install wheel first removes all my errors. I think it should be
> installed by default to save any questions from people like me who don't
> really know how this all works and what depends on what. :)
>
> Because I was already running the beta V5 I just renamed my venv and
> recreated it so my weewx data would work and also if it didn't work I could
> rename it back and all would be good. (in theory) I have installed in
> /opt/weewx
>
> All looks good so far.
>
> https://weather.ubeaut.work/
>
> Thanks for the support.
>
>
> On Saturday 23 December 2023 at 06:53:11 UTC+11 Vince Skahan wrote:
>
>> Works for me on deb12 vagrant and pi5 deb12 based raspios
>>
>> On Friday, December 22, 2023 at 11:42:48 AM UTC-8 Tom Keffer wrote:
>>
>>> I was able to install both paho-mqtt and ephem (NB: these days, it's
>>> just "ephem", not "pyephem") without warnings on Debian 12 by installing
>>> "wheel" first:
>>>
>>> *python3 -m pip install wheel*
>>> *python3 -m pip install ephem*
>>> *python3 -m pip install paho-mqtt*
>>>
>>> Does this work for others? If so, we can put "wheel" in the requirements
>>> list for pip installs.
>>>
>>> -tk
>>>
>>>
>>> On Fri, Dec 22, 2023 at 11:19 AM Greg  wrote:
>>>
>>>> Oops...I meant python version is 3.11 not pip. Pip is at the latest
>>>> version.
>>>> So I guess I install this way  pip 23.1 will enforce this behaviour
>>>> change. A possible replacement is to enable the '--use-pep517'?
>>>> I am running debian 12 by the way. Anyone else seen these messages?
>>>>
>>>> On Saturday 23 December 2023 at 00:09:15 UTC+11 Tom Keffer wrote:
>>>>
>>>>> Thanks, all! Keep them coming.
>>>>>
>>>>> 1. The syslog comment is annoying, but harmless. We've changed the
>>>>> weewx unit service file to specify StandardError of journal+console. 
>>>>> Commit
>>>>> 940eff4
>>>>> <https://github.com/weewx/weewx/commit/940eff463d9af28932fbd3e1464c5e3858567d53>
>>>>> .
>>>>>
>>>>> 2. Missing Vantage records. I've noticed variants of this problem when
>>>>> it takes a long time to download records from the console. My best is that
>>>>> while the console is busy emitting historic records, it neglects to create
>>>>> new ones. Hence, you miss a few records. I don't think it has anything to
>>>>> do with V5, as the driver code hasn't changed much.
>>>>>
>>>>> 3. Greg, I would guess that your problems are due to using a very old
>>>>> version of pip. You're at 3.11, but the current version is 23.3. See the
>>>>> wiki article *Troubleshooting pip installs
>>>>> <https://github.com/weewx/weewx/wiki/pip-troubleshooting>* and see if
>>>>> that helps.
>>>>>
>>>>> -tk
>>>>>
>>>>> On Fri, Dec 22, 2023 at 12:45 AM Greg  wrote:
>>>>>
>>>>>> I installed weewx using the pip method and followed the instructions.
>>>>>> I did a pip list before installing to get the list of what other
>>>>>> dependencies I am using.
>>>>>>  When I installed paho-mqtt and pyephem I got these errors:
>>>>>>
>>>>>> Installing collected packages: paho-mqtt
>>>>>>   DEPRECATION: paho-mqtt is being installed using the legacy
>>>>>> 'setup.py install' method, because it does not have a 'pyproject.toml' 
>>>>>> and
>>>>>> the 'wheel' package is not installed. pip 23.1 will enforce this 
>>>>>> behaviour
>>>>>> change. A possible replacement is to enable the '--use-pep517' option.
>>>>>> Discussion can be found at https://github.com/pypa/pip/issues/8559
>>>>>>   Running setup.py install for paho-mqtt ... done
>>>>>> Successfully installed paho-mqtt-1.6.1
>>>>>>
>>>>>> /opt/weewx$ python3 -m pip install pyephem
>>>>>> Collecting pyephem
>>>>>>   Using cached pyephem-9.99.tar.gz (1.4 kB

Re: [weewx-development] Re: PostgreSQL support, which branch to target?

[Tom Keffer]

I wish I knew more about SQL programming, but, alas, I know just enough to
get by. I'd be very grateful for opinions from a true expert.

On Fri, Dec 22, 2023 at 12:35 PM Raoul Snyman 
wrote:

> Actually, the columns names are fine for me, it was the expression inside
> the SUM function which was throwing me off.
>
> I set up a test database in SQLite and ran those queries, and it looks
> like they are actually just doing a count (boolean true becomes 1, sum of a
> bunch of 1s is a count). I'm going to test changing those queries to
> count() and moving the expression into the where clause.
>
> On Friday, December 22, 2023 at 1:15:09 PM UTC Tom Keffer wrote:
>
>> Typically, this is used to calculate the number of days of precipitation
>> greater than some amount during a time span, using the daily summaries. The
>> parameter %(val)s is the value.
>>
>> In retrospect, the choice of column names in the daily summaries may not
>> have been a good one. The values "sum" in the expression "SUM(sum>90)"
>> represent two different things. The first sum ("SUM") is an aggregation,
>> the second sum ("sum") is a column name.
>>
>> Perhaps you can escape the inner "sum"?
>>
>> -tk
>>
>> On Thu, Dec 21, 2023 at 10:12 PM Raoul Snyman 
>> wrote:
>>
>>> I've been going through the queries in xtypes.py, and I've come across
>>> some queries that don't make sense to me, and definitely throw errors in
>>> PostgreSQL.
>>>
>>> For example:
>>>
>>> SELECT SUM(sum >= %(val)s) FROM %(table_name)s_day_%(obs_key)s WHERE
>>> dateTime >= %(start)s AND dateTime < %(stop)s
>>>
>>> SUM(sum >= 90) is not valid in PostgreSQL. Can you tell me what this is
>>> supposed to be doing, then perhaps I can translate the SQL into something
>>> that's more generic?
>>>
>>> Thanks!
>>>
>>> On Tuesday, December 12, 2023 at 3:27:19 PM UTC Raoul Snyman wrote:
>>>
>>>> On Tuesday, December 12, 2023 at 2:15:31 PM UTC matthew wall wrote:
>>>>
>>>> On Monday, December 11, 2023 at 11:47:04 PM UTC-5 Raoul Snyman wrote:
>>>>
>>>> I still don't have the tests running properly, I couldn't find any docs
>>>> on exactly how to set up the tests, and both the MySQL and PostgreSQL tests
>>>> fail when I just run "make test"
>>>>
>>>>
>>>> i am working on the unit tests.  when we started them we just used
>>>> python directly. i am converting all of it to pytest.
>>>>
>>>> i would like to have the pytest conversion done when v5 comes out of
>>>> beta, but it might happen soon after that.
>>>>
>>>> the core in v5 is pretty solid, but we're still taking care of some
>>>> packaging and integration bits
>>>>
>>>>
>>>> Oh cool, that's good to know. I've used pytest fairly extensively, it's
>>>> a good test framework.
>>>>
>>>>
>>>>
>>> --
>>>
>> You received this message because you are subscribed to the Google Groups
>>> "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/b298f3e6-ead9-4c5a-8458-d204cb0ac03dn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/b298f3e6-ead9-4c5a-8458-d204cb0ac03dn%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/4f0c1553-dd27-4831-a832-c270f2183ff6n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/4f0c1553-dd27-4831-a832-c270f2183ff6n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECBttYxapL4qrONoXf1Nmj_bz6Dyq9QDhX0PqQkhQNTwA%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

I was able to install both paho-mqtt and ephem (NB: these days, it's just
"ephem", not "pyephem") without warnings on Debian 12 by installing "wheel"
first:

*python3 -m pip install wheel*
*python3 -m pip install ephem*
*python3 -m pip install paho-mqtt*

Does this work for others? If so, we can put "wheel" in the requirements
list for pip installs.

-tk


On Fri, Dec 22, 2023 at 11:19 AM Greg  wrote:

> Oops...I meant python version is 3.11 not pip. Pip is at the latest
> version.
> So I guess I install this way  pip 23.1 will enforce this behaviour
> change. A possible replacement is to enable the '--use-pep517'?
> I am running debian 12 by the way. Anyone else seen these messages?
>
> On Saturday 23 December 2023 at 00:09:15 UTC+11 Tom Keffer wrote:
>
>> Thanks, all! Keep them coming.
>>
>> 1. The syslog comment is annoying, but harmless. We've changed the weewx
>> unit service file to specify StandardError of journal+console. Commit
>> 940eff4
>> <https://github.com/weewx/weewx/commit/940eff463d9af28932fbd3e1464c5e3858567d53>
>> .
>>
>> 2. Missing Vantage records. I've noticed variants of this problem when it
>> takes a long time to download records from the console. My best is that
>> while the console is busy emitting historic records, it neglects to create
>> new ones. Hence, you miss a few records. I don't think it has anything to
>> do with V5, as the driver code hasn't changed much.
>>
>> 3. Greg, I would guess that your problems are due to using a very old
>> version of pip. You're at 3.11, but the current version is 23.3. See the
>> wiki article *Troubleshooting pip installs
>> <https://github.com/weewx/weewx/wiki/pip-troubleshooting>* and see if
>> that helps.
>>
>> -tk
>>
>> On Fri, Dec 22, 2023 at 12:45 AM Greg  wrote:
>>
>>> I installed weewx using the pip method and followed the instructions.
>>> I did a pip list before installing to get the list of what other
>>> dependencies I am using.
>>>  When I installed paho-mqtt and pyephem I got these errors:
>>>
>>> Installing collected packages: paho-mqtt
>>>   DEPRECATION: paho-mqtt is being installed using the legacy 'setup.py
>>> install' method, because it does not have a 'pyproject.toml' and the
>>> 'wheel' package is not installed. pip 23.1 will enforce this behaviour
>>> change. A possible replacement is to enable the '--use-pep517' option.
>>> Discussion can be found at https://github.com/pypa/pip/issues/8559
>>>   Running setup.py install for paho-mqtt ... done
>>> Successfully installed paho-mqtt-1.6.1
>>>
>>> /opt/weewx$ python3 -m pip install pyephem
>>> Collecting pyephem
>>>   Using cached pyephem-9.99.tar.gz (1.4 kB)
>>>   Preparing metadata (setup.py) ... done
>>> Requirement already satisfied: ephem in
>>> ./weewx-venv/lib/python3.11/site-packages (from pyephem) (4.1.5)
>>> Installing collected packages: pyephem
>>>   DEPRECATION: pyephem is being installed using the legacy 'setup.py
>>> install' method, because it does not have a 'pyproject.toml' and the
>>> 'wheel' package is not installed. pip 23.1 will enforce this behaviour
>>> change. A possible replacement is to enable the '--use-pep517' option.
>>> Discussion can be found at https://github.com/pypa/pip/issues/8559
>>>
>>> I am currently running it in simulation mode just to see what happens.
>>> On my real system I have installed all my environment and data under /opt
>>> eg /opt/weewx/weewx-venv and /opt/weewx/weewx-data
>>>
>>> Other than that it seems to run.
>>>
>>> My question is should I run the install of those packages above that had
>>> the error with the --use-pep517 option? I read the github information
>>> that was on the link contained in the error message but it made no sense to
>>> me.
>>>
>>> My pip version is: 3.11
>>>
>>>
>>> On Friday 22 December 2023 at 17:05:49 UTC+11 Hartmut Schweidler wrote:
>>>
>>>> Guten Morgen,
>>>>
>>>> Zunächst einmal vielen Dank an alle, die an der Entwicklung von weewx
>>>> V5 beteiligt waren!
>>>>
>>>> Mein Installation erfolgte per "apt upgrade". auf einem Banana Pi + 1TB
>>>> HDD
>>>>
>>>> Die folgenden Pakete werden aktualisiert (Upgrade):
>>>>   weewx
>>>> 1 aktualisiert, 0 neu installiert, 0 zu entfernen und 2 nicht
>>>> aktualisiert.
>>>> Es müssen 2.278

Re: [weewx-development] Re: PostgreSQL support, which branch to target?

[Tom Keffer]

Typically, this is used to calculate the number of days of precipitation
greater than some amount during a time span, using the daily summaries. The
parameter %(val)s is the value.

In retrospect, the choice of column names in the daily summaries may not
have been a good one. The values "sum" in the expression "SUM(sum>90)"
represent two different things. The first sum ("SUM") is an aggregation,
the second sum ("sum") is a column name.

Perhaps you can escape the inner "sum"?

-tk

On Thu, Dec 21, 2023 at 10:12 PM Raoul Snyman 
wrote:

> I've been going through the queries in xtypes.py, and I've come across
> some queries that don't make sense to me, and definitely throw errors in
> PostgreSQL.
>
> For example:
>
> SELECT SUM(sum >= %(val)s) FROM %(table_name)s_day_%(obs_key)s WHERE
> dateTime >= %(start)s AND dateTime < %(stop)s
>
> SUM(sum >= 90) is not valid in PostgreSQL. Can you tell me what this is
> supposed to be doing, then perhaps I can translate the SQL into something
> that's more generic?
>
> Thanks!
>
> On Tuesday, December 12, 2023 at 3:27:19 PM UTC Raoul Snyman wrote:
>
>> On Tuesday, December 12, 2023 at 2:15:31 PM UTC matthew wall wrote:
>>
>> On Monday, December 11, 2023 at 11:47:04 PM UTC-5 Raoul Snyman wrote:
>>
>> I still don't have the tests running properly, I couldn't find any docs
>> on exactly how to set up the tests, and both the MySQL and PostgreSQL tests
>> fail when I just run "make test"
>>
>>
>> i am working on the unit tests.  when we started them we just used python
>> directly. i am converting all of it to pytest.
>>
>> i would like to have the pytest conversion done when v5 comes out of
>> beta, but it might happen soon after that.
>>
>> the core in v5 is pretty solid, but we're still taking care of some
>> packaging and integration bits
>>
>>
>> Oh cool, that's good to know. I've used pytest fairly extensively, it's a
>> good test framework.
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/b298f3e6-ead9-4c5a-8458-d204cb0ac03dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEByGhvKrOfbPw5M7%3D3FnHEaVv29F5xiVZ_UojV0Ekq6Ew%40mail.gmail.com.

Re: [weewx-development] V5.0 release candidate available

[Tom Keffer]

Thanks, all! Keep them coming.

1. The syslog comment is annoying, but harmless. We've changed the weewx
unit service file to specify StandardError of journal+console. Commit
940eff4

.

2. Missing Vantage records. I've noticed variants of this problem when it
takes a long time to download records from the console. My best is that
while the console is busy emitting historic records, it neglects to create
new ones. Hence, you miss a few records. I don't think it has anything to
do with V5, as the driver code hasn't changed much.

3. Greg, I would guess that your problems are due to using a very old
version of pip. You're at 3.11, but the current version is 23.3. See the
wiki article *Troubleshooting pip installs
* and see if that
helps.

-tk

On Fri, Dec 22, 2023 at 12:45 AM Greg  wrote:

> I installed weewx using the pip method and followed the instructions.
> I did a pip list before installing to get the list of what other
> dependencies I am using.
>  When I installed paho-mqtt and pyephem I got these errors:
>
> Installing collected packages: paho-mqtt
>   DEPRECATION: paho-mqtt is being installed using the legacy 'setup.py
> install' method, because it does not have a 'pyproject.toml' and the
> 'wheel' package is not installed. pip 23.1 will enforce this behaviour
> change. A possible replacement is to enable the '--use-pep517' option.
> Discussion can be found at https://github.com/pypa/pip/issues/8559
>   Running setup.py install for paho-mqtt ... done
> Successfully installed paho-mqtt-1.6.1
>
> /opt/weewx$ python3 -m pip install pyephem
> Collecting pyephem
>   Using cached pyephem-9.99.tar.gz (1.4 kB)
>   Preparing metadata (setup.py) ... done
> Requirement already satisfied: ephem in
> ./weewx-venv/lib/python3.11/site-packages (from pyephem) (4.1.5)
> Installing collected packages: pyephem
>   DEPRECATION: pyephem is being installed using the legacy 'setup.py
> install' method, because it does not have a 'pyproject.toml' and the
> 'wheel' package is not installed. pip 23.1 will enforce this behaviour
> change. A possible replacement is to enable the '--use-pep517' option.
> Discussion can be found at https://github.com/pypa/pip/issues/8559
>
> I am currently running it in simulation mode just to see what happens.
> On my real system I have installed all my environment and data under /opt
> eg /opt/weewx/weewx-venv and /opt/weewx/weewx-data
>
> Other than that it seems to run.
>
> My question is should I run the install of those packages above that had
> the error with the --use-pep517 option? I read the github information
> that was on the link contained in the error message but it made no sense to
> me.
>
> My pip version is: 3.11
>
>
> On Friday 22 December 2023 at 17:05:49 UTC+11 Hartmut Schweidler wrote:
>
>> Guten Morgen,
>>
>> Zunächst einmal vielen Dank an alle, die an der Entwicklung von weewx V5
>> beteiligt waren!
>>
>> Mein Installation erfolgte per "apt upgrade". auf einem Banana Pi + 1TB
>> HDD
>>
>> Die folgenden Pakete werden aktualisiert (Upgrade):
>>   weewx
>> 1 aktualisiert, 0 neu installiert, 0 zu entfernen und 2 nicht
>> aktualisiert.
>> Es müssen 2.278 kB an Archiven heruntergeladen werden.
>> Nach dieser Operation werden 2.048 B Plattenplatz zusätzlich benutzt.
>> Holen:1 https://weewx.com/apt-test/python3 buster/main all weewx all
>> 5.0.0rc1-1 [2.278 kB]
>> Es wurden 2.278 kB in 2 s geholt (948 kB/s).
>> Preconfiguring packages ...
>> (Lese Datenbank ... 75361 Dateien und Verzeichnisse sind derzeit
>> installiert.)
>> Vorbereitung zum Entpacken von .../weewx_5.0.0rc1-1_all.deb ...
>> Entpacken von weewx (5.0.0rc1-1) über (5.0.0b17-4) ...
>> weewx (5.0.0rc1-1) wird eingerichtet ...
>> Neue Version der Konfigurationsdatei /etc/weewx/skins/Ftp/skin.conf wird
>> installiert ...
>> Neue Version der Konfigurationsdatei /etc/weewx/skins/Mobile/skin.conf
>> wird installiert ...
>> Neue Version der Konfigurationsdatei /etc/weewx/skins/Rsync/skin.conf
>> wird installiert ...
>>
>> Konfigurationsdatei »/etc/weewx/skins/Seasons/skin.conf«
>>  ==> Geändert (von Ihnen oder von einem Skript) seit der Installation.
>>  ==> Paketverteiler hat eine aktualisierte Version herausgegeben.
>>Wie möchten Sie vorgehen? Ihre Wahlmöglichkeiten sind:
>> Y oder I : Die Version des Paket-Betreuers installieren
>> N oder O : Die momentan installierte Version beibehalten
>>D : Die Unterschiede zwischen den Versionen anzeigen
>>Z : Eine Shell starten, um die Situation zu begutachten
>>  Der Standardweg ist das Beibehalten der momentanen Version.
>> *** skin.conf (Y/I/N/O/D/Z) [Vorgabe=N] ? n
>> ...
>> Neue Version der Konfigurationsdatei /etc/weewx/weewx.conf.dist wird
>> installiert ...
>> Copying previous config file to /etc/weewx/weewx.conf-5.0.0b18-5.0.0rc1
>> Saving distribution config file as /etc/weewx/weewx.conf-5.0.0rc1
>> 

[weewx-development] V5.0 release candidate available

[Tom Keffer]

Celebrate the solstice (coming up in 4 hours) and the start of Winter with
a WeeWX release candidate!

*Changes since b17*
We gave up on a standalone logger and have gone back to whatever your
system uses. This is more robust to permission problems, but may make it
harder to get the logs back out. See the wiki article *View logs
* for tips on how to
convince your system to let you have a look.

As always, you can add a [Logging] section to weewx.conf and set up your
own rotating log system. See the article *WeeWX V4 and logging
* for how to do
this.

*Pip*

For pip installs, please delete your old virtual environment, then install
from scratch by following the pip install instructions
.
While upgrading should work, we are particularly interested in the
experience of a new install, including setting up a daemon and udev files.
Make sure to follow the new instructions that use a daemon setup script.


*Debian*

For Debian package installs, modify /etc/apt/sources.list as follows:

*echo "deb [arch=all] https://weewx.com/apt-test/python3
 buster main" | sudo tee
/etc/apt/sources.list.d/weewx.list*

Note the "apt-test". This tells apt to look there for the beta release,
instead of the normal repository. You may want to change it back when
you're done.


*Red Hat*

For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo

*[weewx]
name=weewx
baseurl=http://weewx.com/yum-test/weewx/el9

enabled=1*

Note the "yum-test". This tells yum to look there for the beta release,
instead of the normal repository. Again, you may want to change things back
when you're done.


*What we're looking for*

1. We are interested in your upgrade experience. Did the installers get
your configuration file right? Skins? Is the daemon configured correctly?

2. With this release, we have included udev rules to set the correct
permissions for devices. Did it work? If not, did unplugging then
replugging the device work?

3. How about logging? Is it going to your system logger? Do the labels look
reasonable?

*Thanks!*

-Tom & Matt

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECFSEWf9immM3fFroWU79GcYyT65ibe8w1hWd-XJ92s1g%40mail.gmail.com.

Re: [weewx-development] No archive db updates but loop works - WeeWx 5.0b15 and b17

[Tom Keffer]

Quick! Buy a lottery ticket! That almost never works.!

On Sat, Dec 16, 2023 at 5:59 PM Chris Alemany  wrote:

> Thanks for this Tom.
> Thankfully the first option, unplugging/de-batterying the Vantage Console
> for 2 minutes to reset it worked.  It doesn’t appear that any information
> was lost.
> Thanks all.
>
> Cheers
> Chris
>
> On Dec 16, 2023, at 16:37, Tom Keffer  wrote:
>
> Chris, you have a classic case of corrupted memory in your logger
> <https://github.com/weewx/weewx/wiki/Troubleshooting-the-Davis-Vantage-station#weewx-generates-html-pages-but-it-does-not-update-them>
> .
>
> On Sat, Dec 16, 2023 at 1:52 PM Chris Alemany  wrote:
>
>> I’m noticing this in the log, is this normal. It seems to me it is
>> talking to the Vantage, knows it has records it needs to get, but tries
>> twice and then gives up. Is that “empty record” a problem?
>>
>> "Dec 16 13:42:16 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Getting archive packets since 2023-12-15 16:10:00 PST (1702685400)
>> Dec 16 13:42:18 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Successfully woke up Vantage console
>> Dec 16 13:42:19 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Retrieving 224 page(s); starting index= 0
>> Dec 16 13:42:19 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Empty record page 0; index 0
>> Dec 16 13:42:19 footiMac weewxd[1741196]: INFO weewx.engine: Starting
>> main packet loop.
>> Dec 16 13:42:21 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Successfully woke up Vantage console
>> Dec 16 13:42:21 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Requesting 200 LOOP packets.
>> Dec 16 13:42:23 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
>> Successfully woke up Vantage console”
>>
>>
>>
>> On Dec 16, 2023, at 13:19, Vince Skahan  wrote:
>>
>> Not really, but your rsync and purple air stuff querying and uploading
>> every two seconds  looks very very unusual.  I'd certainly start there.
>>
>> On Saturday, December 16, 2023 at 1:17:21 PM UTC-8 Chris Alemany wrote:
>>
>>> Do you also have a problem with people using wunderground rapid fire?
>>> I’ve been doing this for years.
>>> Thanks for the advice. I’ll figure it out.
>>>
>>>
>>>
>>> On Dec 16, 2023, at 12:58, Vince Skahan  wrote:
>>>
>>> My guess is you are far doing too much far too often.  Are you really
>>> querying purpleair and also rsync'ing somewhere every TWO seconds ?  That's
>>> a bit much.
>>>
>>> Disable everything you added to vanilla core weewx (temporarily) and run
>>> just unaltered weewx and see if it works.  Then turn your
>>> additions/customizations back on one-by-one and see what is breaking things.
>>>
>>> On Saturday, December 16, 2023 at 12:13:32 PM UTC-8 Chris Alemany wrote:
>>>
>>>> I was basically asking if there is anything more than debug=1 that I
>>>> could enable.
>>>> There is nothing in the debug messages that I’ve been able to detect
>>>> other than silence. The archive should be updated around Dec 16 12:05:21
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/4f9af1bf-0702-4751-821d-61a105d38518n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/4f9af1bf-0702-4751-821d-61a105d38518n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-development+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/c873934b-4206-46db-854e-3ed96f35bdeen%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-development/c873934b-4206-46db-854e-3ed96f35bdeen%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group

Re: [weewx-development] No archive db updates but loop works - WeeWx 5.0b15 and b17

[Tom Keffer]

Chris, you have a classic case of corrupted memory in your logger

.

On Sat, Dec 16, 2023 at 1:52 PM Chris Alemany  wrote:

> I’m noticing this in the log, is this normal. It seems to me it is talking
> to the Vantage, knows it has records it needs to get, but tries twice and
> then gives up. Is that “empty record” a problem?
>
> "Dec 16 13:42:16 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Getting archive packets since 2023-12-15 16:10:00 PST (1702685400)
>
> Dec 16 13:42:18 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Successfully woke up Vantage console
>
> Dec 16 13:42:19 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Retrieving 224 page(s); starting index= 0
>
> Dec 16 13:42:19 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Empty record page 0; index 0
>
> Dec 16 13:42:19 footiMac weewxd[1741196]: INFO weewx.engine: Starting main
> packet loop.
>
> Dec 16 13:42:21 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Successfully woke up Vantage console
>
> Dec 16 13:42:21 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Requesting 200 LOOP packets.
>
> Dec 16 13:42:23 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage:
> Successfully woke up Vantage console”
>
>
>
>
> On Dec 16, 2023, at 13:19, Vince Skahan  wrote:
>
> Not really, but your rsync and purple air stuff querying and uploading
> every two seconds  looks very very unusual.  I'd certainly start there.
>
> On Saturday, December 16, 2023 at 1:17:21 PM UTC-8 Chris Alemany wrote:
>
>> Do you also have a problem with people using wunderground rapid fire?
>> I’ve been doing this for years.
>> Thanks for the advice. I’ll figure it out.
>>
>>
>>
>> On Dec 16, 2023, at 12:58, Vince Skahan  wrote:
>>
>> My guess is you are far doing too much far too often.  Are you really
>> querying purpleair and also rsync'ing somewhere every TWO seconds ?  That's
>> a bit much.
>>
>> Disable everything you added to vanilla core weewx (temporarily) and run
>> just unaltered weewx and see if it works.  Then turn your
>> additions/customizations back on one-by-one and see what is breaking things.
>>
>> On Saturday, December 16, 2023 at 12:13:32 PM UTC-8 Chris Alemany wrote:
>>
>>> I was basically asking if there is anything more than debug=1 that I
>>> could enable.
>>> There is nothing in the debug messages that I’ve been able to detect
>>> other than silence. The archive should be updated around Dec 16 12:05:21
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-developm...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/4f9af1bf-0702-4751-821d-61a105d38518n%40googlegroups.com
>> 
>> .
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/c873934b-4206-46db-854e-3ed96f35bdeen%40googlegroups.com
> 
> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/68E3BA4F-DB82-4F56-88EF-4C5DE78DADAF%40gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEA6QnL7-k6eVppipo_Evo4QwdRTz99O55CT6xJr1RP1fQ%40mail.gmail.com.

Re: [weewx-development] Re: V5.0.0b17 available

[Tom Keffer]

Fixed in commit 4bf9ccd
<https://github.com/weewx/weewx/commit/4bf9ccd2ea3e025398eedf27ab723b6d76ad4b0f>
 .

On Sat, Dec 16, 2023 at 3:47 PM Tom Keffer  wrote:

> No, Paul, it's actually a bug. The variable UTIL_ROOT should have been
> expanded.
>
> -tk
>
> On Sat, Dec 16, 2023 at 12:10 PM pa...@pauland.net 
> wrote:
>
>> Good sleuthing Vince !
>> Running Debian 12 on a test LXD Container. System was already set and
>> running was only testing setup-daemon.systemd
>> Verified  setup-daemon.systemd works properly on a physical machine
>> running Debian 12. Attribute the error to  ENV variables and their handling
>> in the LXD container.
>> So sorry for raising the alarm bell. If I think something seems weird,
>> and it's in a LXD test container I always verify on real hardware... except
>> for today.
>>
>> Paul
>> On Saturday, December 16, 2023 at 1:22:52 PM UTC-5 Vince Skahan wrote:
>>
>>> This worked for me - deb12 in a vagrant vm running as user 'vagrant'.
>>>
>>> I followed the docs, just added --no-prompt to the station setup to not
>>> have it ask questions
>>>
>>> sudo apt update
>>> sudo apt install python3-pip -y
>>> sudo apt install python3-venv -y
>>> python3 -m venv ~/weewx-venv
>>> source ~/weewx-venv/bin/activate
>>> python3 -m pip install weewx
>>> source ~/weewx-venv/bin/activate
>>> weectl station create --no-prompt
>>> sudo sh ~/weewx-data/scripts/setup-daemon.systemd
>>> sudo systemctl start weewx
>>> sudo systemctl status weewx
>>>
>>> What precise os are you running ?
>>> Are you trying to do it in docker perhaps ?
>>>
>>> We'd need to see your exact steps start to finish please.
>>>
>>>
>>> On Saturday, December 16, 2023 at 7:35:11 AM UTC-8 Paul R Anderson wrote:
>>>
>>> Sorry not sure what I'm doing wrong.
>>>
>>> panders@cont-weewx-git:~$ sudo sh
>>> /home/panders/weewx-data/scripts/setup-daemon.systemd
>>>
>>> Cannot find utility files at location '/root/weewx-data/util'
>>> panders@cont-weewx-git:~$
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-development+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/f0bc9c46-7965-4e59-b46d-4d68f8bdad44n%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-development/f0bc9c46-7965-4e59-b46d-4d68f8bdad44n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAEk1oi3oT8JZQhk4iBXNx8_ZwDVkGG5qjam%2B8-rXyyYQ%40mail.gmail.com.

Re: [weewx-development] Re: V5.0.0b17 available

[Tom Keffer]

No, Paul, it's actually a bug. The variable UTIL_ROOT should have been
expanded.

-tk

On Sat, Dec 16, 2023 at 12:10 PM pa...@pauland.net  wrote:

> Good sleuthing Vince !
> Running Debian 12 on a test LXD Container. System was already set and
> running was only testing setup-daemon.systemd
> Verified  setup-daemon.systemd works properly on a physical machine
> running Debian 12. Attribute the error to  ENV variables and their handling
> in the LXD container.
> So sorry for raising the alarm bell. If I think something seems weird, and
> it's in a LXD test container I always verify on real hardware... except for
> today.
>
> Paul
> On Saturday, December 16, 2023 at 1:22:52 PM UTC-5 Vince Skahan wrote:
>
>> This worked for me - deb12 in a vagrant vm running as user 'vagrant'.
>>
>> I followed the docs, just added --no-prompt to the station setup to not
>> have it ask questions
>>
>> sudo apt update
>> sudo apt install python3-pip -y
>> sudo apt install python3-venv -y
>> python3 -m venv ~/weewx-venv
>> source ~/weewx-venv/bin/activate
>> python3 -m pip install weewx
>> source ~/weewx-venv/bin/activate
>> weectl station create --no-prompt
>> sudo sh ~/weewx-data/scripts/setup-daemon.systemd
>> sudo systemctl start weewx
>> sudo systemctl status weewx
>>
>> What precise os are you running ?
>> Are you trying to do it in docker perhaps ?
>>
>> We'd need to see your exact steps start to finish please.
>>
>>
>> On Saturday, December 16, 2023 at 7:35:11 AM UTC-8 Paul R Anderson wrote:
>>
>> Sorry not sure what I'm doing wrong.
>>
>> panders@cont-weewx-git:~$ sudo sh
>> /home/panders/weewx-data/scripts/setup-daemon.systemd
>>
>> Cannot find utility files at location '/root/weewx-data/util'
>> panders@cont-weewx-git:~$
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/f0bc9c46-7965-4e59-b46d-4d68f8bdad44n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEB9-EqQp166naHR%2Bi8%3DU_kXmzW%2Bnwgvabry0%2BA%2Bijbe3Q%40mail.gmail.com.

Re: [weewx-development] Re: V5.0.0b17 available

[Tom Keffer]

Did you run the version of setup-daemon.systemd in ~/weewx-data/scripts? It
should have expanded $HOME to the location of the location of weewx-data.

On Sat, Dec 16, 2023 at 5:58 AM pa...@pauland.net  wrote:

> V5.0.0b17 pip install
> setup-daemon.systemd fails when run with sudo with this error
> Cannot find utility files at location '/root/weewx-data/util'
> Because when run with sudo
> UTIL_ROOT=$HOME/weewx-data/util
> Becomes
> UTIL_ROOT=/root/weewx-data/util
>
> Paul
> On Thursday, December 14, 2023 at 5:52:51 PM UTC-5 Tom Keffer wrote:
>
>> Hopefully, we are getting very close to a final release for V5.0!
>>
>> *Warning!*
>> But, until then, this is still very much a beta release. You could break
>> your installation, and it might take skills to recover. If you're using pip
>> in a virtual environment, the risk is low of breaking other things, but you
>> will have to shut down your production instance temporarily. For a package
>> installer, try it in a VM, or on another computer, first.
>>
>> *Pip*
>>
>> For pip installs, please delete your old virtual environment, then
>> install from scratch by following the new pip install instructions
>> <https://weewx.com/docs/5.0/quickstarts/pip/#install-in-a-virtual-environment>.
>> While upgrading should work, we are particularly interested in the
>> experience of a new install, including setting up a daemon and udev files.
>> Make sure to follow the new instructions that use a daemon setup script.
>>
>>
>> *Debian*
>>
>> For Debian package installs, modify /etc/apt/sources.list as follows:
>>
>> *echo "deb [arch=all] https://weewx.com/apt-test/python3 
>> <https://weewx.com/apt-test/python3> buster main" | sudo tee 
>> /etc/apt/sources.list.d/weewx.list*
>>
>> Note the "apt-test". This tells apt to look there for the beta release,
>> instead of the normal repository. You may want to change it back when
>> you're done.
>>
>>
>> *Red Hat*
>>
>> For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo
>>
>> *[weewx]
>> name=weewx
>> baseurl=http://weewx.com/yum-test/weewx/el9 
>> <http://weewx.com/yum-test/weewx/el9>
>> enabled=1*
>>
>> Note the "yum-test". This tells yum to look there for the beta release,
>> instead of the normal repository. Again, you may want to change things back
>> when you're done.
>>
>>
>> *What we're looking for*
>>
>> 1. We are interested in your upgrade experience. Did the installers get
>> your configuration file right? Skins? Is the daemon configured correctly?
>>
>> 2. With this release, we have included udev rules to set the correct
>> permissions for devices. Did it work? If not, did unplugging then
>> replugging the device work?
>>
>> 3. How about logging? Version 5 will log to a dedicated weewx file,
>> generally /var/log/weewx/weewx.log. Is that what you're seeing?
>>
>>
>> *Thanks!*
>>
>> -Tom & Matt
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/9cdd016b-ecba-4ea6-9b08-45506f7d6cafn%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/9cdd016b-ecba-4ea6-9b08-45506f7d6cafn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECxEt1RR2QuXK%2B9dxvtc5cCLe3m8V1DKR81P4Kmujrj6Q%40mail.gmail.com.

Re: [weewx-development] Google Groups Ending Support for USENET

[Tom Keffer]

Neither weewx-user, nor weewx-development, are Usenet groups, and never
have been. We are unaffected by this change.

Kind of sad what's happened to Usenet. The feed comp.lang.c++ was a big
part of my life in the 1980s and early 90s. Now it's a cesspool of spam.

On Fri, Dec 15, 2023 at 12:22 PM Chuck Rhode 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> > Starting on February 22, 2024, you can no longer use Google Groups
> > (at groups.google.com) to post content to Usenet groups, subscribe
> > to Usenet groups, or view new Usenet content.
>
> > Most of the current Google Groups content is not Usenet content and
> > will not be affected.
>
> +
> https://support.google.com/groups/answer/11036538?visit_id=638382672714230642-1699016088=usenet=1
>
> ... so, I am posting this by eMail (from GMail) to
> weewx-development@googlegroups.com.  I don't suppose weewx-development
> is a USENET group, but...
>
> 1. Will this continue to work after Feb 2024?
>
> 2. May I continue to subscribe to weewx-development by eMail?
>
> ... because — if I have to login to groups.google.com and supply
> credentials every day to read the weewx-development group — I probably
> won't.  This is not meant as a threat.  I'm jus' sayin'.
>
> - --
> .. Be Seeing You,
> .. Chuck Rhode, Sheboygan, WI, USA
> .. Weather:  http://LacusVeris.com/WX
> .. 54° — Wind S 5 mph
>
> -BEGIN PGP SIGNATURE-
>
> iF0EARECAB0WIQT+MY/5I/LMPSswTbVg2/xipKOWUgUCZXy1bwAKCRBg2/xipKOW
> UpmYAJ4/kw77QFQioVkH1lan1ZsKMLjCHwCfRI+x9BKhEuqgNJGugeN4vnoliwQ=
> =ujhO
> -END PGP SIGNATURE-
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/20231215142207.1f73016b%40BigTimber.LacusVeris.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBXQqbMof3%3DK0kyvg9nn02tYOP-vfZd%3DRwA%3Dgob2-Oktg%40mail.gmail.com.

[weewx-development] V5.0.0b17 available

[Tom Keffer]

Hopefully, we are getting very close to a final release for V5.0!

*Warning!*
But, until then, this is still very much a beta release. You could break
your installation, and it might take skills to recover. If you're using pip
in a virtual environment, the risk is low of breaking other things, but you
will have to shut down your production instance temporarily. For a package
installer, try it in a VM, or on another computer, first.

*Pip*

For pip installs, please delete your old virtual environment, then install
from scratch by following the new pip install instructions
.
While upgrading should work, we are particularly interested in the
experience of a new install, including setting up a daemon and udev files.
Make sure to follow the new instructions that use a daemon setup script.


*Debian*

For Debian package installs, modify /etc/apt/sources.list as follows:

*echo "deb [arch=all] https://weewx.com/apt-test/python3
 buster main" | sudo tee
/etc/apt/sources.list.d/weewx.list*

Note the "apt-test". This tells apt to look there for the beta release,
instead of the normal repository. You may want to change it back when
you're done.


*Red Hat*

For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo

*[weewx]
name=weewx
baseurl=http://weewx.com/yum-test/weewx/el9

enabled=1*

Note the "yum-test". This tells yum to look there for the beta release,
instead of the normal repository. Again, you may want to change things back
when you're done.


*What we're looking for*

1. We are interested in your upgrade experience. Did the installers get
your configuration file right? Skins? Is the daemon configured correctly?

2. With this release, we have included udev rules to set the correct
permissions for devices. Did it work? If not, did unplugging then
replugging the device work?

3. How about logging? Version 5 will log to a dedicated weewx file,
generally /var/log/weewx/weewx.log. Is that what you're seeing?


*Thanks!*

-Tom & Matt

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAPKxz_NKXEAoJgySTYnJovEj_AL4xWo1JrSWFEOyixiQ%40mail.gmail.com.

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

No. Version 5.0.0b17 never changes. Version 5.0.0b18 will have any new
changes incorporated into it.

*However*, the code repository always has the latest code in it. It just
hasn't been organized into a formal release.

Does that help?

On Thu, Dec 14, 2023 at 2:17 PM Greg  wrote:

> I guess I am asking is the commit a change that is in the version 17? But
> I think what you are saying is all the commits are done and then you
> release the next version with all the commits in it?
> I am not a programmer so I was wondering how it works.
>
> Thanks
>
> On Friday 15 December 2023 at 09:12:53 UTC+11 Tom Keffer wrote:
>
>> Not sure what you're asking, but try this. The current version is always
>> available as the "development" branch of the repository.
>>
>> On PyPi, once a release has been submitted, it cannot be changed. So, new
>> releases have the version number bumped.
>>
>> Is that what you were asking?
>>
>> On Thu, Dec 14, 2023 at 12:55 PM Greg  wrote:
>>
>>> Does a commit mean if I download it again the fix is in there or do I
>>> wait until the version number increases?
>>>
>>>
>>> On Thursday 14 December 2023 at 11:32:12 UTC+11 Tom Keffer wrote:
>>>
>>>> Changed in commit 4b969ce
>>>> <https://github.com/weewx/weewx/commit/4b969cedd35471972305879d659ac0ae95a42c31>
>>>> .
>>>>
>>>> For TypeError involving None, the error gets logged if debug>1, then
>>>> the program presses on. Any other kind of error results in program
>>>> termination.
>>>>
>>>> This applies *only* to expressions used in StdCalibrate.
>>>>
>>>> -tk
>>>>
>>>> On Wed, Dec 13, 2023 at 3:46 PM Tom Keffer  wrote:
>>>>
>>>>> The error was always there, it's just that previous versions swallowed
>>>>> it.
>>>>>
>>>>> We should probably continue to ignore 'None' type errors. It's
>>>>> difficult for a newbie to write an expression that can handle them.
>>>>>
>>>>> On Wed, Dec 13, 2023 at 3:23 PM Greg  wrote:
>>>>>
>>>>>> I upgraded from 5.0.0.b16 to 17 and got this error:
>>>>>>
>>>>>> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine:
>>>>>> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' 
>>>>>> and
>>>>>> 'int'
>>>>>>
>>>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: binding to loop
>>>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: topic is
>>>>>> jedwood/raw/weather
>>>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: desired unit system is
>>>>>> METRIC
>>>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: data will be uploaded
>>>>>> to mqtt://192.168.1.164:1883/
>>>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: 'pyephem'
>>>>>> detected, extended almanac data is available
>>>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO __main__: Starting up
>>>>>> weewx version 5.0.0b17
>>>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: Using
>>>>>> binding 'wx_binding' to database 'weewx'
>>>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Starting
>>>>>> backfill of daily summaries
>>>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Daily
>>>>>> summaries up to date
>>>>>> Dec 14 10:15:22 moonbi weewxd[491310]: INFO weewx.drivers.fousb:
>>>>>> synchronising to the weather station (quality=0)
>>>>>>
>>>>>>
>>>>>> Dec 14 10:15:52 moonbi weewxd[491310]: INFO weewx.engine: Starting
>>>>>> main packet loop.
>>>>>> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.drivers.fousb:
>>>>>> station status {'rain_overflow': 0, 'lost_connection': 0, 'unknown': 0} 
>>>>>> (0)
>>>>>> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine:
>>>>>> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' 
>>>>>> and
>>>>>> 'int'
>>>>>> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.restx: MQTT:
>>>>>> Published record 2023-12-14 10:15:56 AEDT (1702509356)
>>>>>> ^[^[Dec 14 10:16:56 moonbi weewxd[4

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

Not sure what you're asking, but try this. The current version is always
available as the "development" branch of the repository.

On PyPi, once a release has been submitted, it cannot be changed. So, new
releases have the version number bumped.

Is that what you were asking?

On Thu, Dec 14, 2023 at 12:55 PM Greg  wrote:

> Does a commit mean if I download it again the fix is in there or do I wait
> until the version number increases?
>
>
> On Thursday 14 December 2023 at 11:32:12 UTC+11 Tom Keffer wrote:
>
>> Changed in commit 4b969ce
>> <https://github.com/weewx/weewx/commit/4b969cedd35471972305879d659ac0ae95a42c31>
>> .
>>
>> For TypeError involving None, the error gets logged if debug>1, then the
>> program presses on. Any other kind of error results in program termination.
>>
>> This applies *only* to expressions used in StdCalibrate.
>>
>> -tk
>>
>> On Wed, Dec 13, 2023 at 3:46 PM Tom Keffer  wrote:
>>
>>> The error was always there, it's just that previous versions swallowed
>>> it.
>>>
>>> We should probably continue to ignore 'None' type errors. It's difficult
>>> for a newbie to write an expression that can handle them.
>>>
>>> On Wed, Dec 13, 2023 at 3:23 PM Greg  wrote:
>>>
>>>> I upgraded from 5.0.0.b16 to 17 and got this error:
>>>>
>>>> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine:
>>>> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and
>>>> 'int'
>>>>
>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: binding to loop
>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: topic is
>>>> jedwood/raw/weather
>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: desired unit system is
>>>> METRIC
>>>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: data will be uploaded to
>>>> mqtt://192.168.1.164:1883/
>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: 'pyephem'
>>>> detected, extended almanac data is available
>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO __main__: Starting up weewx
>>>> version 5.0.0b17
>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: Using binding
>>>> 'wx_binding' to database 'weewx'
>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Starting
>>>> backfill of daily summaries
>>>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Daily
>>>> summaries up to date
>>>> Dec 14 10:15:22 moonbi weewxd[491310]: INFO weewx.drivers.fousb:
>>>> synchronising to the weather station (quality=0)
>>>>
>>>>
>>>> Dec 14 10:15:52 moonbi weewxd[491310]: INFO weewx.engine: Starting main
>>>> packet loop.
>>>> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.drivers.fousb:
>>>> station status {'rain_overflow': 0, 'lost_connection': 0, 'unknown': 0} (0)
>>>> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine:
>>>> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and
>>>> 'int'
>>>> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.restx: MQTT:
>>>> Published record 2023-12-14 10:15:56 AEDT (1702509356)
>>>> ^[^[Dec 14 10:16:56 moonbi weewxd[491310]: ERROR weewx.engine:
>>>> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and
>>>> 'int'
>>>> Dec 14 10:16:56 moonbi weewxd[491310]: INFO weewx.restx: MQTT:
>>>> Published record 2023-12-14 10:16:56 AEDT (1702509416)
>>>>
>>>> I downgraded back to 16 and I didn't get that error.
>>>>
>>>> Python 3.11.2
>>>>
>>>> On Thursday 14 December 2023 at 07:39:29 UTC+11 Tom Keffer wrote:
>>>>
>>>>> Commit da31d16
>>>>> <https://github.com/weewx/weewx/commit/da31d1644314fe19e64be6f4a9d10ced27b85a95>
>>>>> .
>>>>>
>>>>>
>>>>> On Wed, Dec 13, 2023 at 5:22 AM Paul R Anderson 
>>>>> wrote:
>>>>>
>>>>>> That sounds great!
>>>>>> Paul
>>>>>>
>>>>>> On Wed, Dec 13, 2023 at 7:58 AM Tom Keffer  wrote:
>>>>>>
>>>>>>> On Wed, Dec 13, 2023 at 4:47 AM Paul R Anderson 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> That would be better than just deleting it. Wondering if  the new
>>>>>>>

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

Changed in commit 4b969ce
<https://github.com/weewx/weewx/commit/4b969cedd35471972305879d659ac0ae95a42c31>
.

For TypeError involving None, the error gets logged if debug>1, then the
program presses on. Any other kind of error results in program termination.

This applies *only* to expressions used in StdCalibrate.

-tk

On Wed, Dec 13, 2023 at 3:46 PM Tom Keffer  wrote:

> The error was always there, it's just that previous versions swallowed it.
>
> We should probably continue to ignore 'None' type errors. It's difficult
> for a newbie to write an expression that can handle them.
>
> On Wed, Dec 13, 2023 at 3:23 PM Greg  wrote:
>
>> I upgraded from 5.0.0.b16 to 17 and got this error:
>>
>> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration
>> loop error unsupported operand type(s) for /: 'NoneType' and 'int'
>>
>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: binding to loop
>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: topic is
>> jedwood/raw/weather
>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: desired unit system is
>> METRIC
>> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: data will be uploaded to
>> mqtt://192.168.1.164:1883/
>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: 'pyephem'
>> detected, extended almanac data is available
>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO __main__: Starting up weewx
>> version 5.0.0b17
>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: Using binding
>> 'wx_binding' to database 'weewx'
>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Starting
>> backfill of daily summaries
>> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Daily
>> summaries up to date
>> Dec 14 10:15:22 moonbi weewxd[491310]: INFO weewx.drivers.fousb:
>> synchronising to the weather station (quality=0)
>>
>>
>> Dec 14 10:15:52 moonbi weewxd[491310]: INFO weewx.engine: Starting main
>> packet loop.
>> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.drivers.fousb: station
>> status {'rain_overflow': 0, 'lost_connection': 0, 'unknown': 0} (0)
>> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration
>> loop error unsupported operand type(s) for /: 'NoneType' and 'int'
>> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.restx: MQTT: Published
>> record 2023-12-14 10:15:56 AEDT (1702509356)
>> ^[^[Dec 14 10:16:56 moonbi weewxd[491310]: ERROR weewx.engine:
>> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and
>> 'int'
>> Dec 14 10:16:56 moonbi weewxd[491310]: INFO weewx.restx: MQTT: Published
>> record 2023-12-14 10:16:56 AEDT (1702509416)
>>
>> I downgraded back to 16 and I didn't get that error.
>>
>> Python 3.11.2
>>
>> On Thursday 14 December 2023 at 07:39:29 UTC+11 Tom Keffer wrote:
>>
>>> Commit da31d16
>>> <https://github.com/weewx/weewx/commit/da31d1644314fe19e64be6f4a9d10ced27b85a95>
>>> .
>>>
>>>
>>> On Wed, Dec 13, 2023 at 5:22 AM Paul R Anderson 
>>> wrote:
>>>
>>>> That sounds great!
>>>> Paul
>>>>
>>>> On Wed, Dec 13, 2023 at 7:58 AM Tom Keffer  wrote:
>>>>
>>>>> On Wed, Dec 13, 2023 at 4:47 AM Paul R Anderson 
>>>>> wrote:
>>>>>
>>>>>> That would be better than just deleting it. Wondering if  the new
>>>>>> 'scripts' subdirectory could be located at ~/weewx-data/util/scripts ?
>>>>>>
>>>>>
>>>>> That's something I thought of, but then it gets buried amongst some
>>>>> unrelated subdirectories, making them harder to find.
>>>>>
>>>>> How about this: just upgrade the scripts within the subdirectory
>>>>> 'scripts'? Then they can peacefully coexist with your scripts.
>>>>>
>>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-development+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/fbbcd4d1-ca19-40fe-9b39-aa418bc0e2c2n%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-development/fbbcd4d1-ca19-40fe-9b39-aa418bc0e2c2n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEARyO25baL_%2BwyqMQA8MWWZENFGFWjLXEaPBXXTwJQPLw%40mail.gmail.com.

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

The error was always there, it's just that previous versions swallowed it.

We should probably continue to ignore 'None' type errors. It's difficult
for a newbie to write an expression that can handle them.

On Wed, Dec 13, 2023 at 3:23 PM Greg  wrote:

> I upgraded from 5.0.0.b16 to 17 and got this error:
>
> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration
> loop error unsupported operand type(s) for /: 'NoneType' and 'int'
>
> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: binding to loop
> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: topic is jedwood/raw/weather
> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: desired unit system is
> METRIC
> Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: data will be uploaded to
> mqtt://192.168.1.164:1883/
> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: 'pyephem'
> detected, extended almanac data is available
> Dec 14 10:15:21 moonbi weewxd[491310]: INFO __main__: Starting up weewx
> version 5.0.0b17
> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: Using binding
> 'wx_binding' to database 'weewx'
> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Starting
> backfill of daily summaries
> Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Daily summaries
> up to date
> Dec 14 10:15:22 moonbi weewxd[491310]: INFO weewx.drivers.fousb:
> synchronising to the weather station (quality=0)
>
>
> Dec 14 10:15:52 moonbi weewxd[491310]: INFO weewx.engine: Starting main
> packet loop.
> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.drivers.fousb: station
> status {'rain_overflow': 0, 'lost_connection': 0, 'unknown': 0} (0)
> Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration
> loop error unsupported operand type(s) for /: 'NoneType' and 'int'
> Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.restx: MQTT: Published
> record 2023-12-14 10:15:56 AEDT (1702509356)
> ^[^[Dec 14 10:16:56 moonbi weewxd[491310]: ERROR weewx.engine:
> StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and
> 'int'
> Dec 14 10:16:56 moonbi weewxd[491310]: INFO weewx.restx: MQTT: Published
> record 2023-12-14 10:16:56 AEDT (1702509416)
>
> I downgraded back to 16 and I didn't get that error.
>
> Python 3.11.2
>
> On Thursday 14 December 2023 at 07:39:29 UTC+11 Tom Keffer wrote:
>
>> Commit da31d16
>> <https://github.com/weewx/weewx/commit/da31d1644314fe19e64be6f4a9d10ced27b85a95>
>> .
>>
>>
>> On Wed, Dec 13, 2023 at 5:22 AM Paul R Anderson 
>> wrote:
>>
>>> That sounds great!
>>> Paul
>>>
>>> On Wed, Dec 13, 2023 at 7:58 AM Tom Keffer  wrote:
>>>
>>>> On Wed, Dec 13, 2023 at 4:47 AM Paul R Anderson 
>>>> wrote:
>>>>
>>>>> That would be better than just deleting it. Wondering if  the new
>>>>> 'scripts' subdirectory could be located at ~/weewx-data/util/scripts ?
>>>>>
>>>>
>>>> That's something I thought of, but then it gets buried amongst some
>>>> unrelated subdirectories, making them harder to find.
>>>>
>>>> How about this: just upgrade the scripts within the subdirectory
>>>> 'scripts'? Then they can peacefully coexist with your scripts.
>>>>
>>>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/fbbcd4d1-ca19-40fe-9b39-aa418bc0e2c2n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/fbbcd4d1-ca19-40fe-9b39-aa418bc0e2c2n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBkNz_CjjnGiT85O0RUvztZz%3DpSLg1c9YTN%3D9vahPAi%2BA%40mail.gmail.com.

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

Commit da31d16
<https://github.com/weewx/weewx/commit/da31d1644314fe19e64be6f4a9d10ced27b85a95>
.


On Wed, Dec 13, 2023 at 5:22 AM Paul R Anderson  wrote:

> That sounds great!
> Paul
>
> On Wed, Dec 13, 2023 at 7:58 AM Tom Keffer  wrote:
>
>> On Wed, Dec 13, 2023 at 4:47 AM Paul R Anderson  wrote:
>>
>>> That would be better than just deleting it. Wondering if  the new
>>> 'scripts' subdirectory could be located at ~/weewx-data/util/scripts ?
>>>
>>
>> That's something I thought of, but then it gets buried amongst some
>> unrelated subdirectories, making them harder to find.
>>
>> How about this: just upgrade the scripts within the subdirectory
>> 'scripts'? Then they can peacefully coexist with your scripts.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDx224%3DgMu%3DtBO%3DfrBKygRfoh2dWe_VVAGpwb8LtenHLw%40mail.gmail.com.

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

On Wed, Dec 13, 2023 at 4:47 AM Paul R Anderson  wrote:

> That would be better than just deleting it. Wondering if  the new
> 'scripts' subdirectory could be located at ~/weewx-data/util/scripts ?
>

That's something I thought of, but then it gets buried amongst some
unrelated subdirectories, making them harder to find.

How about this: just upgrade the scripts within the subdirectory 'scripts'?
Then they can peacefully coexist with your scripts.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAzdmatDpkYJ5Y65Sc1ZPPpJkCCzZYNQ2LDfzQK-xSLRg%40mail.gmail.com.

Re: [weewx-development] WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[Tom Keffer]

We could timestamp the old 'scripts' subdirectory instead of deleting it.

On Wed, Dec 13, 2023 at 4:29 AM pa...@pauland.net  wrote:

> pip update to 5.0.0b17 deletes ~/weewx-data/scripts if it exist, then
> creates new ~/weewx-data/scripts directory to hold setup-daemon.systemd and
> setup-daemon.mac. On previous V4 setup.py installs and newer V5 pip
> installs I have always created a scripts subdirectory within the WeeWX
> directory to hold custom scripts related to WeeWX. The scripts directory
> and it's subdirectories were always safe when updating to a new version of
> WeeWx. I bet I'm not the only user that normally creates a custom scripts
> directory and will be surprised when their by custom scripts disappear. I
> guess it's resonable to assume WeeWX has controll of all files under
> ~/weewx-data/ and user is at own peril by adding custom files. I was just
> surprised that the existing ~/weewx-data/scripts directory was removed
> without warning.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/2fd8ae9a-fc15-4694-95b6-14f1ee220e31n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDB4dcDv5mUKpigUNgvrY%2BjwrfPM2DD%2BwZ6SpKytS11wg%40mail.gmail.com.

Re: [weewx-development] Re: PostgreSQL support, which branch to target?

[Tom Keffer]

That looks like a good, solid start! Well done.

There are some other places in the code that are database-dependent. In
particular, look at the queries in weewx/xtypes.py, dictionary
"group_defs". Maybe you can figure out a way around that.

I don't think the lookup table you supplied in postgres.py will survive the
test of time. There are just too many types out there in the wild to ever
capture them in a static structure. Whenever you create a table, I think it
will be necessary to save the original names in database metadata, then
return that when needed.

Make sure you don't drift too far away from the V5 code. I don't see any
major refactoring coming up, but I can't guarantee it either.

-tk


On Mon, Dec 11, 2023 at 8:47 PM Raoul Snyman  wrote:

> I just wanted to give a little feedback.
>
> Firstly, I have WeeWX logging stuff into PostgreSQL and generating the
> Season report without issues.
>
> But I did have to change some stuff outside of the DB driver. There
> are a lot of assumptions built into the system, specifically around
> non-standard SQL which MySQL uses and SQLite allows because of MySQL.
>
> I've tried to make things slightly more DB-agnostic, and moved a couple
> things to the base Connection object so that they can be overridden. It's
> not very pretty, but it works.
>
> I still don't have the tests running properly, I couldn't find any docs on
> exactly how to set up the tests, and both the MySQL and PostgreSQL tests
> fail when I just run "make test"
>
> Anyway, here's what I have so far:
> https://github.com/rsnyman/weewx/commit/9bbeaad0d4492092d5c068e076a85dd7f6c42a0f
>
>
> On Monday, December 4, 2023 at 10:41:30 AM UTC Cameron D wrote:
>
>> Thanks, I understand the issue a bit better now.  It looks more like a
>> case of whether the DB system is case-preserving. Sqlite is case-preserving
>> but case-insensitive, whereas pg looks more like case-modifying and
>> case-insensitive (unless identifiers are quoted).  Running the mysql server
>> in case-insensitive mode (1) will cause similar problems to pg.
>>
>> On Monday, 4 December 2023 at 1:07:19 pm UTC+10 Raoul Snyman wrote:
>>
>>> On Monday, December 4, 2023 at 2:20:38 AM UTC Cameron D wrote:
>>>
>>> I don't understand the problem.
>>> My MariaDB server is set to *case-sensitive* (file-system and table
>>> names). The column names are mainly camel case (using a trimmed down
>>> version of the old schema with original names, such as 'inTemp' and 'rain').
>>> I have a second instance with an ecowitt DB where the column names are
>>> mixed camel and snake (I took standard CamelCase name and added identifying
>>> suffixes with an underscore). And, of course, all the archive table names
>>> are snake with possible camel suffixes, matching the original column name.
>>>
>>>
>>> PostgreSQL is NOT case-sensitive. It converts everything to lowercase in
>>> order to bypass any case sensitivity issues. The problem is that WeeWX is
>>> expecting the database to be case-sensitive.
>>>
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/00d99404-07bc-4a26-a75a-6c4de377aba8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECQV9m%3DkLVrEuApy02SeevFyMfGpmqK4g-KPTO9eMVrYA%40mail.gmail.com.

Re: [weewx-development] Question about timing

[Tom Keffer]

If a report thread is still running, a new one is not launched. *However*,
if enough time has elapsed (set by option max_wait, default 10 minutes), a
new thread is launched anyway on the assumption that the old thread is a
zombie thread.

On Sun, Dec 10, 2023 at 7:17 AM Paul Dunphy  wrote:

> If I run " $alltime.humidex.max ($alltime.humidex.max.degree_F)
> $alltime.humidex.maxtime.format($ALLTIMEFMT) once a day it takes ~3
> minutes.  This is not a problem yet.  However, if the database size
> increases to the point where the calculation time exceeds 5 minutes, will
> it skip the next interval, do it a minute or two "late," or cause things to
> blow up?  This is more of a curiosity than anything else, but it seems to
> me someone will run $alltime. that will take more
> than the loop interval to complete.
>
> - Paul VE1DX
>
> On Wednesday, November 29, 2023 at 12:43:18 PM UTC-4 Paul Dunphy wrote:
>
>> I asked about the time in the context of providing feedback for
>> development. If it is something on my end, then it is irrelevant in
>> terms of an issue with v5 . . . it's a new SD card, and the CPU is doing
>> nothing else, so I suspect it is a full database search.
>>
>> One table I generate uses $alltime.humidex.max
>> ($alltime.humidex.max.degree_F)
>> $alltime.humidex.maxtime.format($ALLTIMEFMT) . . . I assume that would
>> have to go through the entire database and calculate the value for each
>> record every time.
>>
>> Table "Lake Echo extremes since 2015" (lower right of the page) uses
>> this.  Maybe I need to just calculate it once a day instead of each
>> interval.  Either way, it's me and not related to the development of
>> v5.  :-)
>>
>> https://ve1dx.net/v5/
>>
>> - Paul VE1DX
>>
>> On 2023-11-29 12:12 p.m., Tom Keffer wrote:
>> > The templates are taking 3 minutes or more to run. That could be
>> > because they are very complex, they use tags that require whole
>> > database searches, your disk is on the verge of failure, or the CPUs
>> > are busy doing something else.
>> >
>> > On Wed, Nov 29, 2023 at 7:33 AM Paul Dunphy  wrote:
>> >
>> > Nov 29 11:13:33 jade python3[362677]: weewx[362677] INFO
>> > user.sftp: sftpgenerator: transferred 27 files in 5.47 seconds
>> > Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO
>> > weewx.manager: Added record 2023-11-29 11:15:16 AST (1701270916)
>> > to database 'mem.sdb'
>> > Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO
>> > weewx.manager: Added record 2023-11-29 11:15:16 AST (1701270916)
>> > to daily summary in 'mem.sdb'
>> > Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO
>> > weewx.manager: Added record 2023-11-29 11:15:00 AST (1701270900)
>> > to database 'weewx.sdb'
>> > Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO
>> > weewx.manager: Added record 2023-11-29 11:15:00 AST (1701270900)
>> > to daily summary in 'weewx.sdb'
>> > Nov 29 11:17:01 jade CRON[539036]: pam_unix(cron:session): session
>> > opened for user root(uid=0) by (uid=0)
>> > Nov 29 11:17:01 jade CRON[539037]: (root) CMD (cd / && run-parts
>> > --report /etc/cron.hourly)
>> > Nov 29 11:17:01 jade CRON[539036]: pam_unix(cron:session): session
>> > closed for user root
>> > Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO
>> > weewx.cheetahgenerator: Generated 13 files for report
>> > StandardReport in 191.18 seconds
>> > Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO
>> > weewx.imagegenerator: Generated 12 images for report
>> > StandardReport in 0.41 seconds
>> > Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO
>> > weewx.reportengine: Copied 0 files to
>> > /home/pdunphy/weewx-data/public_html
>> > Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO
>> > weewx.cheetahgenerator: Generated 1 files for report mem in 0.02
>> > seconds
>> > Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO
>> > weewx.imagegenerator: Generated 1 images for report mem in 0.04
>> > seconds
>> > Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO
>> > weewx.reportengine: Copied 0 files to
>> > /home/pdunphy/weewx-data/public_html/mem
>> > Nov 29 11:18:28 jade python3[362677]: weewx[362677] INFO
>> > paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1)
>> > Nov 29 11:18:28 jade python3[362677]: weewx[362677] INFO
>> > paramiko.transport: Authentication (publickey) successful!
>>

Re: [weewx-development] Re: PostgreSQL support, which branch to target?

[Tom Keffer]

Yeah, I'm not going to do that.

3rd party code still wouldn't work.

On Sun, Dec 3, 2023 at 8:04 AM Joel Bion  wrote:

>
> Sent from my iPhone
>
> On Dec 3, 2023, at 7:32 AM, Tom Keffer  wrote:
>
> I'll be merging the V5 branch into the development branch in the next few
> days, but target the V5 branch for now.
>
> As I recall, the "case sensitivity" problem with Postgres is that in order
> to achieve case-insensitivity, it converts *everything* to lower case,
> including column names.
>
> When WeeWX starts up, it reads the schema from the database itself. So,
> something that started out like "outTemp" becomes "outtemp". Then if some
> code needs the outside temperature to calculate, say, dewpoint, it can't
> find it because Python string comparisons are always case-sensitive.
>
> In the intervening years, perhaps Postgres has added a switch to prevent
> this. Don't know.
>
> To work around it in the WeeWX code would require either that all string
> comparisons involving SQL types become case-insensitive, which would break
> countless 3rd party extensions, or the code that reads the schema from the
> database would have to be changed to read it from metadata, or someplace
> else.
>
> There are ways around the problem.
>
> But, it's a biggish project. I'd be mighty grateful if someone took it on.
>
> -tk
>
>
> Detail: Postgres can accept mixed case table names but it requires always
> quoting them, per
> https://stackoverflow.com/questions/20878932/are-postgresql-column-names-case-sensitive#20880247
>
>
> On Saturday, December 2, 2023 at 7:46:32 PM UTC-8 Raoul Snyman wrote:
>
>> Hello,
>>
>> I just came across WeeWX a week or so ago when I was looking for some
>> open source Linux-based software to pull data from my AcuRite weather
>> station. Fantastic project.
>>
>> I am working on adding support for PostgreSQL (my database of choice),
>> and I wanted to check which branch I should be targeting. I see the V5
>> branch, which seems to be the future of WeeWX, but the docs specify
>> targeting the "development" branch.
>>
>> Thanks!
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/9af547f0-449a-4c79-96b4-2f3559359232n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/9af547f0-449a-4c79-96b4-2f3559359232n%40googlegroups.com?utm_medium=email_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECrX86joANHwc5ybOGr0owZEeaH99LbF%2Bo29rNvEnRROw%40mail.gmail.com.

[weewx-development] Re: PostgreSQL support, which branch to target?

[Tom Keffer]

I'll be merging the V5 branch into the development branch in the next few 
days, but target the V5 branch for now.

As I recall, the "case sensitivity" problem with Postgres is that in order 
to achieve case-insensitivity, it converts *everything* to lower case, 
including column names.

When WeeWX starts up, it reads the schema from the database itself. So, 
something that started out like "outTemp" becomes "outtemp". Then if some 
code needs the outside temperature to calculate, say, dewpoint, it can't 
find it because Python string comparisons are always case-sensitive.

In the intervening years, perhaps Postgres has added a switch to prevent 
this. Don't know.

To work around it in the WeeWX code would require either that all string 
comparisons involving SQL types become case-insensitive, which would break 
countless 3rd party extensions, or the code that reads the schema from the 
database would have to be changed to read it from metadata, or someplace 
else. 

There are ways around the problem.

But, it's a biggish project. I'd be mighty grateful if someone took it on.

-tk

On Saturday, December 2, 2023 at 7:46:32 PM UTC-8 Raoul Snyman wrote:

> Hello,
>
> I just came across WeeWX a week or so ago when I was looking for some open 
> source Linux-based software to pull data from my AcuRite weather station. 
> Fantastic project.
>
> I am working on adding support for PostgreSQL (my database of choice), and 
> I wanted to check which branch I should be targeting. I see the V5 branch, 
> which seems to be the future of WeeWX, but the docs specify targeting the 
> "development" branch.
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/9af547f0-449a-4c79-96b4-2f3559359232n%40googlegroups.com.

Re: [weewx-development] I'm disabling Cheetah's useStackFrames option as it is painfully slow with the Python version of NameMapper

[Tom Keffer]

There's a wiki page about troubleshooting pip, which covers this topic. I
should really link it from the home page.

https://github.com/weewx/weewx/wiki/pip-troubleshooting

On Fri, Dec 1, 2023 at 11:36 AM Vince Skahan  wrote:

> Very reluctant to get into cheetah hell, but I thought I'd mention this
> one just in case it is related to v5 development
>
> I'm seeing a warning in my docker pip builds and runtime logs of the v5
> beta but for some reason I do not see it in a pi5 running the same
> version.  Any idea what's going on ?
>
> Example below is deb12 but deb11 in docker had the same warnings...
>
> Output below is from 'docker compose logs'
>
>
> deb12pip   | weewx[1] INFO weewxd: Initializing weewx version 5.0.0b15
> deb12pip   | weewx[1] INFO weewxd: Using Python 3.11.2 (main, Mar 13
> 2023, 12:18:29) [GCC 12.2.0]
> deb12pip   | weewx[1] INFO weewxd: Located at
> /home/weewx/weewx-venv/bin/python3
> deb12pip   | weewx[1] INFO weewxd: Platform
> Linux-6.4.16-linuxkit-aarch64-with-glibc2.36
> deb12pip   | weewx[1] INFO weewxd: Locale is
> 'LC_CTYPE=C.UTF-8;LC_NUMERIC=C;LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C'
> deb12pip   | weewx[1] INFO weewxd: Entry path:
> /home/weewx/weewx-venv/lib/python3.11/site-packages/weewxd.py
> deb12pip   | weewx[1] INFO weewxd: Using configuration file
> /home/weewx/weewx-data/weewx.conf
> deb12pip   | weewx[1] INFO weewxd: Debug is 1
> deb12pip   | weewx[1] INFO weewx.engine: Loading station type
> Simulator (weewx.drivers.simulator)
> deb12pip   | weewx[1] INFO weewx.engine: StdConvert target unit is 0x1
> deb12pip   | weewx[1] INFO weewx.wxservices: StdWXCalculate will use
> data binding wx_binding
> deb12pip   | weewx[1] INFO weewx.engine: Archive will use data binding
> wx_binding
> deb12pip   | weewx[1] INFO weewx.engine: Record generation will be
> attempted in 'hardware'
> deb12pip   | weewx[1] INFO weewx.engine: Using archive interval of 300
> seconds (specified in weewx configuration)
> deb12pip   | weewx[1] INFO weewx.restx: StationRegistry: Registration
> not requested.
> deb12pip   | weewx[1] INFO weewx.restx: Wunderground: Posting not
> enabled.
> deb12pip   | weewx[1] INFO weewx.restx: PWSweather: Posting not
> enabled.
> deb12pip   | weewx[1] INFO weewx.restx: CWOP: Posting not enabled.
> deb12pip   | weewx[1] INFO weewx.restx: WOW: Posting not enabled.
> deb12pip   | weewx[1] INFO weewx.restx: AWEKAS: Posting not enabled.
> deb12pip   | weewx[1] INFO weewx.engine: 'pyephem' detected, extended
> almanac data is available
> deb12pip   | weewx[1] INFO weewxd: Starting up weewx version 5.0.0b15
> deb12pip   | weewx[1] INFO weewx.engine: Clock error is -0.03 seconds
> (positive is fast)
> deb12pip   | weewx[1] INFO weewx.engine: Using binding 'wx_binding' to
> database 'weewx.sdb'
> deb12pip   | weewx[1] INFO weewx.manager: Starting backfill of daily
> summaries
> deb12pip   | weewx[1] INFO weewx.manager: Daily summaries up to date
> deb12pip   | weewx[1] INFO weewx.engine: Starting main packet loop.
> deb12pip   | weewx[1] INFO weewx.manager: Added record 2023-12-01
> 19:25:00 UTC (1701458700) to database 'weewx.sdb'
> deb12pip   | weewx[1] INFO weewx.manager: Added record 2023-12-01
> 19:25:00 UTC (1701458700) to daily summary in 'weewx.sdb'
> deb12pip   |
> /home/weewx/weewx-venv/lib/python3.11/site-packages/Cheetah/Compiler.py:1631:
> UserWarning:
> deb12pip   | You don't have the C version of NameMapper installed! I'm
> disabling Cheetah's useStackFrames option as it is painfully slow with the
> Python version of NameMapper. You should get a copy of Cheetah with
> compiled C version of NameMapper.
> deb12pip   |   warnings.warn(
> deb12pip   | weewx[1] INFO weewx.cheetahgenerator: Generated 8 files
> for report SeasonsReport in 0.50 seconds
> deb12pip   | weewx[1] INFO weewx.imagegenerator: Generated 15 images
> for report SeasonsReport in 0.15 seconds
> deb12pip   | weewx[1] INFO weewx.reportengine: Copied 5 files to
> /home/weewx/weewx-data/public_html
> deb12pip   | weewx[1] INFO user.belchertown: version 1.3.1
> deb12pip   | weewx[1] INFO weewx.cheetahgenerator: Generated 12 files
> for report Belchertown in 0.63 seconds
> deb12pip   | weewx[1] INFO weewx.reportengine: Copied 39 files to
> /home/weewx/weewx-data/public_html/belchertown
>
>
> pi5 pip3 list
>
> PackageVersion
> -- -
> certifi2023.7.22
> charset-normalizer 3.3.2
> configobj  5.0.8
> CT33.3.3
> ephem  4.1.5
> idna   3.4
> paho-mqtt  1.6.1
> Pillow 10.1.0
> pip23.0.1
> PyMySQL1.1.0
> pyserial   3.5
> pyusb  1.2.1
> requests   2.31.0
> setuptools 66.1.1
> 

Re: [weewx-development] Minor item: Could URL references in the skins point to the secure website connection (https vs http)?

[Tom Keffer]

Sure. Done in commit 27f9dab
,
although I can't imagine there are many bad actors who care what part of
weewx.com we browse!

On Thu, Nov 30, 2023 at 4:33 PM jpb...@gmail.com  wrote:

> Hi - I noticed today that the standard skin references "
> http://www.weewx.com; in index.php.tmpl.
>
> Is there a down-side to pointing folks to "https://www.weewx.com; instead?
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/430acfe9-90e0-4842-94c0-c6d4fb4dc30dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBaL037ncWQK9Pbyk28Wq5C5Qngfgs-zXBn7H-n%3DYFVmA%40mail.gmail.com.

Re: [weewx-development] v5 b16 Station Upgrade Error

[Tom Keffer]

Thanks for finding this!

On Thu, Nov 30, 2023 at 3:27 AM G Hammer  wrote:

> I upgraded to b16 and then attempted a station upgrade.
> It errored out on util.
>
> Output uploaded as b16-error.txt
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/4d7cf9cc-ac02-4330-90fe-50574a03d4afn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAFveDuq8ArUmmSaMB5%3D%3DrhKXM%2BTfwiw8U-3Y0Ae_VCNw%40mail.gmail.com.

Re: [weewx-development] Question about timing

[Tom Keffer]

11:23:31 jade python3[362677]: weewx[362677] INFO user.sftp:
> sftpgenerator: transferred 27 files in 5.72 seconds
> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager:
> Added record 2023-11-29 11:25:16 AST (1701271516) to database 'mem.sdb'
> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager:
> Added record 2023-11-29 11:25:16 AST (1701271516) to daily summary in
> 'mem.sdb'
> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager:
> Added record 2023-11-29 11:25:00 AST (1701271500) to database 'weewx.sdb'
> Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager:
> Added record 2023-11-29 11:25:00 AST (1701271500) to daily summary in
> 'weewx.sdb'
> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO
> weewx.cheetahgenerator: Generated 13 files for report StandardReport in
> 189.62 seconds
> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO
> weewx.imagegenerator: Generated 12 images for report StandardReport in 0.41
> seconds
> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO
> weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html
> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO
> weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds
> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO
> weewx.imagegenerator: Generated 1 images for report mem in 0.04 seconds
> Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO
> weewx.reportengine: Copied 0 files to
> /home/pdunphy/weewx-data/public_html/mem
> Nov 29 11:28:26 jade python3[362677]: weewx[362677] INFO
> paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1)
> Nov 29 11:28:26 jade python3[362677]: weewx[362677] INFO
> paramiko.transport: Authentication (publickey) successful!
> Nov 29 11:28:27 jade python3[362677]: weewx[362677] INFO
> paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
> Nov 29 11:28:31 jade python3[362677]: weewx[362677] INFO
> paramiko.transport.sftp: [chan 0] sftp session closed.
> Nov 29 11:28:31 jade python3[362677]: weewx[362677] INFO user.sftp:
> sftpgenerator: transferred 27 files in 5.43 seconds
>
>
> On Wednesday, November 29, 2023 at 11:29:18 AM UTC-4 Tom Keffer wrote:
>
>> Impossible to say without seeing the log.
>>
>> On Wed, Nov 29, 2023 at 6:57 AM Paul Dunphy  wrote:
>>
>>> I am testing v5.0.0b15 in a venv (installed on a Debian-12 based 4 GB Pi
>>> 4B using Vince's script.)  I'm using the Simulator to add records to a copy
>>> of my database with about 860,000 records.  It's working well, but I note
>>> the new record is added at XX:05, XX:10, etc, and then the CPU usage jumps
>>> up for ~3 minutes. Then it uploads the HTML code to my VPS, and the CPU
>>> drops to idle until the next interval ~2 minutes later.
>>>
>>> Am I correct that the system is sweeping through the database,
>>> calculating the derived parameters' min, max, etc.?  With versions before
>>> v5, the upload occurred seconds after the new record was added.
>>>
>>> - Paul VE1DX
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAFTiOo823r15J-SgB%3DAe3bZ015-ZL6p%3DhY%3DSk0J%2Bg42g%40mail.gmail.com.

Re: [weewx-development] Question about timing

[Tom Keffer]

Impossible to say without seeing the log.

On Wed, Nov 29, 2023 at 6:57 AM Paul Dunphy  wrote:

> I am testing v5.0.0b15 in a venv (installed on a Debian-12 based 4 GB Pi
> 4B using Vince's script.)  I'm using the Simulator to add records to a copy
> of my database with about 860,000 records.  It's working well, but I note
> the new record is added at XX:05, XX:10, etc, and then the CPU usage jumps
> up for ~3 minutes. Then it uploads the HTML code to my VPS, and the CPU
> drops to idle until the next interval ~2 minutes later.
>
> Am I correct that the system is sweeping through the database, calculating
> the derived parameters' min, max, etc.?  With versions before v5, the
> upload occurred seconds after the new record was added.
>
> - Paul VE1DX
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBTzZmVbQrAoS8vCE9DDXWsLqOksRiSA4RaApne62FwOw%40mail.gmail.com.

Re: [weewx-development] Re: weectl report build 16

[Tom Keffer]

Yes, which shows that you need to do "weectl report -h".

I don't believe there's a way to get "weectl report" alone to show the help
page. At least, I haven't found it!

On Tue, Nov 28, 2023 at 3:25 PM Steeple Ian  wrote:

> Ok Tom, sorry I misunderstood. So my issue is that running weectl report
> invokes the weectl help page.
>
> On Tuesday, November 28, 2023 at 11:17:24 PM UTC Tom Keffer wrote:
>
>> The commit was not intended to do that. As Vince said, the help page is
>> accessed using "weectl report --help"
>>
>> On Tue, Nov 28, 2023 at 3:05 PM Steeple Ian  wrote:
>>
>>> Tom,
>>> The commit does not seem to have resolved the issue. I am still getting
>>> the help page when I run weectl report from within the virtual environment:
>>> -
>>>
>>> (weewx-venv) xxx@bookworm:~$ sudo systemctl restart weewx
>>> (weewx-venv) xxx@bookworm:~$ weectl report
>>> usage: weectl -v|--version
>>>weectl -h|--help
>>>weectl database --help
>>>weectl debug --help
>>>weectl device --help
>>>weectl extension --help
>>>weectl import --help
>>>weectl report --help
>>>weectl station --help
>>>
>>> weectl is the master utility used by WeeWX. It can invoke several
>>> different
>>> subcommands, listed below. You can explore their utility by using the
>>> --help
>>> option. For example, to find out what the 'database' subcommand can do,
>>> use
>>> 'weectl database --help'.
>>>
>>> options:
>>>   -h, --helpshow this help message and exit
>>>   -v, --version show program's version number and exit
>>>
>>> Available subcommands:
>>>   {database,debug,device,extension,import,report,station}
>>> databaseManage WeeWX databases.
>>> debug   Generate debug info.
>>> device      Manage your hardware.
>>> extension   List, install, or uninstall extensions.
>>> import  Import observation data.
>>> report  List and run WeeWX reports.
>>> station Create, modify, or upgrade a station data area.
>>> (weewx-venv) xxx@bookworm:~$
>>>
>>> On Tuesday, November 28, 2023 at 9:38:42 PM UTC Tom Keffer wrote:
>>>
>>>> The code (wrongly) assumed that a unit system would be declared. Fixed
>>>> in commit 17b956c
>>>> <https://github.com/weewx/weewx/commit/17b956cb2a2f78d6d036cbdf2c89c357c8dee47f>
>>>> .
>>>>
>>>>
>>>> On Tue, Nov 28, 2023 at 10:58 AM Vince Skahan 
>>>> wrote:
>>>>
>>>>> try 'weectl report --help' for the new expanded incantation(s)
>>>>>
>>>>> For me, 'list' no longer works.  It throws an error.
>>>>> But 'run' works.
>>>>>
>>>>> This is a simulator with one added skin.
>>>>> My vantage site with many added skins and extensions fails similarly.
>>>>>
>>>>> (b16)
>>>>>
>>>>> (weewx-venv) pi@pi5:~ $ weectl report run
>>>>> Generating as of last timestamp in the database.
>>>>> The configuration file /home/pi/weewx-data/weewx.conf will be used.
>>>>>
>>>>> (weewx-venv) pi@pi5:~ $ ls -al /var/www/html/weewx/index.html
>>>>> -rw-r--r-- 1 pi pi 32027 Nov 28 10:56 /var/www/html/weewx/index.html
>>>>>
>>>>> (weewx-venv) pi@pi5:~ $ weectl report list
>>>>> Using configuration file /home/pi/weewx-data/weewx.conf
>>>>>
>>>>>   Report  Skin Enabled   Units   Language
>>>>> Traceback (most recent call last):
>>>>>   File "/home/pi/weewx-venv/bin/weectl", line 8, in 
>>>>> sys.exit(main())
>>>>>  ^^
>>>>>   File "/home/pi/weewx-venv/lib/python3.11/site-packages/weectl.py",
>>>>> line 69, in main
>>>>> namespace.func(namespace)
>>>>>   File
>>>>> "/home/pi/weewx-venv/lib/python3.11/site-packages/weectllib/report_cmd.py",
>>>>> line 70, in list_reports
>>>>> weectllib.report_actions.list_reports(namespace.config)
>>>>>   File
>>>>> "/home/pi/weewx-venv/lib/python3.11/site-packages/weectllib/report_actions.py",
>>>

Re: [weewx-development] Re: weectl report build 16

[Tom Keffer]

The commit was not intended to do that. As Vince said, the help page is
accessed using "weectl report --help"

On Tue, Nov 28, 2023 at 3:05 PM Steeple Ian  wrote:

> Tom,
> The commit does not seem to have resolved the issue. I am still getting
> the help page when I run weectl report from within the virtual environment:
> -
>
> (weewx-venv) xxx@bookworm:~$ sudo systemctl restart weewx
> (weewx-venv) xxx@bookworm:~$ weectl report
> usage: weectl -v|--version
>weectl -h|--help
>weectl database --help
>weectl debug --help
>weectl device --help
>weectl extension --help
>weectl import --help
>weectl report --help
>weectl station --help
>
> weectl is the master utility used by WeeWX. It can invoke several different
> subcommands, listed below. You can explore their utility by using the
> --help
> option. For example, to find out what the 'database' subcommand can do, use
> 'weectl database --help'.
>
> options:
>   -h, --helpshow this help message and exit
>   -v, --version show program's version number and exit
>
> Available subcommands:
>   {database,debug,device,extension,import,report,station}
> databaseManage WeeWX databases.
> debug   Generate debug info.
> device  Manage your hardware.
> extension   List, install, or uninstall extensions.
> import  Import observation data.
> report  List and run WeeWX reports.
> station Create, modify, or upgrade a station data area.
> (weewx-venv) xxx@bookworm:~$
>
> On Tuesday, November 28, 2023 at 9:38:42 PM UTC Tom Keffer wrote:
>
>> The code (wrongly) assumed that a unit system would be declared. Fixed in
>> commit 17b956c
>> <https://github.com/weewx/weewx/commit/17b956cb2a2f78d6d036cbdf2c89c357c8dee47f>
>> .
>>
>>
>> On Tue, Nov 28, 2023 at 10:58 AM Vince Skahan  wrote:
>>
>>> try 'weectl report --help' for the new expanded incantation(s)
>>>
>>> For me, 'list' no longer works.  It throws an error.
>>> But 'run' works.
>>>
>>> This is a simulator with one added skin.
>>> My vantage site with many added skins and extensions fails similarly.
>>>
>>> (b16)
>>>
>>> (weewx-venv) pi@pi5:~ $ weectl report run
>>> Generating as of last timestamp in the database.
>>> The configuration file /home/pi/weewx-data/weewx.conf will be used.
>>>
>>> (weewx-venv) pi@pi5:~ $ ls -al /var/www/html/weewx/index.html
>>> -rw-r--r-- 1 pi pi 32027 Nov 28 10:56 /var/www/html/weewx/index.html
>>>
>>> (weewx-venv) pi@pi5:~ $ weectl report list
>>> Using configuration file /home/pi/weewx-data/weewx.conf
>>>
>>>   Report  Skin Enabled   Units   Language
>>> Traceback (most recent call last):
>>>   File "/home/pi/weewx-venv/bin/weectl", line 8, in 
>>> sys.exit(main())
>>>  ^^
>>>   File "/home/pi/weewx-venv/lib/python3.11/site-packages/weectl.py",
>>> line 69, in main
>>> namespace.func(namespace)
>>>   File
>>> "/home/pi/weewx-venv/lib/python3.11/site-packages/weectllib/report_cmd.py",
>>> line 70, in list_reports
>>> weectllib.report_actions.list_reports(namespace.config)
>>>   File
>>> "/home/pi/weewx-venv/lib/python3.11/site-packages/weectllib/report_actions.py",
>>> line 47, in list_reports
>>> unit_system = skin_dict["unit_system"].upper()
>>>   ~^^^
>>>   File
>>> "/home/pi/weewx-venv/lib/python3.11/site-packages/configobj/__init__.py",
>>> line 554, in __getitem__
>>> val = dict.__getitem__(self, key)
>>>   ^^^
>>> KeyError: 'unit_system'
>>>
>>> On Tuesday, November 28, 2023 at 9:26:22 AM UTC-8 Steeple Ian wrote:
>>>
>>>> I do not know if this a known issue: -
>>>>
>>>> Running weectl report from within a virtual environment only brings up
>>>> the weectl help text: -
>>>>
>>>> (weewx-venv) XXX@bookworm:~$ weectl report
>>>> usage: weectl -v|--version
>>>>weectl -h|--help
>>>>weectl database --help
>>>>weectl debug --help
>>>>weectl device --help
>>>>weectl extension --help
>>>>weectl import --help
>>>>weectl

Re: [weewx-development] Re: weectl report build 16

[Tom Keffer]

The code (wrongly) assumed that a unit system would be declared. Fixed in
commit 17b956c

.


On Tue, Nov 28, 2023 at 10:58 AM Vince Skahan  wrote:

> try 'weectl report --help' for the new expanded incantation(s)
>
> For me, 'list' no longer works.  It throws an error.
> But 'run' works.
>
> This is a simulator with one added skin.
> My vantage site with many added skins and extensions fails similarly.
>
> (b16)
>
> (weewx-venv) pi@pi5:~ $ weectl report run
> Generating as of last timestamp in the database.
> The configuration file /home/pi/weewx-data/weewx.conf will be used.
>
> (weewx-venv) pi@pi5:~ $ ls -al /var/www/html/weewx/index.html
> -rw-r--r-- 1 pi pi 32027 Nov 28 10:56 /var/www/html/weewx/index.html
>
> (weewx-venv) pi@pi5:~ $ weectl report list
> Using configuration file /home/pi/weewx-data/weewx.conf
>
>   Report  Skin Enabled   Units   Language
> Traceback (most recent call last):
>   File "/home/pi/weewx-venv/bin/weectl", line 8, in 
> sys.exit(main())
>  ^^
>   File "/home/pi/weewx-venv/lib/python3.11/site-packages/weectl.py", line
> 69, in main
> namespace.func(namespace)
>   File
> "/home/pi/weewx-venv/lib/python3.11/site-packages/weectllib/report_cmd.py",
> line 70, in list_reports
> weectllib.report_actions.list_reports(namespace.config)
>   File
> "/home/pi/weewx-venv/lib/python3.11/site-packages/weectllib/report_actions.py",
> line 47, in list_reports
> unit_system = skin_dict["unit_system"].upper()
>   ~^^^
>   File
> "/home/pi/weewx-venv/lib/python3.11/site-packages/configobj/__init__.py",
> line 554, in __getitem__
> val = dict.__getitem__(self, key)
>   ^^^
> KeyError: 'unit_system'
>
> On Tuesday, November 28, 2023 at 9:26:22 AM UTC-8 Steeple Ian wrote:
>
>> I do not know if this a known issue: -
>>
>> Running weectl report from within a virtual environment only brings up
>> the weectl help text: -
>>
>> (weewx-venv) XXX@bookworm:~$ weectl report
>> usage: weectl -v|--version
>>weectl -h|--help
>>weectl database --help
>>weectl debug --help
>>weectl device --help
>>weectl extension --help
>>weectl import --help
>>weectl report --help
>>weectl station --help
>>
>> weectl is the master utility used by WeeWX. It can invoke several
>> different
>> subcommands, listed below. You can explore their utility by using the
>> --help
>> option. For example, to find out what the 'database' subcommand can do,
>> use
>> 'weectl database --help'.
>>
>> options:
>>   -h, --helpshow this help message and exit
>>   -v, --version show program's version number and exit
>>
>> Available subcommands:
>>   {database,debug,device,extension,import,report,station}
>> databaseManage WeeWX databases.
>> debug   Generate debug info.
>> device  Manage your hardware.
>> extension   List, install, or uninstall extensions.
>> import  Import observation data.
>> report  List and run WeeWX reports.
>> station Create, modify, or upgrade a station data area.
>> (weewx-venv) XXX@bookworm:~$
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/5370cd3a-cd2e-474b-a48c-c7affdd13fa4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zECsHWiED92-tviSEUq%2BHf9o7eoxVO%3DVVdghee22-Qv2Xw%40mail.gmail.com.

Re: [weewx-development] Re: Driver Installation in Version 5

[Tom Keffer]

As we said in my old company for doomed projects, "We'll put a man on it."

Seriously though, there is the "Version 5
<https://github.com/weewx/weewx/wiki/version-5>" design document.

On Sun, Nov 26, 2023 at 2:48 PM Joel Bion  wrote:

> Hi -
>
> I upgraded my setup from 4 to 5 around the time beta 15 came out. The work
> doing that wasn’t hard at all.
>
> The case study I am referring to is _not_ a case study for telling how
> users upgrade to v5. Rather, it’s the case study of a standalone software
> system (weewx) migrating from the old Python setup/install/distribution
> paradigm (distutils? The use of /home/python) to the one Python recommends
> you use from 3.12 forward… with the use of venv, the split between the
> location of program data vs program, etc…
>
> Looking from the sidelines at the development of weewx 5 - it seems there
> were lots of steps for Tom and co. to follow, lots of work to make various
> user upgrade paths work, lots of details.
>
> I am suggesting that the things you had to do might be interesting to
> others. Written up as a case study like a tough case in in a medical
> journal, it might make interesting reading. It could touch upon the
> adequacy of online information to inform you on how you migrated weewx to
> v5, on the applicability of documented conversation steps to your
> particular case, etc. little “gotchas” that you had to solve, etc.
>
>
> Sent from my iPhone
>
> On Nov 26, 2023, at 1:26 PM, Tom Keffer  wrote:
>
> 
> If that's the information Joel is seeking, I honestly don't think it's
> that complicated. Instead of setting up a new station data area by using
> "weectl station create," just use your old one: /home/weewx.
>
> See the detailed instructions *Migrating setup.py installs to Version 5.0
> <https://github.com/weewx/weewx/wiki/v5-upgrade>* in the Wiki. There's
> also a link in the Upgrade Guide.
>
> TL;DR? Read the Guide!
>
> -tk
>
> On Sun, Nov 26, 2023 at 1:05 PM Vince Skahan 
> wrote:
>
>>
>> [...long answer follows...]
>>
>> v5 beta pip user who came from setup.py installation here so let me pass
>> along my experience and what I did
>>
>> This is my personal method and definitely differs from the official
>> steps, but maybe it will help identify one way to approach setting up a
>> test setup before throwing the switch and going live with it
>>
>> I might add that I'm a Vantage VP2 with datalogger user here, so it's
>> easy to stop v4, run v5, and then even go back to v4 if needed (I never
>> needed to do so) and the datalogger will handle catching up an old v4
>> installation.  This is a feature that not all stations have, but it was a
>> nice security blanket here.
>>
>> Anyway - apologies for this being verbose.  Writing it down just in case
>> somebody needs to search for it.
>>
>> 1.  Set up a v5 pip virtual environment and the station's weewx-data
>> directory
>>
>>- created a simulator station using the pip quickstart steps, but did
>>a couple things differently:
>>   - I added the --no-prompt option to 'weectl station create' so it
>>   didn't prompt me
>>   - I didn't put the weewx.service file into place (yet)
>>- installed all the skins and extensions I have in my v4 setup.py
>>config into the new v5 pip weewx-data using 'weectl extension install'
>>- installed any python modules my skins/extensions require (requests
>>and paho-mqtt, to name two) into the python venv
>>   - at this point I could activate the python venv and run weewxd
>>   and see simulated data returned
>>   - and after 5 minutes it should save to the simulator weewx.sdb
>>   and run the reports
>>   - if anything goes boom, you missed adding something to your new
>>   pip based installation
>>- do not (yet) edit weewx.conf to do anything that acts like an
>>uploader (rsync, mqtt, pws, wu, cwop, and so on)
>>- do not (yet) do anything to hook into systemd
>>- if you want to hook the v5 pip public_html into your web at a test
>>path, perhaps http://xyz/weewx-v5, you can do that now
>>
>> 2. Copied my NOAA files into place.  The historical ones take too long to
>> regenerate if you have many years.  When we get to it later the current
>> month/year will be overwritten correctly.
>>
>> 3. Copied my archive db into place.  Don't sweat it getting overwritten
>> with simulator data.  We'll copy the real db (again) when putting v5 pip
>> into production
>>
>> 4. Run weewxd interactively again and let it run

Re: [weewx-development] Re: Driver Installation in Version 5

[Tom Keffer]

enges of moving WeeWX to the current Python installation
>> model.
>>
>> There’s a lesson here for not just the Python developers but others about
>> the difficulty of this transition, because the way WeeWX did things (all
>> things under /home/weewx) was not unreasonable but what has been required
>> to migrate, and the need to support custom package managers, etc…. I just
>> think that this information is useful.
>>
>>
>> Sent from my iPhone
>>
>> On Nov 26, 2023, at 11:33 AM, Tom Keffer  wrote:
>>
>> 
>>
>> /etc/weewx/bin is added to the Python path dynamically when weewxd is
>> started up. However, because it is added to the end of the path, the
>> interpreter will find the version in /usr/share/weewx first.
>>
>> So, why not add it to the beginning of the Python path? Because that
>> causes problems for legacy setup.py installs. In this case, the interpreter
>> will find the code in /home/weewx/bin first, which is the old V4.10 code.
>> The V5 code is under ~/weewx-venv.
>>
>> In summary, there's no good solution except to warn users.
>>
>> -tk
>>
>>
>> On Sun, Nov 26, 2023 at 9:15 AM Karen K  wrote:
>>
>>> Tom Keffer schrieb am Sonntag, 26. November 2023 um 13:50:23 UTC+1:
>>>
>>> This issue is already known (always read the Upgrade Guide!). See
>>> https://weewx.com/docs/5.0/upgrade/#new-location-for-user-directory
>>>
>>>
>>> This is a slightly other problem here. I did a new clean install, no
>>> upgrade.
>>>
>>> My problem was, that the old directory /usr/share/weewx/user still
>>> exists and Python looks for modules there only.
>>>
>>> So the extensions were installed to /etc/weewx/bin/user, but Python did
>>> not look for them there.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-developm...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-developm...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/CAPq0zEAtRxWk2JG7OYC_ontEnv-fFLLdXGUr2KDzf_nyv24XaQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/weewx-development/CAPq0zEAtRxWk2JG7OYC_ontEnv-fFLLdXGUr2KDzf_nyv24XaQ%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/6062f16b-2fbb-452d-91e4-42037cc3ebc0n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/6062f16b-2fbb-452d-91e4-42037cc3ebc0n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAVnxh7Un%3DSzbK9s0DkJpbhXWRsinEW1gXG75vm%3D3JzKA%40mail.gmail.com.

Re: [weewx-development] Re: Driver Installation in Version 5

[Tom Keffer]

Not sure what you're asking for. There is quite a bit of information
in the *Upgrade
Guide <https://weewx.com/docs/5.0/upgrade/>*. What's missing?

On Sun, Nov 26, 2023 at 11:41 AM Joel Bion  wrote:

> I seriously think there is a formal case study that needs to be written
> about the challenges of moving WeeWX to the current Python installation
> model.
>
> There’s a lesson here for not just the Python developers but others about
> the difficulty of this transition, because the way WeeWX did things (all
> things under /home/weewx) was not unreasonable but what has been required
> to migrate, and the need to support custom package managers, etc…. I just
> think that this information is useful.
>
>
> Sent from my iPhone
>
> On Nov 26, 2023, at 11:33 AM, Tom Keffer  wrote:
>
> 
> /etc/weewx/bin is added to the Python path dynamically when weewxd is
> started up. However, because it is added to the end of the path, the
> interpreter will find the version in /usr/share/weewx first.
>
> So, why not add it to the beginning of the Python path? Because that
> causes problems for legacy setup.py installs. In this case, the interpreter
> will find the code in /home/weewx/bin first, which is the old V4.10 code.
> The V5 code is under ~/weewx-venv.
>
> In summary, there's no good solution except to warn users.
>
> -tk
>
>
> On Sun, Nov 26, 2023 at 9:15 AM Karen K  wrote:
>
>> Tom Keffer schrieb am Sonntag, 26. November 2023 um 13:50:23 UTC+1:
>>
>> This issue is already known (always read the Upgrade Guide!). See
>> https://weewx.com/docs/5.0/upgrade/#new-location-for-user-directory
>>
>>
>> This is a slightly other problem here. I did a new clean install, no
>> upgrade.
>>
>> My problem was, that the old directory /usr/share/weewx/user still exists
>> and Python looks for modules there only.
>>
>> So the extensions were installed to /etc/weewx/bin/user, but Python did
>> not look for them there.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-development+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/CAPq0zEAtRxWk2JG7OYC_ontEnv-fFLLdXGUr2KDzf_nyv24XaQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/weewx-development/CAPq0zEAtRxWk2JG7OYC_ontEnv-fFLLdXGUr2KDzf_nyv24XaQ%40mail.gmail.com?utm_medium=email_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEBLcqkY_59Sa0xOW8CkB2kqC%2BR%2BMc2DnzU5sDfQ6dgQbQ%40mail.gmail.com.

Re: [weewx-development] Re: Driver Installation in Version 5

[Tom Keffer]

/etc/weewx/bin is added to the Python path dynamically when weewxd is
started up. However, because it is added to the end of the path, the
interpreter will find the version in /usr/share/weewx first.

So, why not add it to the beginning of the Python path? Because that
causes problems for legacy setup.py installs. In this case, the interpreter
will find the code in /home/weewx/bin first, which is the old V4.10 code.
The V5 code is under ~/weewx-venv.

In summary, there's no good solution except to warn users.

-tk


On Sun, Nov 26, 2023 at 9:15 AM Karen K  wrote:

> Tom Keffer schrieb am Sonntag, 26. November 2023 um 13:50:23 UTC+1:
>
> This issue is already known (always read the Upgrade Guide!). See
> https://weewx.com/docs/5.0/upgrade/#new-location-for-user-directory
>
>
> This is a slightly other problem here. I did a new clean install, no
> upgrade.
>
> My problem was, that the old directory /usr/share/weewx/user still exists
> and Python looks for modules there only.
>
> So the extensions were installed to /etc/weewx/bin/user, but Python did
> not look for them there.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAtRxWk2JG7OYC_ontEnv-fFLLdXGUr2KDzf_nyv24XaQ%40mail.gmail.com.

Re: [weewx-development] Re: Driver Installation in Version 5

[Tom Keffer]

Sorry. I should have followed this thread more closely.

This issue is already known (always read the Upgrade Guide!). See
https://weewx.com/docs/5.0/upgrade/#new-location-for-user-directory

Our plan for upgrades using a package installer is to copy the user
directory over to /etc/weewx/bin/user, then timestamp the old directory
/usr/share/weewx/user..MM.dd.hh, or something similar. However, this
has not been implemented yet.

On Sun, Nov 26, 2023 at 1:35 AM Karen K  wrote:

> Finally, I found out. It refers to the deb packet of version 5.0.0b10.
>
> There are 2 directories called "user": /etc/weewx/bin/user and
> /usr/share/weewx/user. Both of them are included in Python's search path,
> where it looks for modules. Python decides which one it uses to really load
> modules. And it uses the other one.
>
> As a workaround I did:
> cd /etc/weewx/bin
> sudo rm user
> sudo ln -s /usr/share/weewx/user user
>
> After the installation of WeeWX "extensions.py" resided in
> /usr/share/weewx/user. That is the reason I set the link the way I did.
>
> This may be an issue for the development team, @Tom, except it is already
> known.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/4c3d068d-ec47-4935-a7fb-26879811f699n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDOfRgxodYK5_g%2BeDzbjXM0e3VEFNc1hqmfYoO_5HBFLA%40mail.gmail.com.

Re: [weewx-development] Configuring services

[Tom Keffer]

Hi, Rich

If you use MQTTSubscribe as a driver, there is an existing mechanism ---
the configuration editor. Unfortunately, there is not much documentation on
how to use it (well, actually, there is no documentation), but if you look
at existing implementations (such as for the Vantage driver), you can
probably figure it out.

There is no analog for services, although that would not be a bad idea.

-tk


On Sun, Nov 19, 2023 at 5:44 PM bell...@gmail.com 
wrote:

> In the 4.5 years since I released MQTTSubscribe, I’ve leaned so much about
> WeeWX (and continue to be impressed with how well it is designed and
> implemented). It is now time for me to update and improve the process of
> configuring MQTTSubscribe. MQTTSubscribe can be a bit complex to configure
> (that is a whole different discussion). When installing as a driver, one
> can be prompted with configuration options. There is no such mechanism for
> services (and skins), that I know of.
>
> I’m thinking of writing some code similar to wee_config (weectl/station)
> to allow a person to configure MQTTSubscribeService the same way one can
> configure MQTTSubscribeDriver. Has anyone else done something similar? Any
> obvious pitfalls I’m missing?
>
> Thanks. rich
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/b8cc63a0-e08d-4eb4-87e0-c7eca15850ecn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEC5j3YLLEXGw94m9RzwVstFfgU74pRDDY5MSv-DCgXsMQ%40mail.gmail.com.

Re: [weewx-development] is useing WeeWX and logo for extension allowed

[Tom Keffer]

WeeWX is published under the GNU General Public License V3 (GPL3). It is
very permissive. You can pretty much do whatever you want with it. The only
requirement is that you cannot restrict someone else's rights. For example,
you could not put WeeWX inside a hardware box and prevent users from
modifying it.

Your use falls well within the scope of the license.

GPL3 details .

-tk

On Mon, Nov 13, 2023 at 2:15 PM soumas  wrote:

> ---
> Advance information: I hope my question doesn't appear multiple times -
> I've already asked it twice before, but it doesn't appear in the list???
> Last attempt
> ---
>
> Hello!
> I'm in the process of planning a progressive web app as a "skin" for
> WeeWX. Primarily I'm doing this for my RC model airplane friends, but if it
> goes well, I'll be happy to make the app available to the entire WeeWX
> community.
>
> My question now is whether it is allowed to use the wording "WeeWX" as
> part of the project title (e.g. "WeeWX Web App" or the WeeWX logo as part
> of a special "WeeWX Web App logo"? I have seen that there are a few
> Extensions already do that, but I haven't been able to find any official
> statement about it yet.
>
> Best regards,
> Thomas
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/55509ed3-7d73-44d0-8035-60789bf64a77n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAPXf_Ktg%2BYMvu_1pYazUKEUWMrJfWdvNtdAy-CrJh5Tw%40mail.gmail.com.

Re: [weewx-development] errors after b14 upgrade

[Tom Keffer]

Thanks for that report!

This is caused by a fix that went into b15 for some deprecated functions in
Python 3.12. Obviously the fix didn't work.

On Fri, Oct 27, 2023 at 9:54 PM Greg  wrote:

> I upgraded to version 15 from 13 and got these errors:
> I have reverted back to version 13.
>
> Oct 28 15:47:47 moonbi weewx[2890951] DEBUG weewx.restx: Shut down
> StationRegistry thread.
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: Caught
> unrecoverable exception:
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__:   can't
> subtract offset-naive and offset-aware datetimes
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  Traceback (most recent call last):
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewxd.py", line 157,
> in main
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  engine.run()
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line
> 175, in run
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  self.dispatchEvent(weewx.Event(weewx.STARTUP))
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line
> 242, in dispatchEvent
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  callback(event)
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line
> 581, in startup
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  self._catchup(self.engine.console.genStartupRecords)
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line
> 694, in _catchup
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__:   for
> record in generator(lastgood_ts):
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/drivers/fousb.py",
> line 1094, in genArchiveRecords
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  records = self.get_records(since_ts)
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>^^
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/drivers/fousb.py",
> line 1303, in get_records
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  dts, ptr = self.sync(read_period=fixed_block['read_period'])
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
> ^
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: File
> "/opt/weewx/weewx-venv/lib/python3.11/site-packages/weewx/drivers/fousb.py",
> line 1380, in sync
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__:   if
> last_date - prev_date > datetime.timedelta(seconds=50):
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
> ~~^~~
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__: 
>  TypeError: can't subtract offset-naive and offset-aware datetimes
> Oct 28 15:47:47 moonbi weewx[2890951] CRITICAL __main__:   Exiting.
> Oct 28 15:47:47 moonbi systemd[1]: weewx.service: Main process exited,
> code=exited, status=1/FAILURE
> Oct 28 15:47:47 moonbi systemd[1]: weewx.service: Failed with result
> 'exit-code'.
>
>
> On Saturday, 28 October 2023 at 03:11:16 UTC+11 Al Barnes wrote:
>
>> It's working great after upgrading to v5.0.0b15.
>>
>> Thanks!
>>
>> Tom Keffer wrote:
>> > Thanks for spotting this.
>> >
>> > A backwards compatibility shim was inadvertently left out of this beta.
>> > It's needed by older extensions.
>> >
>> > I've uploaded v5.0.0b15. Give it a try.
>> >
>> > On Thu, Oct 26, 2023 at 10:09 PM Al Barnes > > <mailto:vanilla...@gmail.com>> wrote:
>> >
>> > I just upgraded my test unit from b13 to b14 and it throws the
>> > following
>> > errors on startup. I upgraded the same as always with the command
>> > "python3 -m pip install weewx --upgrade".
>> >
>> > Al
>> >
>> > Oct 26 21:56:55 p3-2 systemd[1]: Started weewx.service - WeeWX weather
>&

Re: [weewx-development] errors after b14 upgrade

[Tom Keffer]

Thanks for spotting this.

A backwards compatibility shim was inadvertently left out of this beta.
It's needed by older extensions.

I've uploaded v5.0.0b15. Give it a try.

On Thu, Oct 26, 2023 at 10:09 PM Al Barnes 
wrote:

> I just upgraded my test unit from b13 to b14 and it throws the following
> errors on startup. I upgraded the same as always with the command
> "python3 -m pip install weewx --upgrade".
>
> Al
>
> Oct 26 21:56:55 p3-2 systemd[1]: Started weewx.service - WeeWX weather
> system.
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__:
> Initializing weewx version 5.0.0b14
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__: Using
> Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__: Located
> at /home/al/weewx-venv/bin/python3
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__:
> Platform Linux-6.1.0-rpi4-rpi-v8-aarch64-with-glibc2.36
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__: Locale
> is 'en_US.UTF-8'
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__: Entry
> path: /home/al/weewx-venv/lib/python3.11/site-packages/weewxd.py
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__: Using
> configuration file /home/al/weewx-data/weewx.conf
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO __main__: Debug is 0
> Oct 26 21:56:55 p3-2 python3[22961]: weewx[22961] INFO weewx.engine:
> Loading station type Vantage (weewx.drivers.vantage)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.engine:
> StdConvert target unit is 0x1
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.wxservices:
> StdWXCalculate will use data binding wx_binding
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.engine:
> Archive will use data binding wx_binding
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.engine:
> Record generation will be attempted in 'hardware'
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.engine:
> Using archive interval of 300 seconds (specified by hardware)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.restx:
> StationRegistry: Registration not requested.
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.restx:
> Wunderground: Posting not enabled.
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.restx:
> PWSweather: Posting not enabled.
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.restx:
> CWOP: Posting not enabled.
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.restx: WOW:
> Posting not enabled.
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] INFO weewx.restx:
> AWEKAS: Posting not enabled.
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> Caught unrecoverable exception:
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>   cannot import name 'accumulateLeaves' from 'weeutil.weeutil'
> (/home/al/weewx-venv/lib/python3.11/site-packages/weeutil/weeutil.py)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>   Traceback (most recent call last):
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> File
> "/home/al/weewx-venv/lib/python3.11/site-packages/weewxd.py", line 151,
> in main
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>   engine = weewx.engine.StdEngine(config_dict)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>    ^^^
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> File
> "/home/al/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line
> 90, in __init__
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>   self.loadServices(config_dict)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> File
> "/home/al/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line
> 158, in loadServices
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>   obj = weeutil.weeutil.get_object(svc)(self, config_dict)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> ^^^
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> File
> "/home/al/weewx-venv/lib/python3.11/site-packages/weeutil/weeutil.py",
> line 1393, in get_object
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>   module = importlib.import_module(module_name)
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
>    
> Oct 26 21:56:56 p3-2 python3[22961]: weewx[22961] CRITICAL __main__:
> File "/usr/lib/python3.11/importlib/__init__.py", line 126, in
> import_module
> Oct 26 21:56:56 p3-2