Have you had any odd weewx errors or crashes related to the sunshine
calculations? I ask because I hadn't, but I decided to try to 'backfill' my
database with sunshine times, based on the 5-minute radiation values, and I
ran into a bizarre bug. I used the code shown below (on a copy of my live
weewx database). As you will see, the threshold calculation code is
essentially identical to yours, except that it has been converted to a
regular function (no 'self' parameter) and my station's latitude and
longitude are hard coded in it. When the code is run under Python 3.9.2 on
my Pi, it initially runs without problems, but crashes after 8,000+ records
have been processed with a ValueError on the MaxThreshold vs threshold
comparison, reporting that it can't compare a complex with a float! If I
intercept and log the errors, it turns out that, for a few specific values
of dateTime, the function returns a complex number! Even more bizarrely, it
only seems to do that in the context of the running code. If I manually run
through all the operations from the function code at the Python command
line, using the value of dateTime that produces the first crash, all the
intermediate results and the final result are sane floats.
There appears to be a second issue, possibly related to my reading and
writing the database at relatively high frequency, which stalls the process
after about 18,000 records have been processed, but removing the database
writes allows it to run to completion without abolishing the consistent,
albeit infrequent, ValueErrors.
[backfill.py]
import sqlite3
from datetime import datetime
import time
from math import sin, cos, pi, asin
def sunshineThreshold(mydatetime):
coeff = 0.9 # change to calibrate with your sensor
utcdate = datetime.utcfromtimestamp(mydatetime)
dayofyear = int(time.strftime("%j", time.gmtime(mydatetime)))
theta = 360 * dayofyear / 365
equatemps = 0.0172 + 0.4281 * cos((pi / 180) * theta) - 7.3515 * sin(
(pi / 180) * theta) - 3.3495 * cos(2 * (pi / 180) * theta) - 9.3619
* sin(
2 * (pi / 180) * theta)
latitude = 43.0346213
longitude = -78.689362
corrtemps = longitude * 4
declinaison = asin(0.006918 - 0.399912 * cos((pi / 180) * theta) +
0.070257 * sin(
(pi / 180) * theta) - 0.006758 * cos(2 * (pi / 180) * theta) +
0.000908 * sin(
2 * (pi / 180) * theta)) * (180 / pi)
minutesjour = utcdate.hour * 60 + utcdate.minute
tempsolaire = (minutesjour + corrtemps + equatemps) / 60
angle_horaire = (tempsolaire - 12) * 15
hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / 180) *
declinaison) + cos(
(pi / 180) * latitude) * cos((pi / 180) * declinaison) * cos((pi /
180) * angle_horaire)) * (180 / pi)
seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) * 1080
* pow(
(sin(pi / 180) * hauteur_soleil), 1.25) * coeff
return seuil
database = 'weewx.sdb'
maxThreshold=0
count=0
conn=sqlite3.connect(database)
cur=conn.execute("SELECT dateTime, Radiation from archive")
for row in cur:
count += 1
if (row[1] is not None) and (row[1] > 20):
threshold = sunshineThreshold(row[0])
if threshold > maxThreshold:
maxThreshold = threshold
if row[1] > threshold:
conn.execute("UPDATE archive set SunshineTime = 5 WHERE dateTime =
" + str(row[0]))
if count % 1000 == 0:
print(count, 'Max Threshold', maxThreshold)
conn.close
[/backfill.py]
On Friday, June 10, 2022 at 3:29:40 AM UTC-4 [email protected] wrote:
> On my side, I have looked at the CPU utilization on my raspberry Pi 3B+. I
> have the mqtt service service installed, so at each loop all data of the
> packet are sent to the mqtt broker.
>
> With mqtt and when calculations of the sunshine threshold is done for each
> loop packet, the total CPU utilization of python3 is about 0.75%
> With mqtt and without calculation of sunshine threshold : 0.5% of total
> CPU.
>
> So one can estimate that 0.25 % of total CPU is needed for the calculation
> of the threshold value for each LOOP packet.
>
>
> Le 9 juin 2022 à 22:26, 'Peter Fletcher' via weewx-user <
> [email protected]> a écrit :
>
> After some experimentation, I found that the radiation value in the VP2
> LOOP packets does, indeed, normally change every 50-52 seconds, but,
> perhaps about a fifth of the 'gaps' are a *multiple* of that time - most
> often 100+ or 150+ seconds, but occasionally more than that (I saw one 250+
> second 'gap'). I saw this under conditions of variable sunshine and clouds
> when it seemed unlikely that the actual radiation value would have been
> precisely constant for that length of time, so I am not sure exactly what
> is going on. In any event, I am revising the code I am using on the basis
> of doing the threshold calculation when the radiation level changes, but at
> least every minute, if it remains constant for more than the normal 50-52
> seconds..
>
> On Sunday, June 5, 2022 at 12:33:47 PM UTC-4 [email protected] wrote:
>
>> I think it is also OK to do an average for every 30 seconds. It depends
>> also on the weather station used.
>> For instance, a Davis Vantage Pro 2 ISS transmits an updated solar
>> radiation value every 50 to 60 seconds. So with this weather station, even
>> a 1 minute average would not be very different since anyway the solar
>> radiation values of the LOOP packet are the same for at least 50 seconds.!
>>
>> Le 5 juin 2022 à 18:02, 'Peter Fletcher' via weewx-user <
>> [email protected]> a écrit :
>>
>> I chose to average the LOOP radiation readings and only to do the
>> threshold calculation and make the sun/no sun determination every 30
>> seconds because I thought doing it on every LOOP might overload LOOP
>> processing (I am running weewx on a Pi 3B, which is also doing a few other
>> things which use the CPU). If this is an unnecessary concern, as it may
>> very well be, your modified code is much cleaner than mine.
>>
>> On Saturday, June 4, 2022 at 12:41:08 PM UTC-4 [email protected] wrote:
>>
>>> It is a very good idea to calculate the sunshine duration for each LOOP
>>> packet and sum these values to make the final archive sunshine duration. I
>>> have modified my script accordingly :
>>> https://github.com/Jterrettaz/sunduration.
>>> The logic is the following : for each received LOOP packet, the
>>> radiation is compared to a calculated threshold. If the radiation is above
>>> the threshold value, the sunshine time for the LOOP packet is equal to the
>>> time elapsed between the previous loop packet and this packet (most of the
>>> time 2 seconds with a Vantage Davis Pro).
>>> The final archive sunshine duration is the sum of all the LOOP value
>>> within the archive period.
>>> Le vendredi 3 juin 2022 à 21:59:36 UTC+2, Peter Fletcher a écrit :
>>>
>>>> That makes some sense when you are getting data from an 'external'
>>>> sensor, though there are (IMHO) simpler ways of doing it. weewx already
>>>> has
>>>> access to the LOOP radiation data from the VP2, so handling the processing
>>>> and data storage within weewx makes more sense to me in this case.
>>>>
>>>> On Friday, June 3, 2022 at 3:24:23 PM UTC-4 vince wrote:
>>>>
>>>>> On Friday, June 3, 2022 at 11:17:00 AM UTC-7 Meteo Oberwallis wrote:
>>>>>
>>>>>> if the interval of Weewx and the data logger is set to 10 minutes, I
>>>>>> would have liked to read the value of the solar sensor every minute and
>>>>>> then write it into a separate .sdb database as possible sunshine.
>>>>>
>>>>>
>>>>> Personally I'd use an external program called via cron and posting a
>>>>> message to a MQTT topic. Have weewx subscribe to that topic to get the
>>>>> data into your db.
>>>>>
>>>>> This is how I used to get my DS18b20 temperature sensor data into
>>>>> weewx.
>>>>>
>>>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com
>
> <https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/7f82e0d0-9207-4f00-953b-8c809410b1fen%40googlegroups.com.
