No, I never had weewx crashes related to the sunshine calculations.
What are the dateTime values that trigger the error ?
Le mercredi 29 juin 2022 à 23:23:16 UTC+2, Peter Fletcher a écrit :
> Have you had any odd weewx errors or crashes related to the sunshine
> calculations? I ask because I hadn't, but I decided to try to 'backfill' my
> database with sunshine times, based on the 5-minute radiation values, and I
> ran into a bizarre bug. I used the code shown below (on a copy of my live
> weewx database). As you will see, the threshold calculation code is
> essentially identical to yours, except that it has been converted to a
> regular function (no 'self' parameter) and my station's latitude and
> longitude are hard coded in it. When the code is run under Python 3.9.2 on
> my Pi, it initially runs without problems, but crashes after 8,000+ records
> have been processed with a ValueError on the MaxThreshold vs threshold
> comparison, reporting that it can't compare a complex with a float! If I
> intercept and log the errors, it turns out that, for a few specific values
> of dateTime, the function returns a complex number! Even more bizarrely, it
> only seems to do that in the context of the running code. If I manually run
> through all the operations from the function code at the Python command
> line, using the value of dateTime that produces the first crash, all the
> intermediate results and the final result are sane floats.
> There appears to be a second issue, possibly related to my reading and
> writing the database at relatively high frequency, which stalls the process
> after about 18,000 records have been processed, but removing the database
> writes allows it to run to completion without abolishing the consistent,
> albeit infrequent, ValueErrors.
>
> [backfill.py]
> import sqlite3
> from datetime import datetime
> import time
> from math import sin, cos, pi, asin
>
> def sunshineThreshold(mydatetime):
> coeff = 0.9 # change to calibrate with your sensor
> utcdate = datetime.utcfromtimestamp(mydatetime)
> dayofyear = int(time.strftime("%j", time.gmtime(mydatetime)))
> theta = 360 * dayofyear / 365
> equatemps = 0.0172 + 0.4281 * cos((pi / 180) * theta) - 7.3515 * sin(
> (pi / 180) * theta) - 3.3495 * cos(2 * (pi / 180) * theta) -
> 9.3619 * sin(
> 2 * (pi / 180) * theta)
>
> latitude = 43.0346213
> longitude = -78.689362
>
> corrtemps = longitude * 4
> declinaison = asin(0.006918 - 0.399912 * cos((pi / 180) * theta) +
> 0.070257 * sin(
> (pi / 180) * theta) - 0.006758 * cos(2 * (pi / 180) * theta) +
> 0.000908 * sin(
> 2 * (pi / 180) * theta)) * (180 / pi)
> minutesjour = utcdate.hour * 60 + utcdate.minute
> tempsolaire = (minutesjour + corrtemps + equatemps) / 60
> angle_horaire = (tempsolaire - 12) * 15
> hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / 180) *
> declinaison) + cos(
> (pi / 180) * latitude) * cos((pi / 180) * declinaison) * cos((pi /
> 180) * angle_horaire)) * (180 / pi)
> seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) * 1080
> * pow(
> (sin(pi / 180) * hauteur_soleil), 1.25) * coeff
> return seuil
>
>
> database = 'weewx.sdb'
>
> maxThreshold=0
> count=0
> conn=sqlite3.connect(database)
> cur=conn.execute("SELECT dateTime, Radiation from archive")
> for row in cur:
> count += 1
> if (row[1] is not None) and (row[1] > 20):
> threshold = sunshineThreshold(row[0])
> if threshold > maxThreshold:
> maxThreshold = threshold
> if row[1] > threshold:
> conn.execute("UPDATE archive set SunshineTime = 5 WHERE dateTime =
> " + str(row[0]))
> if count % 1000 == 0:
> print(count, 'Max Threshold', maxThreshold)
> conn.close
> [/backfill.py]
>
> On Friday, June 10, 2022 at 3:29:40 AM UTC-4 [email protected] wrote:
>
>> On my side, I have looked at the CPU utilization on my raspberry Pi 3B+.
>> I have the mqtt service service installed, so at each loop all data of the
>> packet are sent to the mqtt broker.
>>
>> With mqtt and when calculations of the sunshine threshold is done for
>> each loop packet, the total CPU utilization of python3 is about 0.75%
>> With mqtt and without calculation of sunshine threshold : 0.5% of total
>> CPU.
>>
>> So one can estimate that 0.25 % of total CPU is needed for the
>> calculation of the threshold value for each LOOP packet.
>>
>>
>> Le 9 juin 2022 à 22:26, 'Peter Fletcher' via weewx-user <
>> [email protected]> a écrit :
>>
>> After some experimentation, I found that the radiation value in the VP2
>> LOOP packets does, indeed, normally change every 50-52 seconds, but,
>> perhaps about a fifth of the 'gaps' are a *multiple* of that time - most
>> often 100+ or 150+ seconds, but occasionally more than that (I saw one 250+
>> second 'gap'). I saw this under conditions of variable sunshine and clouds
>> when it seemed unlikely that the actual radiation value would have been
>> precisely constant for that length of time, so I am not sure exactly what
>> is going on. In any event, I am revising the code I am using on the basis
>> of doing the threshold calculation when the radiation level changes, but at
>> least every minute, if it remains constant for more than the normal 50-52
>> seconds..
>>
>> On Sunday, June 5, 2022 at 12:33:47 PM UTC-4 [email protected] wrote:
>>
>>> I think it is also OK to do an average for every 30 seconds. It depends
>>> also on the weather station used.
>>> For instance, a Davis Vantage Pro 2 ISS transmits an updated solar
>>> radiation value every 50 to 60 seconds. So with this weather station, even
>>> a 1 minute average would not be very different since anyway the solar
>>> radiation values of the LOOP packet are the same for at least 50 seconds.!
>>>
>>> Le 5 juin 2022 à 18:02, 'Peter Fletcher' via weewx-user <
>>> [email protected]> a écrit :
>>>
>>> I chose to average the LOOP radiation readings and only to do the
>>> threshold calculation and make the sun/no sun determination every 30
>>> seconds because I thought doing it on every LOOP might overload LOOP
>>> processing (I am running weewx on a Pi 3B, which is also doing a few other
>>> things which use the CPU). If this is an unnecessary concern, as it may
>>> very well be, your modified code is much cleaner than mine.
>>>
>>> On Saturday, June 4, 2022 at 12:41:08 PM UTC-4 [email protected] wrote:
>>>
>>>> It is a very good idea to calculate the sunshine duration for each LOOP
>>>> packet and sum these values to make the final archive sunshine duration.
>>>> I
>>>> have modified my script accordingly :
>>>> https://github.com/Jterrettaz/sunduration.
>>>> The logic is the following : for each received LOOP packet, the
>>>> radiation is compared to a calculated threshold. If the radiation is above
>>>> the threshold value, the sunshine time for the LOOP packet is equal to the
>>>> time elapsed between the previous loop packet and this packet (most of
>>>> the
>>>> time 2 seconds with a Vantage Davis Pro).
>>>> The final archive sunshine duration is the sum of all the LOOP value
>>>> within the archive period.
>>>> Le vendredi 3 juin 2022 à 21:59:36 UTC+2, Peter Fletcher a écrit :
>>>>
>>>>> That makes some sense when you are getting data from an 'external'
>>>>> sensor, though there are (IMHO) simpler ways of doing it. weewx already
>>>>> has
>>>>> access to the LOOP radiation data from the VP2, so handling the
>>>>> processing
>>>>> and data storage within weewx makes more sense to me in this case.
>>>>>
>>>>> On Friday, June 3, 2022 at 3:24:23 PM UTC-4 vince wrote:
>>>>>
>>>>>> On Friday, June 3, 2022 at 11:17:00 AM UTC-7 Meteo Oberwallis wrote:
>>>>>>
>>>>>>> if the interval of Weewx and the data logger is set to 10 minutes,
>>>>>>> I would have liked to read the value of the solar sensor every minute
>>>>>>> and
>>>>>>> then write it into a separate .sdb database as possible sunshine.
>>>>>>
>>>>>>
>>>>>> Personally I'd use an external program called via cron and posting a
>>>>>> message to a MQTT topic. Have weewx subscribe to that topic to get the
>>>>>> data into your db.
>>>>>>
>>>>>> This is how I used to get my DS18b20 temperature sensor data into
>>>>>> weewx.
>>>>>>
>>>>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "weewx-user" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/9c7b838c-e86b-42cb-b62a-98a5a33e5f5bn%40googlegroups.com.
