That was going to be my next step! In fact, iterating through a list of the dateTime values that produce the errors in the real code and passing each value to the function confirms that it is the specific dateTime values that are causing the function to misbehave. The returned results are all complex numbers with negative and numerically identical (for a given dateTime) real and imaginary components. It does seem to be a bug in the function. I assume that hauteur_soleil should always be >=0. It appears that, for my latitude and longitude and for the given specific values of dateTime, it becomes negative. The last step in the calculation then involves raising a negative number to a non-integral power, which is guaranteed to produce interesting results! The really odd thing is that math.pow is not returning a ValueError, which the docs say is what should happen under these circumstances, but apparently trying to return a (possibly) valid complex result. On Thursday, June 30, 2022 at 3:07:38 PM UTC-4 [email protected] wrote:
> The only clue I have is that the problem is not due to an « overloading » > of your raspberry pi, but seems to occur with specific dateTime values. > You can try to run your script only with a « bad » dateTime : > > "SELECT dateTime, Radiation from archive where dateTime = 1592614500 » > > Does the error occurs ? If yes, you can try to add debugging print > commands inside the sunshineThreshold function to try to understand. > > > > Le 30 juin 2022 à 19:51, 'Peter Fletcher' via weewx-user < > [email protected]> a écrit : > > It did as it seems you predicted - passed 1592614800 and stopped at > 1632611100. You obviously have a clue as to what is going on. Please > explain! > > On Thursday, June 30, 2022 at 8:59:48 AM UTC-4 [email protected] wrote: > >> If you exclude the first one,1592614500 , with a query like "SELECT >> dateTime, Radiation from archive where dateTime <> 1592614500", will the >> script stop at 1592614800 ( the next dateTime) or will it continue and stop >> at 1632611100 ? >> >> Le 30 juin 2022 à 14:34, 'Peter Fletcher' via weewx-user < >> [email protected]> a écrit : >> >> 1592614500 >> 1632611100 >> 1632611400 >> 1647688800 >> >> I can't see a pattern or any common features. >> >> On Thursday, June 30, 2022 at 3:55:49 AM UTC-4 [email protected] wrote: >> >>> No, I never had weewx crashes related to the sunshine calculations. >>> >>> What are the dateTime values that trigger the error ? >>> >>> >>> >>> Le mercredi 29 juin 2022 à 23:23:16 UTC+2, Peter Fletcher a écrit : >>> >>>> Have you had any odd weewx errors or crashes related to the sunshine >>>> calculations? I ask because I hadn't, but I decided to try to 'backfill' >>>> my >>>> database with sunshine times, based on the 5-minute radiation values, and >>>> I >>>> ran into a bizarre bug. I used the code shown below (on a copy of my live >>>> weewx database). As you will see, the threshold calculation code is >>>> essentially identical to yours, except that it has been converted to a >>>> regular function (no 'self' parameter) and my station's latitude and >>>> longitude are hard coded in it. When the code is run under Python 3.9.2 on >>>> my Pi, it initially runs without problems, but crashes after 8,000+ >>>> records >>>> have been processed with a ValueError on the MaxThreshold vs threshold >>>> comparison, reporting that it can't compare a complex with a float! If I >>>> intercept and log the errors, it turns out that, for a few specific values >>>> of dateTime, the function returns a complex number! Even more bizarrely, >>>> it >>>> only seems to do that in the context of the running code. If I manually >>>> run >>>> through all the operations from the function code at the Python command >>>> line, using the value of dateTime that produces the first crash, all the >>>> intermediate results and the final result are sane floats. >>>> There appears to be a second issue, possibly related to my reading and >>>> writing the database at relatively high frequency, which stalls the >>>> process >>>> after about 18,000 records have been processed, but removing the database >>>> writes allows it to run to completion without abolishing the consistent, >>>> albeit infrequent, ValueErrors. >>>> >>>> [backfill.py] >>>> import sqlite3 >>>> from datetime import datetime >>>> import time >>>> from math import sin, cos, pi, asin >>>> >>>> def sunshineThreshold(mydatetime): >>>> coeff = 0.9 # change to calibrate with your sensor >>>> utcdate = datetime.utcfromtimestamp(mydatetime) >>>> dayofyear = int(time.strftime("%j", time.gmtime(mydatetime))) >>>> theta = 360 * dayofyear / 365 >>>> equatemps = 0.0172 + 0.4281 * cos((pi / 180) * theta) - 7.3515 * >>>> sin( >>>> (pi / 180) * theta) - 3.3495 * cos(2 * (pi / 180) * theta) - >>>> 9.3619 * sin( >>>> 2 * (pi / 180) * theta) >>>> >>>> latitude = 43.0346213 >>>> longitude = -78.689362 >>>> >>>> corrtemps = longitude * 4 >>>> declinaison = asin(0.006918 - 0.399912 * cos((pi / 180) * theta) + >>>> 0.070257 * sin( >>>> (pi / 180) * theta) - 0.006758 * cos(2 * (pi / 180) * theta) + >>>> 0.000908 * sin( >>>> 2 * (pi / 180) * theta)) * (180 / pi) >>>> minutesjour = utcdate.hour * 60 + utcdate.minute >>>> tempsolaire = (minutesjour + corrtemps + equatemps) / 60 >>>> angle_horaire = (tempsolaire - 12) * 15 >>>> hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / 180) >>>> * declinaison) + cos( >>>> (pi / 180) * latitude) * cos((pi / 180) * declinaison) * >>>> cos((pi / 180) * angle_horaire)) * (180 / pi) >>>> seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) * >>>> 1080 * pow( >>>> (sin(pi / 180) * hauteur_soleil), 1.25) * coeff >>>> return seuil >>>> >>>> >>>> database = 'weewx.sdb' >>>> >>>> maxThreshold=0 >>>> count=0 >>>> conn=sqlite3.connect(database) >>>> cur=conn.execute("SELECT dateTime, Radiation from archive") >>>> for row in cur: >>>> count += 1 >>>> if (row[1] is not None) and (row[1] > 20): >>>> threshold = sunshineThreshold(row[0]) >>>> if threshold > maxThreshold: >>>> maxThreshold = threshold >>>> if row[1] > threshold: >>>> conn.execute("UPDATE archive set SunshineTime = 5 WHERE >>>> dateTime = " + str(row[0])) >>>> if count % 1000 == 0: >>>> print(count, 'Max Threshold', maxThreshold) >>>> conn.close >>>> [/backfill.py] >>>> >>>> On Friday, June 10, 2022 at 3:29:40 AM UTC-4 [email protected] wrote: >>>> >>>>> On my side, I have looked at the CPU utilization on my raspberry Pi >>>>> 3B+. I have the mqtt service service installed, so at each loop all data >>>>> of the packet are sent to the mqtt broker. >>>>> >>>>> With mqtt and when calculations of the sunshine threshold is done for >>>>> each loop packet, the total CPU utilization of python3 is about 0.75% >>>>> With mqtt and without calculation of sunshine threshold : 0.5% of >>>>> total CPU. >>>>> >>>>> So one can estimate that 0.25 % of total CPU is needed for the >>>>> calculation of the threshold value for each LOOP packet. >>>>> >>>>> >>>>> Le 9 juin 2022 à 22:26, 'Peter Fletcher' via weewx-user < >>>>> [email protected]> a écrit : >>>>> >>>>> After some experimentation, I found that the radiation value in the >>>>> VP2 LOOP packets does, indeed, normally change every 50-52 seconds, but, >>>>> perhaps about a fifth of the 'gaps' are a *multiple* of that time - >>>>> most often 100+ or 150+ seconds, but occasionally more than that (I saw >>>>> one >>>>> 250+ second 'gap'). I saw this under conditions of variable sunshine and >>>>> clouds when it seemed unlikely that the actual radiation value would have >>>>> been precisely constant for that length of time, so I am not sure exactly >>>>> what is going on. In any event, I am revising the code I am using on the >>>>> basis of doing the threshold calculation when the radiation level >>>>> changes, >>>>> but at least every minute, if it remains constant for more than the >>>>> normal >>>>> 50-52 seconds.. >>>>> >>>>> On Sunday, June 5, 2022 at 12:33:47 PM UTC-4 [email protected] wrote: >>>>> >>>>>> I think it is also OK to do an average for every 30 seconds. It >>>>>> depends also on the weather station used. >>>>>> For instance, a Davis Vantage Pro 2 ISS transmits an updated solar >>>>>> radiation value every 50 to 60 seconds. So with this weather station, >>>>>> even >>>>>> a 1 minute average would not be very different since anyway the solar >>>>>> radiation values of the LOOP packet are the same for at least 50 >>>>>> seconds.! >>>>>> >>>>>> Le 5 juin 2022 à 18:02, 'Peter Fletcher' via weewx-user < >>>>>> [email protected]> a écrit : >>>>>> >>>>>> I chose to average the LOOP radiation readings and only to do the >>>>>> threshold calculation and make the sun/no sun determination every 30 >>>>>> seconds because I thought doing it on every LOOP might overload LOOP >>>>>> processing (I am running weewx on a Pi 3B, which is also doing a few >>>>>> other >>>>>> things which use the CPU). If this is an unnecessary concern, as it may >>>>>> very well be, your modified code is much cleaner than mine. >>>>>> >>>>>> On Saturday, June 4, 2022 at 12:41:08 PM UTC-4 [email protected] >>>>>> wrote: >>>>>> >>>>>>> It is a very good idea to calculate the sunshine duration for each >>>>>>> LOOP packet and sum these values to make the final archive sunshine >>>>>>> duration. I have modified my script accordingly : >>>>>>> https://github.com/Jterrettaz/sunduration. >>>>>>> The logic is the following : for each received LOOP packet, the >>>>>>> radiation is compared to a calculated threshold. If the radiation is >>>>>>> above >>>>>>> the threshold value, the sunshine time for the LOOP packet is equal to >>>>>>> the >>>>>>> time elapsed between the previous loop packet and this packet (most of >>>>>>> the >>>>>>> time 2 seconds with a Vantage Davis Pro). >>>>>>> The final archive sunshine duration is the sum of all the LOOP value >>>>>>> within the archive period. >>>>>>> Le vendredi 3 juin 2022 à 21:59:36 UTC+2, Peter Fletcher a écrit : >>>>>>> >>>>>>>> That makes some sense when you are getting data from an 'external' >>>>>>>> sensor, though there are (IMHO) simpler ways of doing it. weewx >>>>>>>> already has >>>>>>>> access to the LOOP radiation data from the VP2, so handling the >>>>>>>> processing >>>>>>>> and data storage within weewx makes more sense to me in this case. >>>>>>>> >>>>>>>> On Friday, June 3, 2022 at 3:24:23 PM UTC-4 vince wrote: >>>>>>>> >>>>>>>>> On Friday, June 3, 2022 at 11:17:00 AM UTC-7 Meteo Oberwallis >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> if the interval of Weewx and the data logger is set to 10 >>>>>>>>>> minutes, I would have liked to read the value of the solar sensor >>>>>>>>>> every >>>>>>>>>> minute and then write it into a separate .sdb database as possible >>>>>>>>>> sunshine. >>>>>>>>> >>>>>>>>> >>>>>>>>> Personally I'd use an external program called via cron and posting >>>>>>>>> a message to a MQTT topic. Have weewx subscribe to that topic to get >>>>>>>>> the >>>>>>>>> data into your db. >>>>>>>>> >>>>>>>>> This is how I used to get my DS18b20 temperature sensor data into >>>>>>>>> weewx. >>>>>>>>> >>>>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "weewx-user" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "weewx-user" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "weewx-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "weewx-user" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/c07ed2bb-1e3d-43e2-b08c-08a7a3aa92dbn%40googlegroups.com > > <https://groups.google.com/d/msgid/weewx-user/c07ed2bb-1e3d-43e2-b08c-08a7a3aa92dbn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f287c1b3-1005-409c-82a9-a072e375d5e9n%40googlegroups.com.
