Hello, everyone. I eagerly await your experiences. It's very interesting what you can do. Now I wanted to ask you if there is already a finished code that you can use? Did I understand correctly that with these calculations you can measure the hours of sunshine by minutes and not just by the interval of the data logger? Can you use it to fill up past values? So e.g. from the year 2019 or something?
Best regards Stefan Peter Fletcher schrieb am Freitag, 1. Juli 2022 um 19:37:54 UTC+2: > I'm sure you are right about the triggering circumstances. The code in my > weewx User routines was correct (using the Meteo France limit), but I had > put in some modifications around the hauteur_soleil test in connection with > the method I am using to average and 'normalize' the observations for the > archive reports. I suspect that, in taking out those modifications for the > archive update code, I inadvertently also removed the original test. > > On Friday, July 1, 2022 at 10:47:43 AM UTC-4 [email protected] wrote: > >> OK. >> The variable "hauteur_soleil is the elevation of the sun, in degree. >> From sunset to sunrise, this value will be negative (i.e. sun below the >> horizon). >> The formula developed by Metro France was validated to be effective only >> if sun elevation is > 3°. >> >> In my own implantation of this algorithm, I decided to start the >> calculation of the sun threshold as soon as the sun elevation was > 0° >> *and* that the global radiation as measured by the Davis pyranometer was >> greeter than 20 W/m2. >> >> I suspect that the few dateTime that trigger your errors were in a >> situation in which sun elevation was just below zero and that the measured >> radiation was just higher than 20 W/m2 >> >> Le 1 juil. 2022 à 16:19, 'Peter Fletcher' via weewx-user < >> [email protected]> a écrit : >> >> Something like that is obviously needed. I don't know how my copy of the >> code omitted the test for hauteur_soleil being > 0. There are a number of >> sources from which I may have copied it, and I didn't keep links to all of >> them, but I can't imagine why I would have deleted the test if my source >> had included it. I will obviously add it to my working code. >> >> On Friday, July 1, 2022 at 2:45:42 AM UTC-4 [email protected] wrote: >> >>> OK. What I don't understand is that my code had always a test >>> >>> if hauteur_soleil > 3: (according to Météo France) >>> >>> or more recently >>> >>> if hauteur_soleil > 0: >>> >>> in the function, and this test is not on your function. >>> >>> It should be : >>> >>> hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / 180) * >>> declinaison) + cos( >>> (pi / 180) * latitude) * cos((pi / 180) * declinaison) * cos((pi >>> / 180) * angle_horaire)) * (180 / pi) >>> If hauteur_soleil > 0: >>> >>> seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) * >>> 1080 * pow( >>> (sin(pi / 180) * hauteur_soleil), 1.25) * coeff >>> else : >>> seuil = 0 >>> return seuil >>> >>> Le 30 juin 2022 à 22:52, 'Peter Fletcher' via weewx-user < >>> [email protected]> a écrit : >>> >>> That was going to be my next step! In fact, iterating through a list of >>> the dateTime values that produce the errors in the real code and passing >>> each value to the function confirms that it is the specific dateTime values >>> that are causing the function to misbehave. The returned results are all >>> complex numbers with negative and numerically identical (for a given >>> dateTime) real and imaginary components. It does seem to be a bug in the >>> function. I assume that hauteur_soleil should always be >=0. It appears >>> that, for my latitude and longitude and for the given specific values of >>> dateTime, it becomes negative. The last step in the calculation then >>> involves raising a negative number to a non-integral power, which is >>> guaranteed to produce interesting results! The really odd thing is that >>> math.pow is not returning a ValueError, which the docs say is what should >>> happen under these circumstances, but apparently trying to return a >>> (possibly) valid complex result. >>> On Thursday, June 30, 2022 at 3:07:38 PM UTC-4 [email protected] wrote: >>> >>>> The only clue I have is that the problem is not due to an >>>> « overloading » of your raspberry pi, but seems to occur with specific >>>> dateTime values. >>>> You can try to run your script only with a « bad » dateTime : >>>> >>>> "SELECT dateTime, Radiation from archive where dateTime = 1592614500 » >>>> >>>> Does the error occurs ? If yes, you can try to add debugging print >>>> commands inside the sunshineThreshold function to try to understand. >>>> >>>> >>>> >>>> Le 30 juin 2022 à 19:51, 'Peter Fletcher' via weewx-user <weewx...@ >>>> googlegroups.com> a écrit : >>>> >>>> It did as it seems you predicted - passed 1592614800 and stopped at >>>> 1632611100. You obviously have a clue as to what is going on. Please >>>> explain! >>>> >>>> On Thursday, June 30, 2022 at 8:59:48 AM UTC-4 [email protected] >>>> wrote: >>>> >>>>> If you exclude the first one,1592614500 , with a query like "SELECT >>>>> dateTime, Radiation from archive where dateTime <> 1592614500", will the >>>>> script stop at 1592614800 ( the next dateTime) or will it continue and >>>>> stop >>>>> at 1632611100 ? >>>>> >>>>> Le 30 juin 2022 à 14:34, 'Peter Fletcher' via weewx-user <weewx...@ >>>>> googlegroups.com> a écrit : >>>>> >>>>> 1592614500 >>>>> 1632611100 >>>>> 1632611400 >>>>> 1647688800 >>>>> >>>>> I can't see a pattern or any common features. >>>>> >>>>> On Thursday, June 30, 2022 at 3:55:49 AM UTC-4 [email protected] >>>>> wrote: >>>>> >>>>>> No, I never had weewx crashes related to the sunshine calculations. >>>>>> >>>>>> What are the dateTime values that trigger the error ? >>>>>> >>>>>> >>>>>> >>>>>> Le mercredi 29 juin 2022 à 23:23:16 UTC+2, Peter Fletcher a écrit : >>>>>> >>>>>>> Have you had any odd weewx errors or crashes related to the sunshine >>>>>>> calculations? I ask because I hadn't, but I decided to try to >>>>>>> 'backfill' my >>>>>>> database with sunshine times, based on the 5-minute radiation values, >>>>>>> and I >>>>>>> ran into a bizarre bug. I used the code shown below (on a copy of my >>>>>>> live >>>>>>> weewx database). As you will see, the threshold calculation code is >>>>>>> essentially identical to yours, except that it has been converted to a >>>>>>> regular function (no 'self' parameter) and my station's latitude and >>>>>>> longitude are hard coded in it. When the code is run under Python 3.9.2 >>>>>>> on >>>>>>> my Pi, it initially runs without problems, but crashes after 8,000+ >>>>>>> records >>>>>>> have been processed with a ValueError on the MaxThreshold vs threshold >>>>>>> comparison, reporting that it can't compare a complex with a float! If >>>>>>> I >>>>>>> intercept and log the errors, it turns out that, for a few specific >>>>>>> values >>>>>>> of dateTime, the function returns a complex number! Even more >>>>>>> bizarrely, it >>>>>>> only seems to do that in the context of the running code. If I manually >>>>>>> run >>>>>>> through all the operations from the function code at the Python command >>>>>>> line, using the value of dateTime that produces the first crash, all >>>>>>> the >>>>>>> intermediate results and the final result are sane floats. >>>>>>> There appears to be a second issue, possibly related to my reading >>>>>>> and writing the database at relatively high frequency, which stalls the >>>>>>> process after about 18,000 records have been processed, but removing >>>>>>> the >>>>>>> database writes allows it to run to completion without abolishing the >>>>>>> consistent, albeit infrequent, ValueErrors. >>>>>>> >>>>>>> [backfill.py] >>>>>>> import sqlite3 >>>>>>> from datetime import datetime >>>>>>> import time >>>>>>> from math import sin, cos, pi, asin >>>>>>> >>>>>>> def sunshineThreshold(mydatetime): >>>>>>> coeff = 0.9 # change to calibrate with your sensor >>>>>>> utcdate = datetime.utcfromtimestamp(mydatetime) >>>>>>> dayofyear = int(time.strftime("%j", time.gmtime(mydatetime))) >>>>>>> theta = 360 * dayofyear / 365 >>>>>>> equatemps = 0.0172 + 0.4281 * cos((pi / 180) * theta) - 7.3515 >>>>>>> * sin( >>>>>>> (pi / 180) * theta) - 3.3495 * cos(2 * (pi / 180) * theta) >>>>>>> - 9.3619 * sin( >>>>>>> 2 * (pi / 180) * theta) >>>>>>> >>>>>>> latitude = 43.0346213 >>>>>>> longitude = -78.689362 >>>>>>> >>>>>>> corrtemps = longitude * 4 >>>>>>> declinaison = asin(0.006918 - 0.399912 * cos((pi / 180) * >>>>>>> theta) + 0.070257 * sin( >>>>>>> (pi / 180) * theta) - 0.006758 * cos(2 * (pi / 180) * >>>>>>> theta) + 0.000908 * sin( >>>>>>> 2 * (pi / 180) * theta)) * (180 / pi) >>>>>>> minutesjour = utcdate.hour * 60 + utcdate.minute >>>>>>> tempsolaire = (minutesjour + corrtemps + equatemps) / 60 >>>>>>> angle_horaire = (tempsolaire - 12) * 15 >>>>>>> hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / >>>>>>> 180) * declinaison) + cos( >>>>>>> (pi / 180) * latitude) * cos((pi / 180) * declinaison) * >>>>>>> cos((pi / 180) * angle_horaire)) * (180 / pi) >>>>>>> seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) >>>>>>> * 1080 * pow( >>>>>>> (sin(pi / 180) * hauteur_soleil), 1.25) * coeff >>>>>>> return seuil >>>>>>> >>>>>>> >>>>>>> database = 'weewx.sdb' >>>>>>> >>>>>>> maxThreshold=0 >>>>>>> count=0 >>>>>>> conn=sqlite3.connect(database) >>>>>>> cur=conn.execute("SELECT dateTime, Radiation from archive") >>>>>>> for row in cur: >>>>>>> count += 1 >>>>>>> if (row[1] is not None) and (row[1] > 20): >>>>>>> threshold = sunshineThreshold(row[0]) >>>>>>> if threshold > maxThreshold: >>>>>>> maxThreshold = threshold >>>>>>> if row[1] > threshold: >>>>>>> conn.execute("UPDATE archive set SunshineTime = 5 WHERE >>>>>>> dateTime = " + str(row[0])) >>>>>>> if count % 1000 == 0: >>>>>>> print(count, 'Max Threshold', maxThreshold) >>>>>>> conn.close >>>>>>> [/backfill.py] >>>>>>> >>>>>>> On Friday, June 10, 2022 at 3:29:40 AM UTC-4 [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> On my side, I have looked at the CPU utilization on my raspberry Pi >>>>>>>> 3B+. I have the mqtt service service installed, so at each loop all >>>>>>>> data >>>>>>>> of the packet are sent to the mqtt broker. >>>>>>>> >>>>>>>> With mqtt and when calculations of the sunshine threshold is done >>>>>>>> for each loop packet, the total CPU utilization of python3 is about >>>>>>>> 0.75% >>>>>>>> With mqtt and without calculation of sunshine threshold : 0.5% of >>>>>>>> total CPU. >>>>>>>> >>>>>>>> So one can estimate that 0.25 % of total CPU is needed for the >>>>>>>> calculation of the threshold value for each LOOP packet. >>>>>>>> >>>>>>>> >>>>>>>> Le 9 juin 2022 à 22:26, 'Peter Fletcher' via weewx-user <weewx...@ >>>>>>>> googlegroups.com> a écrit : >>>>>>>> >>>>>>>> After some experimentation, I found that the radiation value in the >>>>>>>> VP2 LOOP packets does, indeed, normally change every 50-52 seconds, >>>>>>>> but, >>>>>>>> perhaps about a fifth of the 'gaps' are a *multiple* of that time >>>>>>>> - most often 100+ or 150+ seconds, but occasionally more than that (I >>>>>>>> saw >>>>>>>> one 250+ second 'gap'). I saw this under conditions of variable >>>>>>>> sunshine >>>>>>>> and clouds when it seemed unlikely that the actual radiation value >>>>>>>> would >>>>>>>> have been precisely constant for that length of time, so I am not sure >>>>>>>> exactly what is going on. In any event, I am revising the code I am >>>>>>>> using >>>>>>>> on the basis of doing the threshold calculation when the radiation >>>>>>>> level >>>>>>>> changes, but at least every minute, if it remains constant for more >>>>>>>> than >>>>>>>> the normal 50-52 seconds.. >>>>>>>> >>>>>>>> On Sunday, June 5, 2022 at 12:33:47 PM UTC-4 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I think it is also OK to do an average for every 30 seconds. It >>>>>>>>> depends also on the weather station used. >>>>>>>>> For instance, a Davis Vantage Pro 2 ISS transmits an >>>>>>>>> updated solar radiation value every 50 to 60 seconds. So with this >>>>>>>>> weather >>>>>>>>> station, even a 1 minute average would not be very different since >>>>>>>>> anyway >>>>>>>>> the solar radiation values of the LOOP packet are the same for at >>>>>>>>> least 50 >>>>>>>>> seconds.! >>>>>>>>> >>>>>>>>> Le 5 juin 2022 à 18:02, 'Peter Fletcher' via weewx-user <weewx...@ >>>>>>>>> googlegroups.com> a écrit : >>>>>>>>> >>>>>>>>> I chose to average the LOOP radiation readings and only to do the >>>>>>>>> threshold calculation and make the sun/no sun determination every 30 >>>>>>>>> seconds because I thought doing it on every LOOP might overload LOOP >>>>>>>>> processing (I am running weewx on a Pi 3B, which is also doing a few >>>>>>>>> other >>>>>>>>> things which use the CPU). If this is an unnecessary concern, as it >>>>>>>>> may >>>>>>>>> very well be, your modified code is much cleaner than mine. >>>>>>>>> >>>>>>>>> On Saturday, June 4, 2022 at 12:41:08 PM UTC-4 [email protected] >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It is a very good idea to calculate the sunshine duration for >>>>>>>>>> each LOOP packet and sum these values to make the final archive >>>>>>>>>> sunshine >>>>>>>>>> duration. I have modified my script accordingly : >>>>>>>>>> https://github.com/Jterrettaz/sunduration. >>>>>>>>>> The logic is the following : for each received LOOP packet, the >>>>>>>>>> radiation is compared to a calculated threshold. If the radiation is >>>>>>>>>> above >>>>>>>>>> the threshold value, the sunshine time for the LOOP packet is equal >>>>>>>>>> to the >>>>>>>>>> time elapsed between the previous loop packet and this packet (most >>>>>>>>>> of the >>>>>>>>>> time 2 seconds with a Vantage Davis Pro). >>>>>>>>>> The final archive sunshine duration is the sum of all the LOOP >>>>>>>>>> value within the archive period. >>>>>>>>>> Le vendredi 3 juin 2022 à 21:59:36 UTC+2, Peter Fletcher a écrit : >>>>>>>>>> >>>>>>>>>>> That makes some sense when you are getting data from an >>>>>>>>>>> 'external' sensor, though there are (IMHO) simpler ways of doing >>>>>>>>>>> it. weewx >>>>>>>>>>> already has access to the LOOP radiation data from the VP2, so >>>>>>>>>>> handling the >>>>>>>>>>> processing and data storage within weewx makes more sense to me in >>>>>>>>>>> this >>>>>>>>>>> case. >>>>>>>>>>> >>>>>>>>>>> On Friday, June 3, 2022 at 3:24:23 PM UTC-4 vince wrote: >>>>>>>>>>> >>>>>>>>>>>> On Friday, June 3, 2022 at 11:17:00 AM UTC-7 Meteo Oberwallis >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> if the interval of Weewx and the data logger is set to 10 >>>>>>>>>>>>> minutes, I would have liked to read the value of the solar sensor >>>>>>>>>>>>> every >>>>>>>>>>>>> minute and then write it into a separate .sdb database as >>>>>>>>>>>>> possible sunshine. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Personally I'd use an external program called via cron and >>>>>>>>>>>> posting a message to a MQTT topic. Have weewx subscribe to that >>>>>>>>>>>> topic to >>>>>>>>>>>> get the data into your db. >>>>>>>>>>>> >>>>>>>>>>>> This is how I used to get my DS18b20 temperature sensor data >>>>>>>>>>>> into weewx. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe >>>>>>>>> . >>>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>>> [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "weewx-user" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> >>>>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "weewx-user" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "weewx-user" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/c07ed2bb-1e3d-43e2-b08c-08a7a3aa92dbn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/c07ed2bb-1e3d-43e2-b08c-08a7a3aa92dbn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "weewx-user" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/f287c1b3-1005-409c-82a9-a072e375d5e9n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-user/f287c1b3-1005-409c-82a9-a072e375d5e9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "weewx-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/77cd4de6-3b1c-4bc2-9dbc-8d8316bd8a9bn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/77cd4de6-3b1c-4bc2-9dbc-8d8316bd8a9bn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d7887ea5-f97d-4327-8a72-782d701ebd15n%40googlegroups.com.
