If you exclude the first one,1592614500 , with a query like "SELECT dateTime, Radiation from archive where dateTime <> 1592614500", will the script stop at 1592614800 ( the next dateTime) or will it continue and stop at 1632611100 ?
> Le 30 juin 2022 à 14:34, 'Peter Fletcher' via weewx-user > <[email protected]> a écrit : > > 1592614500 > 1632611100 > 1632611400 > 1647688800 > > I can't see a pattern or any common features. > > On Thursday, June 30, 2022 at 3:55:49 AM UTC-4 [email protected] > <http://gmail.com/> wrote: > No, I never had weewx crashes related to the sunshine calculations. > > What are the dateTime values that trigger the error ? > > > > Le mercredi 29 juin 2022 à 23:23:16 UTC+2, Peter Fletcher a écrit : > Have you had any odd weewx errors or crashes related to the sunshine > calculations? I ask because I hadn't, but I decided to try to 'backfill' my > database with sunshine times, based on the 5-minute radiation values, and I > ran into a bizarre bug. I used the code shown below (on a copy of my live > weewx database). As you will see, the threshold calculation code is > essentially identical to yours, except that it has been converted to a > regular function (no 'self' parameter) and my station's latitude and > longitude are hard coded in it. When the code is run under Python 3.9.2 on my > Pi, it initially runs without problems, but crashes after 8,000+ records have > been processed with a ValueError on the MaxThreshold vs threshold comparison, > reporting that it can't compare a complex with a float! If I intercept and > log the errors, it turns out that, for a few specific values of dateTime, the > function returns a complex number! Even more bizarrely, it only seems to do > that in the context of the running code. If I manually run through all the > operations from the function code at the Python command line, using the value > of dateTime that produces the first crash, all the intermediate results and > the final result are sane floats. > There appears to be a second issue, possibly related to my reading and > writing the database at relatively high frequency, which stalls the process > after about 18,000 records have been processed, but removing the database > writes allows it to run to completion without abolishing the consistent, > albeit infrequent, ValueErrors. > > [backfill.py] > import sqlite3 > from datetime import datetime > import time > from math import sin, cos, pi, asin > > def sunshineThreshold(mydatetime): > coeff = 0.9 # change to calibrate with your sensor > utcdate = datetime.utcfromtimestamp(mydatetime) > dayofyear = int(time.strftime("%j", time.gmtime(mydatetime))) > theta = 360 * dayofyear / 365 > equatemps = 0.0172 + 0.4281 * cos((pi / 180) * theta) - 7.3515 * sin( > (pi / 180) * theta) - 3.3495 * cos(2 * (pi / 180) * theta) - 9.3619 * > sin( > 2 * (pi / 180) * theta) > > latitude = 43.0346213 > longitude = -78.689362 > > corrtemps = longitude * 4 > declinaison = asin(0.006918 - 0.399912 * cos((pi / 180) * theta) + > 0.070257 * sin( > (pi / 180) * theta) - 0.006758 * cos(2 * (pi / 180) * theta) + > 0.000908 * sin( > 2 * (pi / 180) * theta)) * (180 / pi) > minutesjour = utcdate.hour * 60 + utcdate.minute > tempsolaire = (minutesjour + corrtemps + equatemps) / 60 > angle_horaire = (tempsolaire - 12) * 15 > hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / 180) * > declinaison) + cos( > (pi / 180) * latitude) * cos((pi / 180) * declinaison) * cos((pi / > 180) * angle_horaire)) * (180 / pi) > seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) * 1080 * > pow( > (sin(pi / 180) * hauteur_soleil), 1.25) * coeff > return seuil > > > database = 'weewx.sdb' > > maxThreshold=0 > count=0 > conn=sqlite3.connect(database) > cur=conn.execute("SELECT dateTime, Radiation from archive") > for row in cur: > count += 1 > if (row[1] is not None) and (row[1] > 20): > threshold = sunshineThreshold(row[0]) > if threshold > maxThreshold: > maxThreshold = threshold > if row[1] > threshold: > conn.execute("UPDATE archive set SunshineTime = 5 WHERE dateTime = " > + str(row[0])) > if count % 1000 == 0: > print(count, 'Max Threshold', maxThreshold) > conn.close > [/backfill.py] > > On Friday, June 10, 2022 at 3:29:40 AM UTC-4 [email protected] <> wrote: > On my side, I have looked at the CPU utilization on my raspberry Pi 3B+. I > have the mqtt service service installed, so at each loop all data of the > packet are sent to the mqtt broker. > > With mqtt and when calculations of the sunshine threshold is done for each > loop packet, the total CPU utilization of python3 is about 0.75% > With mqtt and without calculation of sunshine threshold : 0.5% of total CPU. > > So one can estimate that 0.25 % of total CPU is needed for the calculation of > the threshold value for each LOOP packet. > > > >> Le 9 juin 2022 à 22:26, 'Peter Fletcher' via weewx-user >> <[email protected] <>> a écrit : >> > >> After some experimentation, I found that the radiation value in the VP2 LOOP >> packets does, indeed, normally change every 50-52 seconds, but, perhaps >> about a fifth of the 'gaps' are a multiple of that time - most often 100+ or >> 150+ seconds, but occasionally more than that (I saw one 250+ second 'gap'). >> I saw this under conditions of variable sunshine and clouds when it seemed >> unlikely that the actual radiation value would have been precisely constant >> for that length of time, so I am not sure exactly what is going on. In any >> event, I am revising the code I am using on the basis of doing the threshold >> calculation when the radiation level changes, but at least every minute, if >> it remains constant for more than the normal 50-52 seconds.. >> >> On Sunday, June 5, 2022 at 12:33:47 PM UTC-4 [email protected] >> <http://gmail.com/> wrote: >> I think it is also OK to do an average for every 30 seconds. It depends >> also on the weather station used. >> For instance, a Davis Vantage Pro 2 ISS transmits an updated solar >> radiation value every 50 to 60 seconds. So with this weather station, even a >> 1 minute average would not be very different since anyway the solar >> radiation values of the LOOP packet are the same for at least 50 seconds.! >> >> >>> Le 5 juin 2022 à 18:02, 'Peter Fletcher' via weewx-user >>> <[email protected] <>> a écrit : >>> >> >>> I chose to average the LOOP radiation readings and only to do the threshold >>> calculation and make the sun/no sun determination every 30 seconds because >>> I thought doing it on every LOOP might overload LOOP processing (I am >>> running weewx on a Pi 3B, which is also doing a few other things which use >>> the CPU). If this is an unnecessary concern, as it may very well be, your >>> modified code is much cleaner than mine. >>> >>> On Saturday, June 4, 2022 at 12:41:08 PM UTC-4 [email protected] >>> <http://gmail.com/> wrote: >>> It is a very good idea to calculate the sunshine duration for each LOOP >>> packet and sum these values to make the final archive sunshine duration. I >>> have modified my script accordingly : >>> https://github.com/Jterrettaz/sunduration >>> <https://github.com/Jterrettaz/sunduration>. >>> The logic is the following : for each received LOOP packet, the radiation >>> is compared to a calculated threshold. If the radiation is above the >>> threshold value, the sunshine time for the LOOP packet is equal to the time >>> elapsed between the previous loop packet and this packet (most of the time >>> 2 seconds with a Vantage Davis Pro). >>> The final archive sunshine duration is the sum of all the LOOP value within >>> the archive period. >>> Le vendredi 3 juin 2022 à 21:59:36 UTC+2, Peter Fletcher a écrit : >>> That makes some sense when you are getting data from an 'external' sensor, >>> though there are (IMHO) simpler ways of doing it. weewx already has access >>> to the LOOP radiation data from the VP2, so handling the processing and >>> data storage within weewx makes more sense to me in this case. >>> >>> On Friday, June 3, 2022 at 3:24:23 PM UTC-4 vince wrote: >>> On Friday, June 3, 2022 at 11:17:00 AM UTC-7 Meteo Oberwallis wrote: >>> if the interval of Weewx and the data logger is set to 10 minutes, I would >>> have liked to read the value of the solar sensor every minute and then >>> write it into a separate .sdb database as possible sunshine. >>> >>> Personally I'd use an external program called via cron and posting a >>> message to a MQTT topic. Have weewx subscribe to that topic to get the >>> data into your db. >>> >>> This is how I used to get my DS18b20 temperature sensor data into weewx. >>> >>> >> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "weewx-user" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe >>> <https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "weewx-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe >> <https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "weewx-user" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe > <https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com > > <https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/22566406-EC9E-485D-9177-1A9F8300794C%40gmail.com.
