It's probably not too bad to fork the extension and add the CachedValues method as seen in the restx extension <https://github.com/weewx/weewx/blob/master/bin/weewx/restx.py#L652-L679> (read: copy/paste). Once tested and working this should satisfy option 3.
Other important information found for CachedValues are here <https://github.com/weewx/weewx/blob/master/bin/weewx/restx.py#L652-L679>, and here <https://github.com/weewx/weewx/blob/master/bin/weewx/restx.py#L636-L645>. You can see on every loop it updates the cached dictionary of those values with the update() function. Then it outputs the observations that just came in, plus the missing ones from Cached values, to the LOOP with the get_packet() function. On Wednesday, October 16, 2019 at 12:03:41 PM UTC-4, Dave wrote: > > On Tuesday, October 15, 2019 at 11:39:03 PM UTC-7, gjr80 wrote: >> >> Hi, >> >> The problem you are experiencing is due an incompatibility between your >> station emitting what is known as partial loop packets (ie not all obs are >> included in all loop packets) and the mode in which you are operating the >> csv extension. >> >> You are running the CSV extension such that it emits loop data to file >> and includes a header line. Subsequent loop packets are appended to the >> file. The header line is only written (1) when the csv output file is first >> created or (2) on each loop packet but only if mode = 'overwrite'. In >> your case you are using 'append' mode so the header line is only written >> when the csv output file is first created. Think now of the situation where >> the first loop packet has, say, observation outTemp but the next loop >> packet does not. So the header line is written and includes 'outTemp' in >> the relevant place. The line of csv data for that packet is written to file >> and each field matches with the header line. When the next loop packet is >> processed there is no outTemp field in the loop packet so nothing is >> written in outTemp's place in the line of csv data (well in fact something >> will be written it will be the next field if there is one). Consequently >> when your parser parses that line it expects to see outTemp data when in >> fact it sees the next value instead (perhaps it was outTempBatteryStatus) >> and hence you get nonsense values from your parser. >> > > Thank you very much for this explanation. I am familiar with this idea but > It seems the behavior changed between 3.5.0 and 3.9.2. It used to write > each header line when mode was 'append'. Since part of the upgrade was > moving weewx to another server, I still have the old server's data and > binaries available. If you look in the weewx.csv for the 3.5.0 server: > > > dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,maxSolarRad,outHumidity,outTemp,rainRate,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windSpeed,windchill > > 1571075730,None,71.4621394073,None,None,3276.5581345,56.6531442082,70.3,76.0399971396,None,None,62,70.3,0,3,100.0,0,1228,0,1,1.65011333748,70.3 > > dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,maxSolarRad,rain,rainRate,rain_total,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windDir,windSpeed,windchill > > 1571075748,None,None,None,None,None,None,None,None,None,None,0.0,0,110.4646,3,100.0,0,1228,0,1,180.0,2.16448441021,None > > dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,inTemp,maxSolarRad,outHumidity,outTemp,pressure,rainRate,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windSpeed,windchill > > 1571075766,29.9423211803,71.4621394073,29.9470837144,None,3276.5581345,56.6531442082,70.3,76.0399971396,None,74.39,None,62,70.3,29.7622563497,0,3,100.0,0,1228,0,1,1.65011333748,70.3 > > > ...it's writing a header line each time, even in 'append' mode. My config > for that server was: > > [CSV] > filename = /var/db/weewx/weewx.csv > header = true > > > So it's clear that something changed to exhibit the behavior you describe > above instead of the old behavior. > > >> Solutions, there are a few possibilities: >> >> 1. use 'overwrite' mode instead of append, in this mode the header line >> is written each time a packet is processed but there is only ever one line >> in your csv output file >> 2. delete the csv output file after each loop packet is read by your >> parser (this is a pretty lame solution) >> 3. rewrite the csv extension to cache loop packet data so that a >> 'complete' loop packet is written each time >> 4. emit archive record data rather than loop packet data >> > > IMO option 3 provides the best solution as there's zero way to parse a > partial packet without keys for the values. ;) This would also restore my > nagios plugin's proper functionality. Unfortunately, I've too little > experience in python to attempt 3 in a public software repo. > > Those other solutions, for various reasons, are non-optimal. 1 and 2 will > suffer from the lack of caching of data. 4 makes some sense, but for this > comment: > > # If the station hardware supports data logging then the archive > interval > # will be downloaded from the station. Otherwise, specify it (in > seconds). > archive_interval = 300 > > > I have a need for immediate data. Not knowing what my archive interval > will be is a deal breaker. > > Sadly, it seems I will have to use weewx within my own script and parse > it's stdout myself. Thanks for all the insight and responses. :D > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/01d0012e-8dce-4f20-87ca-2a81351e17a8%40googlegroups.com.
