On Tuesday, October 15, 2019 at 11:39:03 PM UTC-7, gjr80 wrote:
>
> Hi,
>
> The problem you are experiencing is due an incompatibility between your
> station emitting what is known as partial loop packets (ie not all obs are
> included in all loop packets) and the mode in which you are operating the
> csv extension.
>
> You are running the CSV extension such that it emits loop data to file and
> includes a header line. Subsequent loop packets are appended to the file.
> The header line is only written (1) when the csv output file is first
> created or (2) on each loop packet but only if mode = 'overwrite'. In
> your case you are using 'append' mode so the header line is only written
> when the csv output file is first created. Think now of the situation where
> the first loop packet has, say, observation outTemp but the next loop
> packet does not. So the header line is written and includes 'outTemp' in
> the relevant place. The line of csv data for that packet is written to file
> and each field matches with the header line. When the next loop packet is
> processed there is no outTemp field in the loop packet so nothing is
> written in outTemp's place in the line of csv data (well in fact something
> will be written it will be the next field if there is one). Consequently
> when your parser parses that line it expects to see outTemp data when in
> fact it sees the next value instead (perhaps it was outTempBatteryStatus)
> and hence you get nonsense values from your parser.
>
Thank you very much for this explanation. I am familiar with this idea but
It seems the behavior changed between 3.5.0 and 3.9.2. It used to write
each header line when mode was 'append'. Since part of the upgrade was
moving weewx to another server, I still have the old server's data and
binaries available. If you look in the weewx.csv for the 3.5.0 server:
dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,maxSolarRad,outHumidity,outTemp,rainRate,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windSpeed,windchill
1571075730,None,71.4621394073,None,None,3276.5581345,56.6531442082,70.3,76.0399971396,None,None,62,70.3,0,3,100.0,0,1228,0,1,1.65011333748,70.3
dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,maxSolarRad,rain,rainRate,rain_total,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windDir,windSpeed,windchill
1571075748,None,None,None,None,None,None,None,None,None,None,0.0,0,110.4646,3,100.0,0,1228,0,1,180.0,2.16448441021,None
dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,inTemp,maxSolarRad,outHumidity,outTemp,pressure,rainRate,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windSpeed,windchill
1571075766,29.9423211803,71.4621394073,29.9470837144,None,3276.5581345,56.6531442082,70.3,76.0399971396,None,74.39,None,62,70.3,29.7622563497,0,3,100.0,0,1228,0,1,1.65011333748,70.3
...it's writing a header line each time, even in 'append' mode. My config
for that server was:
[CSV]
filename = /var/db/weewx/weewx.csv
header = true
So it's clear that something changed to exhibit the behavior you describe
above instead of the old behavior.
> Solutions, there are a few possibilities:
>
> 1. use 'overwrite' mode instead of append, in this mode the header line is
> written each time a packet is processed but there is only ever one line in
> your csv output file
> 2. delete the csv output file after each loop packet is read by your
> parser (this is a pretty lame solution)
> 3. rewrite the csv extension to cache loop packet data so that a
> 'complete' loop packet is written each time
> 4. emit archive record data rather than loop packet data
>
IMO option 3 provides the best solution as there's zero way to parse a
partial packet without keys for the values. ;) This would also restore my
nagios plugin's proper functionality. Unfortunately, I've too little
experience in python to attempt 3 in a public software repo.
Those other solutions, for various reasons, are non-optimal. 1 and 2 will
suffer from the lack of caching of data. 4 makes some sense, but for this
comment:
# If the station hardware supports data logging then the archive
interval
# will be downloaded from the station. Otherwise, specify it (in
seconds).
archive_interval = 300
I have a need for immediate data. Not knowing what my archive interval will
be is a deal breaker.
Sadly, it seems I will have to use weewx within my own script and parse
it's stdout myself. Thanks for all the insight and responses. :D
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/14fdcabf-c513-4efe-8e2a-ddc63bd95cae%40googlegroups.com.
