Can't comment on what changed on your install from 3.5.0 to 3.9.2. The only
WeeWX code that could affect the csv output would be the AcuRite driver,
and I see there have been no changes to this driver since the 3.5.0 release
that could account for the changed behaviour. Perhaps you are using a
different csv extension version. The logic for creating the csv output is
wholly within the csv extension.
In any case, given the old sample output below you could mimic that output
by making a simple alteration to the csv extension code. In csv.py you
could change (untested):
header = None
if self.emit_header and (
not os.path.exists(filename) or flag == "w"):
header = '# %s\n' % ','.join(self.sort_keys(data))
to
header = '# %s\n' % ','.join(self.sort_keys(data))
be aware that this change will mean that the header is always written
before each loop packet is written, none of the config options in
weewx.conf will change this. Of course the other issue is that you now have
an orphan version of the csv extension and any subsequent upgrades/installs
of the extension will overwrite your changes.
Gary
On Thursday, 17 October 2019 02:03:41 UTC+10, Dave wrote:
>
> On Tuesday, October 15, 2019 at 11:39:03 PM UTC-7, gjr80 wrote:
>>
>> Hi,
>>
>> The problem you are experiencing is due an incompatibility between your
>> station emitting what is known as partial loop packets (ie not all obs are
>> included in all loop packets) and the mode in which you are operating the
>> csv extension.
>>
>> You are running the CSV extension such that it emits loop data to file
>> and includes a header line. Subsequent loop packets are appended to the
>> file. The header line is only written (1) when the csv output file is first
>> created or (2) on each loop packet but only if mode = 'overwrite'. In
>> your case you are using 'append' mode so the header line is only written
>> when the csv output file is first created. Think now of the situation where
>> the first loop packet has, say, observation outTemp but the next loop
>> packet does not. So the header line is written and includes 'outTemp' in
>> the relevant place. The line of csv data for that packet is written to file
>> and each field matches with the header line. When the next loop packet is
>> processed there is no outTemp field in the loop packet so nothing is
>> written in outTemp's place in the line of csv data (well in fact something
>> will be written it will be the next field if there is one). Consequently
>> when your parser parses that line it expects to see outTemp data when in
>> fact it sees the next value instead (perhaps it was outTempBatteryStatus)
>> and hence you get nonsense values from your parser.
>>
>
> Thank you very much for this explanation. I am familiar with this idea but
> It seems the behavior changed between 3.5.0 and 3.9.2. It used to write
> each header line when mode was 'append'. Since part of the upgrade was
> moving weewx to another server, I still have the old server's data and
> binaries available. If you look in the weewx.csv for the 3.5.0 server:
>
>
> dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,maxSolarRad,outHumidity,outTemp,rainRate,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windSpeed,windchill
>
> 1571075730,None,71.4621394073,None,None,3276.5581345,56.6531442082,70.3,76.0399971396,None,None,62,70.3,0,3,100.0,0,1228,0,1,1.65011333748,70.3
>
> dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,maxSolarRad,rain,rainRate,rain_total,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windDir,windSpeed,windchill
>
> 1571075748,None,None,None,None,None,None,None,None,None,None,0.0,0,110.4646,3,100.0,0,1228,0,1,180.0,2.16448441021,None
>
> dateTime,altimeter,appTemp,barometer,channel,cloudbase,dewpoint,heatindex,humidex,inDewpoint,inTemp,maxSolarRad,outHumidity,outTemp,pressure,rainRate,rssi,rxCheckPercent,sensor_battery,sensor_id,txTempBatteryStatus,usUnits,windSpeed,windchill
>
> 1571075766,29.9423211803,71.4621394073,29.9470837144,None,3276.5581345,56.6531442082,70.3,76.0399971396,None,74.39,None,62,70.3,29.7622563497,0,3,100.0,0,1228,0,1,1.65011333748,70.3
>
>
> ...it's writing a header line each time, even in 'append' mode. My config
> for that server was:
>
> [CSV]
> filename = /var/db/weewx/weewx.csv
> header = true
>
>
> So it's clear that something changed to exhibit the behavior you describe
> above instead of the old behavior.
>
>
>> Solutions, there are a few possibilities:
>>
>> 1. use 'overwrite' mode instead of append, in this mode the header line
>> is written each time a packet is processed but there is only ever one line
>> in your csv output file
>> 2. delete the csv output file after each loop packet is read by your
>> parser (this is a pretty lame solution)
>> 3. rewrite the csv extension to cache loop packet data so that a
>> 'complete' loop packet is written each time
>> 4. emit archive record data rather than loop packet data
>>
>
> IMO option 3 provides the best solution as there's zero way to parse a
> partial packet without keys for the values. ;) This would also restore my
> nagios plugin's proper functionality. Unfortunately, I've too little
> experience in python to attempt 3 in a public software repo.
>
> Those other solutions, for various reasons, are non-optimal. 1 and 2 will
> suffer from the lack of caching of data. 4 makes some sense, but for this
> comment:
>
> # If the station hardware supports data logging then the archive
> interval
> # will be downloaded from the station. Otherwise, specify it (in
> seconds).
> archive_interval = 300
>
>
> I have a need for immediate data. Not knowing what my archive interval
> will be is a deal breaker.
>
> Sadly, it seems I will have to use weewx within my own script and parse
> it's stdout myself. Thanks for all the insight and responses. :D
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/7b2b6319-d2d5-43cc-ab69-3afe12ccf119%40googlegroups.com.
