Working with TEXT records.
I've gone through the wiki entry
https://github.com/weewx/weewx/wiki/Storing-text-in-the-database
and that all works well. I can add TEXT to the database and display it
in the weewx skin. So far so good.
Using a modified filepile.py, I can add what I want via the new_archive
record hook. What I can't seem to do is add historical data that is in
TEXT form, via weewx. Fair enough, it makes sense.
I figured wee_import may be a solution so I crafted a csv file and used
wee_import but it baulks on the text field. The csv files,
/var/tmp/weewxaddnotes.csv has the contents...
timestamp,extraTemp1,extraTemp2,ANComment
1641729300,23.0,22.0,"Test.\\nThen test again"
It appears to be wanting a weewx_unit_name and I'm missing a clue stick.
I can't find a name that it will accept without error. If the
weewx_unit_name field is left blank (only the csv_field_name is given)
it gives the following.
Obtaining raw import data for period 1 ...
**** Unable to parse source-to-WeeWX field map.
**** No units specified for source field 'ANComment' in /home/weewx/csv.conf.
**** Nothing done, exiting.
Which makes sense as I had the following...
[[FieldMap]]
dateTime = timestamp, unix_epoch
interval =
barometer =
pressure =
altimeter =
inTemp =
outTemp =
extraTemp1 = extraTemp1, degree_C
extraTemp2 = extraTemp2, degree_C
ANComment = ANComment
If I tell wee_imports to ignore the ANComment entry, as follows...
ANComment =
then everything else imports as expected so it's falling over on the
ANComment field
ignore_invalid_data in the csv.conf file is tantalizing close (a numeric
field containing non-numeric data) but I don't want it skipped or
halted. I want it accepted.
Can I write anything into that 2nd field that will allow a TEXT entry to
skip the conversion process, and allow it to be written as-is to the
database?
On reflection, even if I can use wee_import to add the data I want. I
need to get it correct and complete the first time as once a RECORD is
written (and all it's fields filled) then I can't update it later
without going the manual route as weewx appears to be missing an inbuilt
sql UPDATE archive method. That or I can't find it.
And, that's probably so we don't shoot ourselves in the foot.
But it sure would be nice to write a historical record using weewx or
preferably wee_import (with all their built in sanity checks) and then,
if needed later, be able to update any missing fields, or correct errant
values without going back to the command line.
I understand the risk of blindly updating existing records, but in this
case they are all known empty or new fields, and if they do have
something in them then it needs changing anyway!
Caveat Emptor or something along those lines! :-)
As an aside. I'm doing this as I have a whole host of manual rain
records that should be entered as they are sanity checks on the
automatic gauges, weewx seems as good a place as any to store them.
At the same time I'll be adding extraTemp values that I've recorded
from the haystack.
It'd be nice to plot them, and change them, as they use existing
fields - extraTemp1 etc.
Along with that, a TEXT comment as to why some readings appear as they
do would be useful, or where the floods were after the recent epic
rainfall, or why the rain values appear to be stuffed, or which area of
the haystack the probe is at.
The latest Tongan earthquake is probably another example given the
recent barometric burps it caused.
The possibilities expand as I write.
So I better go.
But... In summary.
wee_import appears to baulk on the new add TEXT method. Is there a quick
fix?
The ability to sql UPDATE any historical weewx record would be useful. 1
or many RECORD fields at a time.
Individual confirmations not required. Other than accepting a one-time
answer that; 'yes', I will bandage my own toes and blame no-one else if
it backfires, because we always have our backups.
Ha! Perhaps a csv.conf entry named pieces_are_all_yours with a default
of no? Changing to 'yes' will 'turn on' the UPDATE method. ;-)
--
Cheers
Glenn
rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/20220117131349.322f6yi3nbpn6bcq%40gmail.com.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM CSV FILES
#
# Copyright (c) 2009-2019 Tom Keffer <[email protected]> and Gary Roderick.
# See the file LICENSE.txt for your rights.
##############################################################################
# Specify the source. Available options are:
# CSV - import obs from a single CSV format file
# WU - import obs from a Weather Underground PWS history
# Cumulus - import obs from a one or more Cumulus monthly log files
# WD - import obs from a one or more WD monthly log files
# Format is:
# source = (CSV | WU | Cumulus)
source = CSV
##############################################################################
[CSV]
# Parameters used when importing from a CSV file
# Path and name of our CSV source file. Format is:
# file = full path and filename
file = /var/tmp/weewxaddnotes.csv
# The character used to separate fields. Format is:
# delimiter = <single character>
# Default is , (comma).
delimiter = ','
# If there is no mapped interval field how will the interval field be
# determined for the imported records. Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
#
# Note: If there is a mapped interval field then this setting will be
# ignored.
# Format is:
# interval = (derive | conf | x)
#!#!# interval = derive
interval = conf
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = False
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
#calc_missing = True
calc_missing = True
# Specify how imported data fields that contain invalid data (eg a numeric
# field containing non-numeric data) are handled. Available options are:
# True - The invalid data is ignored, the WeeWX target field is set to
# None and the import continues.
# False - The import is halted.
# Format is:
# ignore_invalid_data = (True | False)
# Default is True.
ignore_invalid_data = True
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a CSV import UV_sensor should be set to False if a UV sensor was
# NOT present when the import data was created. Otherwise it may be set to
# True or omitted. Format is:
# UV_sensor = (True | False)
UV_sensor = False
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a CSV import solar_sensor should be set to False if a solar radiation
# sensor was NOT present when the import data was created. Otherwise it may
# be set to True or omitted. Format is:
# solar_sensor = (True | False)
solar_sensor = False
# Date-time format of CSV field from which the WeeWX archive record
# dateTime field is to be extracted. wee_import first attempts to interpret
# date/time info in this format, if this fails it then attempts to
# interpret it as a timestamp and if this fails it then raises an error.
# Uses Python strptime() format codes.
# raw_datetime_format = Python strptime() format string
raw_datetime_format = %Y-%m-%d %H:%M:%S
# Does the imported rain field represent the total rainfall since the last
# record or a cumulative value. Available options are:
# discrete - rain field represents total rainfall since last record
# cumulative - rain field represents a cumulative rainfall reset at
# midnight
# rain = (discrete | cumulative)
rain = cumulative
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the WeeWX limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
# where :
# lower is the lower limit of acceptable wind direction in degrees
# (may be negative)
# upper is the upper limit of acceptable wind direction in degrees
#
# Imported values from lower to upper will be normalised to the range 0 to
# 360. Values outside of the parameter range will be stored as None.
# Default is -360,360.
wind_direction = -360,360
# Map CSV record fields to WeeWX archive fields. Format is:
#
# weewx_archive_field_name = csv_field_name, weewx_unit_name
#
# where:
# weewx_archive_field_name - An observation name in the WeeWX database
# schema.
# csv_field_name - The name of a field from the CSV file.
# weewx_unit_name - The name of the units, as defined in WeeWX,
# used by csv_field_name. wee_import will do
# the necessary conversions to the unit system
# used by the WeeWX archive.
# For example,
# outTemp = Temp, degree_C
# would map the CSV field Temp, in degrees C, to the archive field outTemp.
#
# A mapping for WeeWX field dateTime is mandatory and the WeeWX unit name
# for the dateTime mapping must be unix_epoch. For example,
# dateTime = csv_date_and_time, unix_epoch
# would map the CSV field csv_date_and_time to the WeeWX dateTime field with
# the csv_date_and_time field being interpreted first using the format
# specified at the raw_datetime_format config option and if that fails as a
# unix epoch timestamp.
#
# Field mapping to the WeeWX usUnits archive field is currently not
# supported. If a usUnits field exists in the CSV data it should not be
# mapped, rather WeeWX unit names should included against each field to be
# imported in the field map.
#
# WeeWX archive fields that do not exist in the CSV data may be omitted.
# Any omitted fields that are derived (eg dewpoint) may be calculated
# during import using the equivalent of the WeeWX StdWXCalculate service
# through setting the calc-missing parameter above.
[[FieldMap]]
dateTime = timestamp, unix_epoch
interval =
barometer =
pressure =
altimeter =
inTemp =
outTemp =
extraTemp1 = extraTemp1, degree_C
extraTemp2 = extraTemp2, degree_C
ANComment = ANComment
inHumidity =
outHumidity =
windSpeed =
windDir =
windGust =
windGustDir =
rainRate =
rain =
dewpoint =
windchill =
heatindex =
ET =
radiation =
UV =
timestamp,extraTemp1,extraTemp2,ANComment
1641729300,23.0,22.0,"Test.\\nThen test again"
