Glenn,
I have created a new branch wee_import_text off development for this and
have modified the wee_import code to import text to a WeeWX TEXT field.
There are a dew conditions:
1. text import only works with CSV imports
2. import source fields that are to be imported as text must use 'text' as
the WeeWX units specifier in the field map in the import config file, eg:
[[FieldMap]]
....
aComment = import_text_field, text
3. the destination text field has been added to the WeeWX database and
WeeWX configured as per the Storing text in the database
<https://github.com/weewx/weewx/wiki/Storing-text-in-the-database> wiki page
I have only done basic testing of the code so there may be some corner
cases I have not covered, but you should be right to give it a go if you
want. As I said I have taken a branch off development, since there has been
quite a few changes in development since the last release the most reliable
option is for you to clone and install wee_import_text. However, the
wee_import code does not change often so you could probably get away with
running v4.5.1 and just downloading and installing the modified weeimport.py
.
To use the new wee_import:
1. Either clone and install from the wee_import_text branch or if using
v4.5.1 (or any other recent version at your own peril) replace
BIN/weeimport/weeimport.py with weeimport.py from the wee_import_text
branch. You can download weeimport.py from the wee_import_text branch
directly using wget as follows:
$ wget
https://raw.githubusercontent.com/weewx/weewx/wee_import_text/bin/weeimport/weeimport.py
2. edit your import config file and add your text field import to the field
map as outlined above
3. stop WeeWX and run your import using --dry-run and then without --dry-run
Let me know if you have any issues.
Gary
On Tuesday, 18 January 2022 at 10:17:56 UTC+10 Glenn McKechnie wrote:
> Thanks Gary, I appreciate it.
>
> Hopefully UPDATE won't raise any serious weewx issues.
> I look forward to seeing it raised as an issue :)
>
> On 18/01/2022, gjr80 <[email protected]> wrote:
> > Correct, wee_import will not accept text only fields. When we put
> together
> > wee_import we only included support for numeric fields; text fields were
> > omitted, perhaps not intentionally, but more likely because no real
> > consideration had been given to including them in the schema. That being
> > said, it should be a fairly simple process to modify wee_import to
> support
> > text fields, at least for CSV imports anyway. Having had a quick
> wee_import
> >
> > refresher this morning I am thinking we could support something like
> > ANComment
> > = ANComment, text in the field map. Let me look at this further and see
> > what I can come up with.
> >
> > As for supporting UPDATE, that will require further thought as I suspect
> > that will introduce a number of wider issues. I'll probably raise an
> issue
> > for that one.
> >
> > Gary
> > On Monday, 17 January 2022 at 23:14:00 UTC+10 Glenn McKechnie wrote:
> >
> >> Working with TEXT records.
> >>
> >> I've gone through the wiki entry
> >> https://github.com/weewx/weewx/wiki/Storing-text-in-the-database
> >> and that all works well. I can add TEXT to the database and display it
> >> in the weewx skin. So far so good.
> >>
> >> Using a modified filepile.py, I can add what I want via the new_archive
> >> record hook. What I can't seem to do is add historical data that is in
> >> TEXT form, via weewx. Fair enough, it makes sense.
> >>
> >> I figured wee_import may be a solution so I crafted a csv file and used
> >> wee_import but it baulks on the text field. The csv files,
> >> /var/tmp/weewxaddnotes.csv has the contents...
> >>
> >> timestamp,extraTemp1,extraTemp2,ANComment
> >> 1641729300,23.0,22.0,"Test.\\nThen test again"
> >>
> >> It appears to be wanting a weewx_unit_name and I'm missing a clue stick.
> >> I can't find a name that it will accept without error. If the
> >> weewx_unit_name field is left blank (only the csv_field_name is given)
> >> it gives the following.
> >>
> >> Obtaining raw import data for period 1 ...
> >> **** Unable to parse source-to-WeeWX field map.
> >> **** No units specified for source field 'ANComment' in
> >> /home/weewx/csv.conf.
> >> **** Nothing done, exiting.
> >>
> >> Which makes sense as I had the following...
> >>
> >> [[FieldMap]]
> >> dateTime = timestamp, unix_epoch
> >> interval =
> >> barometer =
> >> pressure =
> >> altimeter =
> >> inTemp =
> >> outTemp =
> >> extraTemp1 = extraTemp1, degree_C
> >> extraTemp2 = extraTemp2, degree_C
> >> ANComment = ANComment
> >>
> >> If I tell wee_imports to ignore the ANComment entry, as follows...
> >> ANComment =
> >> then everything else imports as expected so it's falling over on the
> >> ANComment field
> >>
> >> ignore_invalid_data in the csv.conf file is tantalizing close (a numeric
> >> field containing non-numeric data) but I don't want it skipped or
> >> halted. I want it accepted.
> >>
> >> Can I write anything into that 2nd field that will allow a TEXT entry to
> >> skip the conversion process, and allow it to be written as-is to the
> >> database?
> >>
> >> On reflection, even if I can use wee_import to add the data I want. I
> >> need to get it correct and complete the first time as once a RECORD is
> >> written (and all it's fields filled) then I can't update it later
> >> without going the manual route as weewx appears to be missing an inbuilt
> >> sql UPDATE archive method. That or I can't find it.
> >>
> >> And, that's probably so we don't shoot ourselves in the foot.
> >>
> >> But it sure would be nice to write a historical record using weewx or
> >> preferably wee_import (with all their built in sanity checks) and then,
> >> if needed later, be able to update any missing fields, or correct errant
> >> values without going back to the command line.
> >>
> >> I understand the risk of blindly updating existing records, but in this
> >> case they are all known empty or new fields, and if they do have
> >> something in them then it needs changing anyway!
> >> Caveat Emptor or something along those lines! :-)
> >>
> >>
> >> As an aside. I'm doing this as I have a whole host of manual rain
> >> records that should be entered as they are sanity checks on the
> >> automatic gauges, weewx seems as good a place as any to store them.
> >> At the same time I'll be adding extraTemp values that I've recorded
> >> from the haystack.
> >> It'd be nice to plot them, and change them, as they use existing
> >> fields - extraTemp1 etc.
> >>
> >> Along with that, a TEXT comment as to why some readings appear as they
> >> do would be useful, or where the floods were after the recent epic
> >> rainfall, or why the rain values appear to be stuffed, or which area of
> >> the haystack the probe is at.
> >> The latest Tongan earthquake is probably another example given the
> >> recent barometric burps it caused.
> >>
> >> The possibilities expand as I write.
> >> So I better go.
> >>
> >> But... In summary.
> >>
> >> wee_import appears to baulk on the new add TEXT method. Is there a quick
> >> fix?
> >>
> >> The ability to sql UPDATE any historical weewx record would be useful. 1
> >> or many RECORD fields at a time.
> >>
> >> Individual confirmations not required. Other than accepting a one-time
> >> answer that; 'yes', I will bandage my own toes and blame no-one else if
> >> it backfires, because we always have our backups.
> >>
> >> Ha! Perhaps a csv.conf entry named pieces_are_all_yours with a default
> >> of no? Changing to 'yes' will 'turn on' the UPDATE method. ;-)
> >>
> >>
> >> --
> >>
> >> Cheers
> >> Glenn
> >>
> >> rorpi - read only raspberry pi & various weewx addons
> >> https://github.com/glennmckechnie
> >>
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "weewx-user" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/weewx-user/75ec1eac-bd90-402d-a450-452ab9e4a960n%40googlegroups.com
> .
> >
>
>
> --
>
>
> Cheers
> Glenn
>
> rorpi - read only raspberry pi & various weewx addons
> https://github.com/glennmckechnie
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/9529a74b-7c3e-4c62-afa2-14f36c1dd935n%40googlegroups.com.
