Gary
Thanks for looking into the issue. Not in the area right now but later
this evening will check out the datebase.
Thanks
Rich
On Monday, January 29, 2018 at 7:21:32 AM UTC-5, gjr80 wrote:
> Rich,
>
> I sat down and imported the Sep17 log you posted under weeWX 3.8.0 and it
> imported 3877 records without any error. The Cumulus monthly log file uses
> a fixed format date and time fields as the first 2 fields of the line; the
> formats are d/m/Y and H:M:S and this is hard coded into wee_import so
> there is nothing that could be set incorrectly d/m wise.
>
> In terms of the date-time order of what is being imported, wee_import
> orders the Cumulus logs in oldest to newest order and then proceeds to work
> throught each file from oldest to newest importing all data in each file
> (which happens to be in oldest to newest order as well). wee_import
> processes the imported records in tranches, nominally 250 records per
> tranche. I don't see any way that data could be processed twice during an
> import (it can certainly be processed twice if you import it twice but not
> if you only import it once). All told I don't see how any of this could
> have caused the errors you received. All in all I am at a bit of a loss.
>
> I am wondering if somehow some of your data found its way into your
> archive. Could I ask you to do a little interrogating of your archive?
> Given there were issues with the September 2017 data and that you posted
> the September 2017 log file lets have a look at that month. I'm assuming
> you are using SQLite and you database is /home/weewx/archive/weewx.sdb,
> open your archive with SQLite3:
>
> $ sqlite3 /home/weewx/archive/weewx.sdb
> sqlite>
>
> Check how many records are in the month of Spetember 2017:
>
> sqlite> SELECT COUNT(dateTime) FROM archive WHERE
> dateTime>=strftime('%s','2017-09-01
> 00:00:00', 'utc') AND dateTime<strftime('%s','2017-10-01 00:00:00', 'utc'
> );
>
> That should return a number, there are 8377 valid records in your
> September 2017 monthly log file so if the answer is 8377 you have almost a
> full month of data (there were a few records missing 30*288=8640). Assuming
> there is almost full month of data in you archive lets pick a record from a
> period that was not imported, say 20 September 2017 13:55 and see what it
> holds:
>
> sqlite> SELECT outTemp, outHumidity, dewpoint FROM archive WHERE dateTime=
> strftime('%s','2017-09-20 13:55:00', 'utc');
> 82.2|55.0|64.4
>
> outTemp, outHumidity and dewpoint are the 3rd, 4th and 5th fields on each
> line so easy to check. On your imported data I get 82.5F, 55.0% and 64.4F,
> how does that compare with your results? If the same I strongly suspect
> your data was somehow imported beforehand, you can check further
> records/obs by changing the fields and date/time in the above query. If
> your results don't agree it looks like you have some bogus data somehow. If
> you have bogus data then I guess the only solution is to delete it and
> re-import. That is easy to do for just one month, happy to give you the SQL
> separately if required. I guess if your results agree with what's in you
> monthly log file then you need to make a decision whether you trust you
> have all of the correct data or not (do you look at other months/days/times
> - there is a lot to look at), if you trust it then not much else to do but
> continue importing your older data (I would import one month at a time and
> check that there is no data in your archive for that month before you
> import it, do the month import and see if it all imported OK or of you have
> more errors, generally easier to pick uon on and fix an issue this way then
> doing a multi-year import and then having mountains to deal with.).
>
> One other thing you can check, the logs. After a wee_import run you
> should have a summary written to log (unless you specifically turn it off),
> I received something like:
>
> Jan 29 21:05:39 stretch22 wee_import[1256]: manager: Added record 2017-09-
> 30 23:55:00 AEST (1506779700) to database 'weewx.sdb'
> Jan 29 21:05:39 stretch22 wee_import[1256]: manager: Added record 2017-09-
> 30 23:55:00 AEST (1506779700) to daily summary in 'weewx.sdb'
> Jan 29 21:05:39 stretch22 wee_import[1256]: wee_import: Mapped data saved
> to archive successfully for period 1.
> Jan 29 21:05:39 stretch22 wee_import[1256]: wee_import: Finished import.
> 8377 raw records resulted in 8377 unique records being processed in 25.55
> seconds.
>
> when I imported your September data. Worthwhile having at look at the
> numbers to see how many duplicates you had (there were none for the above
> import), should give an idea of the extent of the issue though not much
> other help.
>
> One final thing, if you do do any more imports keep a copy of the screen
> output, it should be providing progress on what wee_import is doing as
> well as the log.
>
> Gary
>
> On Monday, 29 January 2018 06:27:04 UTC+10, gjr80 wrote:
>>
>> I did see the first couple of posts this morning but it was way too early
>> and I needed to get going. I will have a look later when home again, but my
>> first thoughts were that
>>
>> manager: Unable to add record 2017-11-15 00:00:00 EST (1512018000) to
>> database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>
>> indicates that a record of that timestamp already exists in the archive.
>> I guess DD/MM MM/DD could cause this but if the date format is consistent
>> across all source records/files it shouldn't.
>>
>> As for order of import, I thought wee_import ordered the Cumulus monthly
>> log files from oldest to newest, though I suspect it would not take too
>> much to upset this, it is only a rudimentary check/order based on file
>> name. But as you say rebuilding the daily summaries will fix this.
>>
>> Gary
>>
>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
