Thank you Gary. This might be the error from my side. I just copied the whole bunch of logs into my import directory. I will make subdir for every year, then I am going to test again. Maybe going backward from 2020 to 2005 might be a good idea to find out my problems.
Version of my WeeWX is 4.4.0, running on a fresh Raspian image, installed on Friday. Let me test tonight and I will report you back. Peter gjr80 schrieb am Montag, 1. Februar 2021 um 04:38:54 UTC+1: > Peter, > > I ran wee_import under WeeWX v4.4.0/python3 using the import config file > and log files posted and the import processed without error: > > gary@buster31:~$ /home/weewx/bin/wee_import > --import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00 > --to=2020-12-31T23:59 --verbose > Using WeeWX configuration file /home/weewx/weewx.conf > > Starting wee_import... > Weather Display monthly log files in the '/tmp/wdisplay' directory will be > imported > The following options will be used: > config=/home/weewx/weewx.conf, > import-config=/var/tmp/wd-import-test.conf > > from=2020-12-01T00:00, to=2020-12-31T23:59 > dry-run=None, calc_missing=True, ignore_invalid_data=True > monthly logs are in METRIC units > tranche=250, interval=1 > UV=True, radiation=True ignore extreme temperature and humidity=True > Using database binding 'wx_binding', which is bound to database 'weewx.sdb' > Destination table 'archive' unit system is '0x01' (US). > Missing derived observations will be calculated. > Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and > up to and > including 2020-12-31 23:59:00 AEST (1609423140) will be imported. > > Starting import ... > Records covering multiple periods have been identified for import. > Obtaining raw import data for period 1 ... > Raw import data read successfully for period 1. > Mapping raw import data for period 1 ... > Mapped 44639 records. > Raw import data mapped successfully for period 1. > Saving mapped data to archive for period 1 ... > 44639 records identified for import. > Proceeding will save all imported records in the WeeWX archive. > Are you sure you want to proceed (y/n)? y > Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST > (1609423140) > Mapped data saved to archive successfully for period 1. > Calculating missing derived observations ... > Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST > (1609423200) > Recalculating daily summaries... > Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800) > Finished recalculating daily summaries > Finished calculating missing derived observations > Finished import > 44639 records were processed and 44639 unique records imported in 249.23 > seconds. > Those records with a timestamp already in the archive will not have been > imported. Confirm successful import in the WeeWX log file. > > So I am confident there is nothing wrong with the import config file and > log files posted. Granted it was a setup.py install but should not really > make a difference to wee_import. > > You said you had 15 years of WD data, when you did the import were there > other older log files in /tmp/wdisplay? The reason I ask is that whilst > you can limit the time frame over which wee_import imports data, all that > does is apply a filter to the final processing/saving of records. > wee_import will still go through and read all log files it finds starting > at the oldest. So if you had 15 years of WD data in a directory and tried > to import the most recent month all 15 years of log files will be read and > processed in a limited manner. The initial processing of the raw import > data involves assembling the individual date and time components from each > row in the log. That appears to be where your import failed. When I wrote > the WD component of wee_import I found a number of data inconsistencies > in the WD log files I was using so I guess it is possible you may have > similar inconsistencies. Have a look through the oldest WD log file set in > /tmp/wdisplay and see if you can spot an obvious error/inconsistency in > the date and time data. If you can't spot anything could you post the > respective log files here. Alternatively, if you let me know what version > of WeeWX you are running I can provide you with an instrumented version of > wee_import to log details of the offending record/log. > > Gary > On Monday, 1 February 2021 at 00:59:35 UTC+10 peter.su...@gmail.com wrote: > >> Hi Gary, >> >> thanks for your quick answer. >> Please check the attached file. All I edited was the path to data and >> filenames. I limited it to only one type of file, with my last change. >> >> No, problem when it takes time. >> Thanks for checking out >> >> Peter >> gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1: >> >>> Hi, >>> >>> Nothing leaps out at me having a quick look at the output and log files >>> you posted. I'll endeavour to have a look at this tomorrow (bit late here). >>> Could I ask you to post the import config file you are using, >>> /etc/weewx/wd-import-test.conf. There should be nothing sensitive in >>> the file. >>> >>> Gary >>> >>> On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com >>> wrote: >>> >>>> Hello, >>>> after using my Vantage pro 2 with Weather Display for about 15 years, I >>>> think now its time for a change. :) >>>> I am making the first tests for transferring data, before switch the >>>> live system. >>>> >>>> My problem, I cannot import the log data from wdisplay. >>>> I'm getting this error message: >>>> >>>> pi@WEEWX-DEMO:/etc/weewx $ wee_import >>>> --import-config=wd-import-test.conf --from=2020-12-01T00:00 >>>> --to=2020-12-31T23:59 --verbose >>>> Using WeeWX configuration file /etc/weewx/weewx.conf >>>> Starting wee_import... >>>> Weather Display monthly log files in the '/tmp/wdisplay' directory will >>>> be imported >>>> The following options will be used: >>>> config=/etc/weewx/weewx.conf, >>>> import-config=/etc/weewx/wd-import-test.conf >>>> from=2020-12-01T00:00, to=2020-12-31T23:59 >>>> dry-run=None, calc_missing=True, ignore_invalid_data=True >>>> monthly logs are in METRIC units >>>> tranche=250, interval=1 >>>> UV=True, radiation=True ignore extreme temperature and >>>> humidity=True >>>> Using database binding 'wx_binding', which is bound to database >>>> 'weewx.sdb' >>>> Destination table 'archive' unit system is '0x01' (US). >>>> Missing derived observations will be calculated. >>>> Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) and >>>> up to and >>>> including 2020-12-31 23:59:00 CET (1609455540) will be imported. >>>> Starting import ... >>>> Records covering multiple periods have been identified for import. >>>> Period 1 ... >>>> Obtaining raw import data for period 1 ... >>>> Traceback (most recent call last): >>>> File "/usr/share/weewx/wee_import", line 903, in <module> >>>> main() >>>> File "/usr/share/weewx/wee_import", line 833, in main >>>> source_obj.run() >>>> File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run >>>> _raw_data = self.getRawData(period) >>>> File "/usr/share/weewx/weeimport/wdimport.py", line 598, in getRawData >>>> _ts = "%s" % int(time.mktime(_tt)) >>>> OverflowError: mktime argument out of range >>>> >>>> >>>> Any ideas? File content is looking ok. >>>> It is happening with all time tests across this 15 years. >>>> Here a month sample data. >>>> >>>> Thanks for any ideas... >>>> Peter >>>> >>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c810aafa-0c39-483c-8e5c-10af1fec8eb4n%40googlegroups.com.
