Okay so i have made some progress but will need the communities help on this. I have tried to include as much information as possible but please let me know if you need anything else. Also find the link to a folder containing relevant files below.
I downloaded my data from WeatherLink.com 1 year at a time in CSV format. I have data from 7/28/2019 through 10/25/2023. I then modified the CSV to fit the formatting requested by Weewx. This included changing the inter-cardinal wind directions to degree (done with excel formula), deleting some header rows, and changing the names of the column headers. I copied these column headers into my import config file and changed other settings in there. I stopped Weewx and ran a dry-run. There were no errors on the --dry-run so ran it for real for the first year of data (July-2019-June-2020). This worked fine. I could see all the new data on the webpage. Great. Now onto the next year: (July-2020-June-2021). Same process as above but the data never showed up on the webpage. I tried it again but same result. After this happened, I loaded a backup of the weewx database and combined all data into 1 file and imported that. Again, are huge holes in the data. I now have data from July-2019 to June-2020. No data from July-2020 to December-2022. Then I have normal data in all of 2023 (January-present). I cant figure out whats wrong. I looked the data over again but can find no differences between years. While the import was happening I saw lots of these logs saying, (as an example): Oct 30 09:43:37 jm-Virtual-Machine wee_import[195988] INFO weewx.manager: Added record 2022-04-27 19:00:00 EDT (1651100400) to database 'weewx.sdb' But I also saw lots of these logs: Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 05:00:00 EST (1643623200) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 05:30:00 EST (1643625000) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 06:00:00 EST (1643626800) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 06:30:00 EST (1643628600) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 07:00:00 EST (1643630400) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 07:30:00 EST (1643632200) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 08:00:00 EST (1643634000) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 08:30:00 EST (1643635800) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Oct 30 10:53:18 jm-Virtual-Machine wee_import[199232] ERROR *weewx*.manager: Unable to add record 2022-01-31 09:00:00 EST (1643637600) to database ' *weewx*.sdb': UNIQUE constraint failed: archive.dateTime Not sure what these mean but there were a LOT of them. Maybe these are the cause of my issues. Below is a link to: 1. csv.conf 2. PWS_All2.csv - all of my historical weather data 3. Weewx_log_Import_All.txt - Log of the full import showing the above mentioned errors and successes 4. wee_debug --info output Link to Files <https://hinsonventures-my.sharepoint.com/:f:/g/personal/jmhinson_hinsoncompanies_com/EqAENr-xHx9Dml15lovr09EBZZXe1vA2U42p8aAoqNMC0A?e=1QfeO8> Any help I can get in resolving this is much appreciated. Here is the command output from the combined import jm@jm-Virtual-Machine:/var/lib/weewx$ sudo wee_import --import-config=/etc/ weewx/import/csv.conf /usr/share/weewx/wee_import:719: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives from distutils.version import StrictVersion Using WeeWX configuration file /etc/weewx/weewx.conf Starting wee_import... A CSV import from source file '/home/jm/Desktop/PWS_All2.csv' has been requested. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. All WeeWX UV fields will be set to None. All WeeWX radiation fields will be set to None. Starting import ... 74365 records identified for import. Proceeding will save all imported records in the WeeWX archive. Are you sure you want to proceed (y/n)? y Unique records processed: 74365; Last timestamp: 2023-10-25 10:00:00 EDT ( 1698242400) Calculating missing derived observations ... Processing record: 74365; Last record: 2023-10-26 00:00:00 EDT (1698292800) Recalculating daily summaries... Records processed: 74000; time: 2023-10-17 19:30:00 EDT (1697585400) Finished recalculating daily summaries Finished calculating missing derived observations Finished import 74365 records were processed and 74365 unique records imported in 250.55 seconds. Those records with a timestamp already in the archive will not have been imported. Confirm successful import in the WeeWX log file. On Sunday, October 29, 2023 at 5:33:02 PM UTC-4 The Reckoning wrote: > Hey guys, > > I have a Vantage Pro 2 and have had it connected to WeatherLink.com via a > WeatherLink IP connector. > > I had to turn the upload off because i now want to store my data on Weewx. > Is it possible to retrieve all of that historical data from the Weatherlink > site? > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/513ee7e6-d3bc-4dfb-8805-b71b4b2a4dc6n%40googlegroups.com.
