Perhaps a little late as you seem to have solved your issues but a few points post importing data.
1. Checking for a successful import by looking at the WeeWX generated web pages is a poor choice as it often provides misleading information. When folks import data it is often to obtain historical data from some days (or months or years) ago; WeeWX typically displays such data in plots or on historical stats pages. Many plots (particularly week, month and year) use aggregate periods of one, three or 24 hours when obtaining their source data. WeeWX plots are updated every aggregate period so some plots may not update for one, three or 24 hours. Stats type pages typically update each report cycle, so they may well (or may not) give an indication if data was successfully imported, but this indication will likely before for a few specific points in time rather than an extensive report (such as provided in a plot). The best measure of a success import is to look at the data in the database (the archive table) and to look at the WeeWX log. 2. '*unable to add record UNIQUE constraint failed*' log entries are entirely normal and not considered an error, it is just WeeWX saying that a record with the same timestamp as the current imported record already exists in the WeeWX database and the imported record has been discarded. Unfortunately wee_import cannot tell (well not in an efficient manner) if an imported record already exists in the database, so hence the summary reporting within wee_import states the number of records processed and the number of unique records imported but refers the user to the WeeWX log for what actually happened. 3. There is no need for dropping or rebuilding daily summaries when importing. wee_import automatically updates the daily summaries. A drop and rebuild should only be used where an import failed mid-stream (and then it will unlikely be required due to the transactional nature of database operations performed by wee_import (and WeeWX)). Arguably, unnecessary dropping and rebuilding actually loses data as this can reduce the granularity of highs/lows (both in terms of value and time) recorded in the daily summaries for each observation - but that is another story. 4. NOAA format reports are updated each report cycle but only the current month and year report are updated; WeeWX does not go back and recreate any earlier/missing month/year reports. The solution here is to delete all NOAA format reports from the WeeWX machine and this will force WeeWX to regenerate all NOAA format reports for all time on the next report cycle. This can take quite a few minutes (database size dependent) so for multiple imports it may be time effective to perform one import, delete and check NOAA format report generation and then force regeneration. Once satisfied the first import works, perform the rest of the imports then force NOAA format report regeneration once the imports are complete. 5. Plots can suffer a similar problem to the NOAA format reports but due to a different mechanism (refer 1. above). The solution; however, is the same; delete all plots on the WeeWX machine to force regeneration. Again this can take a long time so for multiple imports it may be more time effective to perform one import, delete and check plot generation and then force plot regeneration once the imports are complete. Gary On Tuesday, 31 October 2023 at 05:37:41 UTC+10 [email protected] wrote: > Another update. Man have I learned a lot about weewx and databases in > general through this process. I am typing all of this up in the hopes that > it helps someone in the future who encounters a similar issue. > > I examined the daily summary databases such as archive_day_outTemp and > they all looked normal. In short, the data import worked as expected with > no issues. All databases have all my data and are up to date. > > To fix this issue, I deleted everything from */var/www/html/weewx, *stopped > and restarted weewx, and viola! All summaries appeared. It looks like the > NOAA Monthly and yearly summaries did not regenerate since I did the > import. I had assumed that they would have done so when I ran *sudo > wee_database --drop-daily* and *sudo wee_database **--rebuild-daily *but > they did not. > On Monday, October 30, 2023 at 2:36:41 PM UTC-4 The Reckoning wrote: > >> I exported my database (with all imports complete) to a CSV via sqlite3. >> >> All the data is there but the Daily/Monthly/Yearly summaries wont >> generate for certain days. I don't see anything different about the data >> for these days/months. See for yourself. CSV Linked below. The file is >> called weewx_db_export.csv. >> >> Link >> <https://hinsonventures-my.sharepoint.com/:f:/g/personal/jmhinson_hinsoncompanies_com/EqAENr-xHx9Dml15lovr09EBZZXe1vA2U42p8aAoqNMC0A?e=UFAw7p> >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c3c77c7e-d5b2-4780-b5ee-86e8034e2ca4n%40googlegroups.com.
