Thanks for your detailed responses. That is great info! On Monday, October 30, 2023 at 4:20:44 PM UTC-4 gjr80 wrote:
> Perhaps a little late as you seem to have solved your issues but a few > points post importing data. > > 1. Checking for a successful import by looking at the WeeWX generated web > pages is a poor choice as it often provides misleading information. When > folks import data it is often to obtain historical data from some days (or > months or years) ago; WeeWX typically displays such data in plots or on > historical stats pages. Many plots (particularly week, month and year) use > aggregate periods of one, three or 24 hours when obtaining their source > data. WeeWX plots are updated every aggregate period so some plots may not > update for one, three or 24 hours. Stats type pages typically update each > report cycle, so they may well (or may not) give an indication if data was > successfully imported, but this indication will likely before for a few > specific points in time rather than an extensive report (such as provided > in a plot). The best measure of a success import is to look at the data in > the database (the archive table) and to look at the WeeWX log. > > 2. '*unable to add record UNIQUE constraint failed*' log entries are > entirely normal and not considered an error, it is just WeeWX saying that a > record with the same timestamp as the current imported record already > exists in the WeeWX database and the imported record has been discarded. > Unfortunately wee_import cannot tell (well not in an efficient manner) if > an imported record already exists in the database, so hence the summary > reporting within wee_import states the number of records processed and > the number of unique records imported but refers the user to the WeeWX log > for what actually happened. > > 3. There is no need for dropping or rebuilding daily summaries when > importing. wee_import automatically updates the daily summaries. A drop > and rebuild should only be used where an import failed mid-stream (and then > it will unlikely be required due to the transactional nature of database > operations performed by wee_import (and WeeWX)). Arguably, unnecessary > dropping and rebuilding actually loses data as this can reduce the > granularity of highs/lows (both in terms of value and time) recorded in the > daily summaries for each observation - but that is another story. > > 4. NOAA format reports are updated each report cycle but only the current > month and year report are updated; WeeWX does not go back and recreate any > earlier/missing month/year reports. The solution here is to delete all NOAA > format reports from the WeeWX machine and this will force WeeWX to > regenerate all NOAA format reports for all time on the next report cycle. > This can take quite a few minutes (database size dependent) so for multiple > imports it may be time effective to perform one import, delete and check > NOAA format report generation and then force regeneration. Once satisfied > the first import works, perform the rest of the imports then force NOAA > format report regeneration once the imports are complete. > > 5. Plots can suffer a similar problem to the NOAA format reports but due > to a different mechanism (refer 1. above). The solution; however, is the > same; delete all plots on the WeeWX machine to force regeneration. Again > this can take a long time so for multiple imports it may be more time > effective to perform one import, delete and check plot generation and then > force plot regeneration once the imports are complete. > > Gary > On Tuesday, 31 October 2023 at 05:37:41 UTC+10 hinso...@gmail.com wrote: > >> Another update. Man have I learned a lot about weewx and databases in >> general through this process. I am typing all of this up in the hopes that >> it helps someone in the future who encounters a similar issue. >> >> I examined the daily summary databases such as archive_day_outTemp and >> they all looked normal. In short, the data import worked as expected with >> no issues. All databases have all my data and are up to date. >> >> To fix this issue, I deleted everything from */var/www/html/weewx, *stopped >> and restarted weewx, and viola! All summaries appeared. It looks like the >> NOAA Monthly and yearly summaries did not regenerate since I did the >> import. I had assumed that they would have done so when I ran *sudo >> wee_database --drop-daily* and *sudo wee_database **--rebuild-daily *but >> they did not. >> On Monday, October 30, 2023 at 2:36:41 PM UTC-4 The Reckoning wrote: >> >>> I exported my database (with all imports complete) to a CSV via sqlite3. >>> >>> All the data is there but the Daily/Monthly/Yearly summaries wont >>> generate for certain days. I don't see anything different about the data >>> for these days/months. See for yourself. CSV Linked below. The file is >>> called weewx_db_export.csv. >>> >>> Link >>> <https://hinsonventures-my.sharepoint.com/:f:/g/personal/jmhinson_hinsoncompanies_com/EqAENr-xHx9Dml15lovr09EBZZXe1vA2U42p8aAoqNMC0A?e=UFAw7p> >>> >>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/66685bee-3abb-45ee-a879-754fa86cf967n%40googlegroups.com.
