Great! I confirm this helped! Many thanks, Tom! Regarding corrupt (?) data: I have a Vantage Vue station but as you noticed I combine data from other sources, too. One of them is soil temperature coming from a Wemos D1. It might have been around 2021 when I started with this and maybe entered garbage data. Honestly, I don't remember.
I have updated the database file on the desktop PC now. If ain't too much risky, can I merge this database which has last record logged on December 23 with the temporary one (which I started that same day)? Is the procedure described anywhere in the weewx docs perhaps? I am very happy for your support and patience, thank you. četrtek, 26. december 2024 ob 20:49:52 UTC+1 je oseba Tom Keffer napisala: > That gave us the information that we needed. The problem is that starting > at dateTime=1628754300 field soilTemp1 and soilTemp2 have entries 'Inf'. > > This is 3+ years ago, around 2021-08-21 0900. Any idea what happened then? > > To fix (make a backup first): > > *sudo sqlite3 /var/lib/weewx/weewx.sdb* > sqlite>* update archive set soilTemp1=null where soilTemp1=9e999;* > sqlite>* update archive set soilTemp2=null where soilTemp2=9e999;* > sqlite>* .quit* > > The number "9e999" serves as value "Inf". > > Then do your database update. > > -tk > > > On Wed, Dec 25, 2024 at 10:54 PM mihec <[email protected]> wrote: > >> Tom, apologies for my misunderstanding. You were looking for the syslog I >> believe... >> Here it is a fresh one. >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Initializing >> weectl version 5.1.0 >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Command line: >> /usr/share/weewx/weectl.py database rebuild-daily >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Using Python: >> 3.10.12 (main, Nov 6 2024, 20:22:13) [GCC 11.4.0] >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Located at: >> /usr/bin/python3 >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Platform: >> Linux-6.8.0-47-generic-x86_64-with-glibc2.35 >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Locale: >> 'LC_CTYPE=en_US.UTF-8;LC_NUMERIC=sl_SI.UTF-8;LC_TIME=sl_SI.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=sl_SI.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=sl_SI.UTF-8;LC_NAME=sl_SI.UTF-8;LC_ADDRESS=sl_SI.UTF-8;LC_TELEPHONE=sl_SI.UTF-8;LC_MEASUREMENT=sl_SI.UTF-8;LC_IDENTIFICATION=sl_SI.UTF-8' >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Entry path: >> /usr/share/weewx/weectl.py >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: WEEWX_ROOT: >> /etc/weewx >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Config file: >> /etc/weewx/weewx.conf >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: User module: >> /etc/weewx/bin/user >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Debug: 0 >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: User: >> root >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Group: >> root >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Groups: >> root >> Dec 26 07:50:21 hostname weectl[309641]: INFO weectl-database: All daily >> summaries will be rebuilt. >> Dec 26 07:50:23 hostname weectl[309641]: INFO weectl-database: Rebuilding >> daily summaries in database 'weewx.sdb' ... >> Dec 26 07:50:23 hostname weectl[309641]: INFO weewx.manager: Daily >> summaries at V2.0. Patching to V4.0 >> Dec 26 07:50:23 hostname weectl[309641]: INFO weewx.manager: >> recalculate_weights: Using database 'weewx.sdb' >> Dec 26 07:51:34 hostname weectl[309641]: ERROR weewx.manager: No column >> error: no such column: inf >> Dec 26 07:51:34 hostname weectl[309641]: ERROR weewx.manager: *** sql >> statement: 'UPDATE archive_day_soilTemp1 SET sum=inf, count=1434, wsum=inf, >> sumtime=86040.0 WHERE dateTime = ?;' >> Dec 26 07:51:34 hostname weectl[309641]: ERROR weewx.manager: *** start >> time=1628719200.0 >> >> četrtek, 26. december 2024 ob 01:35:53 UTC+1 je oseba Tom Keffer napisala: >> >>> I am not seeing the log entries from the instrumented code that I >>> included. It will tell us much more about what is going on. Please include >>> it. >>> >>> For the record, I was also able to rebuild the daily summaries >>> uneventfully using your dumped archive table. >>> >>> But, it occurred to me, midnight here on the west coast of the USA >>> (UTC-7) is a different time zone from you, which I assume is CEST. I set up >>> a VM using CEST but, again, it rebuilt the summaries uneventfully. >>> >>> Really need to see the log. >>> >>> -tk >>> >>> >>> On Wed, Dec 25, 2024 at 2:40 PM mihec <[email protected]> wrote: >>> >>>> Thanks for checking. I ran the command again and monitored memfree but >>>> it was nothing to be worried about. >>>> Here is how I ran the command (note the from-to and where the >>>> processing was when I captured this status): >>>> >>>> username@hostname:~$ sudo weectl database rebuild-daily >>>> --from=2021-01-01 --to=2022-01-01 >>>> Using configuration file /etc/weewx/weewx.conf >>>> Daily summaries from 2021-01-01 through 2022-01-01, inclusive, will be >>>> rebuilt. >>>> Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y >>>> Rebuilding daily summaries in database 'weewx.sdb' ... >>>> Daily summaries at V2.0. Patching to V4.0 >>>> Processed through: 2020-08-28 00:00:00 CEST (1598565600) >>>> >>>> It again fails at the same point. I'm sorry to bother you so much with >>>> this. >>>> I just tried on a laptop where I run weewx 4.10. The 'wee_database >>>> --rebuild-daily' fails with same error at same point as on the other two >>>> systems. >>>> For the record, the RPi is 3B+ running bullseye 32-bit, my desktop is >>>> old i7-6700 3.4GHz with 8 GB, running Ubuntu 22.04.1. Laptop is i7-7500 >>>> 2.70 GHz with 8 GB running Ubuntu 22.04.1. >>>> >>>> >>>> sreda, 25. december 2024 ob 23:12:39 UTC+1 je oseba Tom Keffer napisala: >>>> >>>>> I was thinking the same. >>>>> >>>>> On Wed, Dec 25, 2024 at 2:10 PM vince <[email protected]> wrote: >>>>> >>>>>> Thank you. I built a db from it and ran rebuild-daily and it worked >>>>>> just fine with no errors in 2.5 minutes. No errors running database >>>>>> check. No errors running database update. >>>>>> >>>>>> Computer was a m1 mac mini with 16 GB of RAM. I wonder if you're >>>>>> running out of memory and need to split your rebuild-daily up into >>>>>> pieces >>>>>> with the --from and --to options. Just a guess. >>>>>> >>>>>> >>>>>> On Wednesday, December 25, 2024 at 1:56:50 PM UTC-8 mihec wrote: >>>>>> >>>>>>> This is the dump: https://we.tl/t-X3r6gbimBx >>>>>>> >>>>>>> sreda, 25. december 2024 ob 22:48:59 UTC+1 je oseba mihec napisala: >>>>>>> >>>>>>>> I replaced the original manager.py with attached one. That file is >>>>>>>> in the /usr/share/weewx/weewx/ directory. >>>>>>>> I will prepare the dump of the database. >>>>>>>> >>>>>>>> username@hostname:~$ sudo weectl database rebuild-daily >>>>>>>> >>>>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>>>> All daily summaries will be rebuilt. >>>>>>>> Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y >>>>>>>> Rebuilding daily summaries in database 'weewx.sdb' ... >>>>>>>> Daily summaries at V2.0. Patching to V4.0 >>>>>>>> Traceback (most recent call last):0:00 CEST (1628719200) >>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn >>>>>>>> return fn(*args, **kwargs) >>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute >>>>>>>> return sqlite3.Cursor.execute(self, *args, **kwargs) >>>>>>>> sqlite3.OperationalError: no such column: inf >>>>>>>> >>>>>>>> During handling of the above exception, another exception occurred: >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/usr/share/weewx/weectl.py", line 75, in <module> >>>>>>>> main() >>>>>>>> File "/usr/share/weewx/weectl.py", line 67, in main >>>>>>>> namespace.func(namespace) >>>>>>>> File "/usr/share/weewx/weectllib/__init__.py", line 90, in >>>>>>>> dispatch >>>>>>>> namespace.action_func(config_dict, namespace) >>>>>>>> File "/usr/share/weewx/weectllib/database_cmd.py", line 329, in >>>>>>>> rebuild_daily >>>>>>>> weectllib.database_actions.rebuild_daily(config_dict, >>>>>>>> File "/usr/share/weewx/weectllib/database_actions.py", line 134, >>>>>>>> in rebuild_daily >>>>>>>> with weewx.manager.open_manager(manager_dict, initialize=True) >>>>>>>> as dbm: >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 924, in >>>>>>>> open_manager >>>>>>>> return >>>>>>>> manager_cls.open_with_create(manager_dict['database_dict'], >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 244, in >>>>>>>> open_with_create >>>>>>>> dbmanager = cls(connection, table_name=table_name, >>>>>>>> schema=schema) >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1076, in __init__ >>>>>>>> self.patch_sums() >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1537, in patch_sums >>>>>>>> >>>>>>>> self.recalculate_weights(start_d=datetime.date(2020, 6, 1)) >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1451, in >>>>>>>> recalculate_weights >>>>>>>> self._do_tranche(mark_d, end_of_tranche_d, weight_fn, >>>>>>>> progress_fn) >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1490, in >>>>>>>> _do_tranche >>>>>>>> self._set_day_sums(day_accum, cursor) >>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1516, in >>>>>>>> _set_day_sums >>>>>>>> >>>>>>>> cursor.execute(update_sql, (day_accum.timespan.start,)) >>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 50, in guarded_fn >>>>>>>> raise weedb.NoColumnError(e) >>>>>>>> weedb.NoColumnError: no such column: inf >>>>>>>> >>>>>>>> >>>>>>>> sreda, 25. december 2024 ob 21:13:17 UTC+1 je oseba Tom Keffer >>>>>>>> napisala: >>>>>>>> >>>>>>>>> Thanks. That tells me what I needed to know to create a useful >>>>>>>>> instrumented version of manager.py. Replace the one you have with the >>>>>>>>> attached version. It will give more information about exactly what >>>>>>>>> created >>>>>>>>> the error. Post the log. >>>>>>>>> >>>>>>>>> -tk >>>>>>>>> >>>>>>>>> On Wed, Dec 25, 2024 at 9:29 AM mihec <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, first I want to wish you Merry Christmas and Happy Holidays! >>>>>>>>>> >>>>>>>>>> This is the output of the "weectl database rebuild-daily". Please >>>>>>>>>> note that I've copied the weewx.sdb database file to a desktop PC to >>>>>>>>>> make >>>>>>>>>> it run much faster. I've replaced the real username and hostname >>>>>>>>>> with >>>>>>>>>> something generic. The weewx's version is same as in RPi. >>>>>>>>>> >>>>>>>>>> When executed, the 'rebuild' process starts around June 2020 and >>>>>>>>>> goes on normally until it fails on August 12, 2021 (epoch >>>>>>>>>> 1628719200). In the output below, you only see the status when >>>>>>>>>> it fails already. I can provide with a screenshot if that helps. >>>>>>>>>> The thing is that any command on the database (update, check or >>>>>>>>>> rebuild) fails on the same date (record?). >>>>>>>>>> >>>>>>>>>> username@host:~$ ls -la weewx.sdb >>>>>>>>>> -rwxrwxr-x 1 username username 682086400 dec 25 18:18 weewx.sdb >>>>>>>>>> username@host:~$ weewxd -v >>>>>>>>>> 5.1.0 >>>>>>>>>> username@host:~$ weectl -v >>>>>>>>>> weectl 5.1.0 >>>>>>>>>> username@host:~$ sudo weectl database rebuild-daily >>>>>>>>>> [sudo] password for username: >>>>>>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>>>>>> All daily summaries will be rebuilt. >>>>>>>>>> Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y >>>>>>>>>> Rebuilding daily summaries in database 'weewx.sdb' ... >>>>>>>>>> Daily summaries at V2.0. Patching to V4.0 >>>>>>>>>> Traceback (most recent call last):0:00 CEST (1628719200) >>>>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn >>>>>>>>>> return fn(*args, **kwargs) >>>>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute >>>>>>>>>> return sqlite3.Cursor.execute(self, *args, **kwargs) >>>>>>>>>> sqlite3.OperationalError: no such column: inf >>>>>>>>>> >>>>>>>>>> During handling of the above exception, another exception >>>>>>>>>> occurred: >>>>>>>>>> >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>> File "/usr/share/weewx/weectl.py", line 75, in <module> >>>>>>>>>> main() >>>>>>>>>> File "/usr/share/weewx/weectl.py", line 67, in main >>>>>>>>>> namespace.func(namespace) >>>>>>>>>> File "/usr/share/weewx/weectllib/__init__.py", line 90, in >>>>>>>>>> dispatch >>>>>>>>>> namespace.action_func(config_dict, namespace) >>>>>>>>>> File "/usr/share/weewx/weectllib/database_cmd.py", line 329, in >>>>>>>>>> rebuild_daily >>>>>>>>>> weectllib.database_actions.rebuild_daily(config_dict, >>>>>>>>>> File "/usr/share/weewx/weectllib/database_actions.py", line >>>>>>>>>> 134, in rebuild_daily >>>>>>>>>> with weewx.manager.open_manager(manager_dict, >>>>>>>>>> initialize=True) as dbm: >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 924, in >>>>>>>>>> open_manager >>>>>>>>>> return >>>>>>>>>> manager_cls.open_with_create(manager_dict['database_dict'], >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 244, in >>>>>>>>>> open_with_create >>>>>>>>>> dbmanager = cls(connection, table_name=table_name, >>>>>>>>>> schema=schema) >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1076, in __init__ >>>>>>>>>> self.patch_sums() >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1531, in >>>>>>>>>> patch_sums >>>>>>>>>> self.recalculate_weights(start_d=datetime.date(2020, 6, 1)) >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1451, in >>>>>>>>>> recalculate_weights >>>>>>>>>> self._do_tranche(mark_d, end_of_tranche_d, weight_fn, >>>>>>>>>> progress_fn) >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1490, in >>>>>>>>>> _do_tranche >>>>>>>>>> self._set_day_sums(day_accum, cursor) >>>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1515, in >>>>>>>>>> _set_day_sums >>>>>>>>>> cursor.execute(update_sql, (day_accum.timespan.start,)) >>>>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 50, in guarded_fn >>>>>>>>>> raise weedb.NoColumnError(e) >>>>>>>>>> weedb.NoColumnError: no such column: inf >>>>>>>>>> username@host:~$ >>>>>>>>>> >>>>>>>>>> torek, 24. december 2024 ob 15:15:16 UTC+1 je oseba Tom Keffer >>>>>>>>>> napisala: >>>>>>>>>> >>>>>>>>>>> Let's put aside the database update and just focus on rebuilding >>>>>>>>>>> the daily summaries. >>>>>>>>>>> >>>>>>>>>>> Could you please include the stack trace when doing "weectl >>>>>>>>>>> database rebuild-daily"? Once I see where it fails, I can send you >>>>>>>>>>> an >>>>>>>>>>> instrumented version of the database code so we can see exactly >>>>>>>>>>> what is >>>>>>>>>>> happening. >>>>>>>>>>> >>>>>>>>>>> -tk >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 24, 2024 at 3:39 AM mihec <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Well, I've checked also the weectl database check which fails >>>>>>>>>>>> at the same point as the update. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> torek, 24. december 2024 ob 12:31:36 UTC+1 je oseba >>>>>>>>>>>> Auchtermuchty Weather napisala: >>>>>>>>>>>> >>>>>>>>>>>>> That's not what it does now. >>>>>>>>>>>>> >>>>>>>>>>>>> " Databases created earlier than 3.7.0 (released >>>>>>>>>>>>> 11-March-2017) have a flaw that prevents them from being used >>>>>>>>>>>>> with archive >>>>>>>>>>>>> intervals that change with time. This utility check whether your >>>>>>>>>>>>> database >>>>>>>>>>>>> is affected." >>>>>>>>>>>>> >>>>>>>>>>>>> $ weectl database check >>>>>>>>>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>>>>>>>>> Checking daily summary tables version... >>>>>>>>>>>>> Daily summary tables are at version 4.0. >>>>>>>>>>>>> Interval Weighting Fix is not required. >>>>>>>>>>>>> $ >>>>>>>>>>>>> >>>>>>>>>>>>> On Tuesday, 24 December 2024 at 11:13:48 UTC Graham Eddy wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> weewx used to include a tool for picking up things like a >>>>>>>>>>>>>> string where a number should be. maybe `weectl database check’ >>>>>>>>>>>>>> still does >>>>>>>>>>>>>> this >>>>>>>>>>>>>> *⊣GE⊢* >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24 Dec 2024, at 4:17 pm, mihec <[email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think there must be an invalid entry in my database (Aug >>>>>>>>>>>>>> 12, 2021). My SQLite database management knowledge is basically >>>>>>>>>>>>>> zero, so >>>>>>>>>>>>>> I'm looking for advice >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> >>>>>>>>>>> To view this discussion visit >>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/e60483cc-3910-45f8-aff4-7b4e95939cd9n%40googlegroups.com >>>>>>>>>>>> >>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/e60483cc-3910-45f8-aff4-7b4e95939cd9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> >>>>>>>>> To view this discussion visit >>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/e92cf44c-d06f-4de1-ba04-440bceed755fn%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/e92cf44c-d06f-4de1-ba04-440bceed755fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion visit >>>>>> https://groups.google.com/d/msgid/weewx-user/124fd290-b296-4c56-a4d4-43e7b30ae2d7n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/weewx-user/124fd290-b296-4c56-a4d4-43e7b30ae2d7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion visit >>>> https://groups.google.com/d/msgid/weewx-user/8fe396c2-d6b2-4aac-a8c8-fc014e9d2df8n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/8fe396c2-d6b2-4aac-a8c8-fc014e9d2df8n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion visit >> https://groups.google.com/d/msgid/weewx-user/bb0c8c81-d7b4-401f-8758-5561660d001fn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/bb0c8c81-d7b4-401f-8758-5561660d001fn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/8dc469ac-2f09-4600-9487-84b4287bd39bn%40googlegroups.com.
