That gave us the information that we needed. The problem is that starting at dateTime=1628754300 field soilTemp1 and soilTemp2 have entries 'Inf'.
This is 3+ years ago, around 2021-08-21 0900. Any idea what happened then? To fix (make a backup first): *sudo sqlite3 /var/lib/weewx/weewx.sdb* sqlite>* update archive set soilTemp1=null where soilTemp1=9e999;* sqlite>* update archive set soilTemp2=null where soilTemp2=9e999;* sqlite>* .quit* The number "9e999" serves as value "Inf". Then do your database update. -tk On Wed, Dec 25, 2024 at 10:54 PM mihec <[email protected]> wrote: > Tom, apologies for my misunderstanding. You were looking for the syslog I > believe... > Here it is a fresh one. > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Initializing > weectl version 5.1.0 > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Command line: > /usr/share/weewx/weectl.py database rebuild-daily > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Using Python: > 3.10.12 (main, Nov 6 2024, 20:22:13) [GCC 11.4.0] > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Located at: > /usr/bin/python3 > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Platform: > Linux-6.8.0-47-generic-x86_64-with-glibc2.35 > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Locale: > 'LC_CTYPE=en_US.UTF-8;LC_NUMERIC=sl_SI.UTF-8;LC_TIME=sl_SI.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=sl_SI.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=sl_SI.UTF-8;LC_NAME=sl_SI.UTF-8;LC_ADDRESS=sl_SI.UTF-8;LC_TELEPHONE=sl_SI.UTF-8;LC_MEASUREMENT=sl_SI.UTF-8;LC_IDENTIFICATION=sl_SI.UTF-8' > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Entry path: > /usr/share/weewx/weectl.py > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: WEEWX_ROOT: > /etc/weewx > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Config file: > /etc/weewx/weewx.conf > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: User module: > /etc/weewx/bin/user > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Debug: 0 > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: User: root > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Group: root > Dec 26 07:50:21 hostname weectl[309641]: INFO weectllib: Groups: root > Dec 26 07:50:21 hostname weectl[309641]: INFO weectl-database: All daily > summaries will be rebuilt. > Dec 26 07:50:23 hostname weectl[309641]: INFO weectl-database: Rebuilding > daily summaries in database 'weewx.sdb' ... > Dec 26 07:50:23 hostname weectl[309641]: INFO weewx.manager: Daily > summaries at V2.0. Patching to V4.0 > Dec 26 07:50:23 hostname weectl[309641]: INFO weewx.manager: > recalculate_weights: Using database 'weewx.sdb' > Dec 26 07:51:34 hostname weectl[309641]: ERROR weewx.manager: No column > error: no such column: inf > Dec 26 07:51:34 hostname weectl[309641]: ERROR weewx.manager: *** sql > statement: 'UPDATE archive_day_soilTemp1 SET sum=inf, count=1434, wsum=inf, > sumtime=86040.0 WHERE dateTime = ?;' > Dec 26 07:51:34 hostname weectl[309641]: ERROR weewx.manager: *** start > time=1628719200.0 > > četrtek, 26. december 2024 ob 01:35:53 UTC+1 je oseba Tom Keffer napisala: > >> I am not seeing the log entries from the instrumented code that I >> included. It will tell us much more about what is going on. Please include >> it. >> >> For the record, I was also able to rebuild the daily summaries >> uneventfully using your dumped archive table. >> >> But, it occurred to me, midnight here on the west coast of the USA >> (UTC-7) is a different time zone from you, which I assume is CEST. I set up >> a VM using CEST but, again, it rebuilt the summaries uneventfully. >> >> Really need to see the log. >> >> -tk >> >> >> On Wed, Dec 25, 2024 at 2:40 PM mihec <[email protected]> wrote: >> >>> Thanks for checking. I ran the command again and monitored memfree but >>> it was nothing to be worried about. >>> Here is how I ran the command (note the from-to and where the processing >>> was when I captured this status): >>> >>> username@hostname:~$ sudo weectl database rebuild-daily >>> --from=2021-01-01 --to=2022-01-01 >>> Using configuration file /etc/weewx/weewx.conf >>> Daily summaries from 2021-01-01 through 2022-01-01, inclusive, will be >>> rebuilt. >>> Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y >>> Rebuilding daily summaries in database 'weewx.sdb' ... >>> Daily summaries at V2.0. Patching to V4.0 >>> Processed through: 2020-08-28 00:00:00 CEST (1598565600) >>> >>> It again fails at the same point. I'm sorry to bother you so much with >>> this. >>> I just tried on a laptop where I run weewx 4.10. The 'wee_database >>> --rebuild-daily' fails with same error at same point as on the other two >>> systems. >>> For the record, the RPi is 3B+ running bullseye 32-bit, my desktop is >>> old i7-6700 3.4GHz with 8 GB, running Ubuntu 22.04.1. Laptop is i7-7500 >>> 2.70 GHz with 8 GB running Ubuntu 22.04.1. >>> >>> >>> sreda, 25. december 2024 ob 23:12:39 UTC+1 je oseba Tom Keffer napisala: >>> >>>> I was thinking the same. >>>> >>>> On Wed, Dec 25, 2024 at 2:10 PM vince <[email protected]> wrote: >>>> >>>>> Thank you. I built a db from it and ran rebuild-daily and it worked >>>>> just fine with no errors in 2.5 minutes. No errors running database >>>>> check. No errors running database update. >>>>> >>>>> Computer was a m1 mac mini with 16 GB of RAM. I wonder if you're >>>>> running out of memory and need to split your rebuild-daily up into pieces >>>>> with the --from and --to options. Just a guess. >>>>> >>>>> >>>>> On Wednesday, December 25, 2024 at 1:56:50 PM UTC-8 mihec wrote: >>>>> >>>>>> This is the dump: https://we.tl/t-X3r6gbimBx >>>>>> >>>>>> sreda, 25. december 2024 ob 22:48:59 UTC+1 je oseba mihec napisala: >>>>>> >>>>>>> I replaced the original manager.py with attached one. That file is >>>>>>> in the /usr/share/weewx/weewx/ directory. >>>>>>> I will prepare the dump of the database. >>>>>>> >>>>>>> username@hostname:~$ sudo weectl database rebuild-daily >>>>>>> >>>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>>> All daily summaries will be rebuilt. >>>>>>> Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y >>>>>>> Rebuilding daily summaries in database 'weewx.sdb' ... >>>>>>> Daily summaries at V2.0. Patching to V4.0 >>>>>>> Traceback (most recent call last):0:00 CEST (1628719200) >>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn >>>>>>> return fn(*args, **kwargs) >>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute >>>>>>> return sqlite3.Cursor.execute(self, *args, **kwargs) >>>>>>> sqlite3.OperationalError: no such column: inf >>>>>>> >>>>>>> During handling of the above exception, another exception occurred: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/share/weewx/weectl.py", line 75, in <module> >>>>>>> main() >>>>>>> File "/usr/share/weewx/weectl.py", line 67, in main >>>>>>> namespace.func(namespace) >>>>>>> File "/usr/share/weewx/weectllib/__init__.py", line 90, in dispatch >>>>>>> namespace.action_func(config_dict, namespace) >>>>>>> File "/usr/share/weewx/weectllib/database_cmd.py", line 329, in >>>>>>> rebuild_daily >>>>>>> weectllib.database_actions.rebuild_daily(config_dict, >>>>>>> File "/usr/share/weewx/weectllib/database_actions.py", line 134, >>>>>>> in rebuild_daily >>>>>>> with weewx.manager.open_manager(manager_dict, initialize=True) >>>>>>> as dbm: >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 924, in open_manager >>>>>>> return >>>>>>> manager_cls.open_with_create(manager_dict['database_dict'], >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 244, in >>>>>>> open_with_create >>>>>>> dbmanager = cls(connection, table_name=table_name, schema=schema) >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1076, in __init__ >>>>>>> self.patch_sums() >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1537, in patch_sums >>>>>>> >>>>>>> self.recalculate_weights(start_d=datetime.date(2020, 6, 1)) >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1451, in >>>>>>> recalculate_weights >>>>>>> self._do_tranche(mark_d, end_of_tranche_d, weight_fn, >>>>>>> progress_fn) >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1490, in _do_tranche >>>>>>> self._set_day_sums(day_accum, cursor) >>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1516, in >>>>>>> _set_day_sums >>>>>>> >>>>>>> cursor.execute(update_sql, (day_accum.timespan.start,)) >>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 50, in guarded_fn >>>>>>> raise weedb.NoColumnError(e) >>>>>>> weedb.NoColumnError: no such column: inf >>>>>>> >>>>>>> >>>>>>> sreda, 25. december 2024 ob 21:13:17 UTC+1 je oseba Tom Keffer >>>>>>> napisala: >>>>>>> >>>>>>>> Thanks. That tells me what I needed to know to create a useful >>>>>>>> instrumented version of manager.py. Replace the one you have with the >>>>>>>> attached version. It will give more information about exactly what >>>>>>>> created >>>>>>>> the error. Post the log. >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> On Wed, Dec 25, 2024 at 9:29 AM mihec <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, first I want to wish you Merry Christmas and Happy Holidays! >>>>>>>>> >>>>>>>>> This is the output of the "weectl database rebuild-daily". Please >>>>>>>>> note that I've copied the weewx.sdb database file to a desktop PC to >>>>>>>>> make >>>>>>>>> it run much faster. I've replaced the real username and hostname with >>>>>>>>> something generic. The weewx's version is same as in RPi. >>>>>>>>> >>>>>>>>> When executed, the 'rebuild' process starts around June 2020 and >>>>>>>>> goes on normally until it fails on August 12, 2021 (epoch >>>>>>>>> 1628719200). In the output below, you only see the status when it >>>>>>>>> fails already. I can provide with a screenshot if that helps. >>>>>>>>> The thing is that any command on the database (update, check or >>>>>>>>> rebuild) fails on the same date (record?). >>>>>>>>> >>>>>>>>> username@host:~$ ls -la weewx.sdb >>>>>>>>> -rwxrwxr-x 1 username username 682086400 dec 25 18:18 weewx.sdb >>>>>>>>> username@host:~$ weewxd -v >>>>>>>>> 5.1.0 >>>>>>>>> username@host:~$ weectl -v >>>>>>>>> weectl 5.1.0 >>>>>>>>> username@host:~$ sudo weectl database rebuild-daily >>>>>>>>> [sudo] password for username: >>>>>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>>>>> All daily summaries will be rebuilt. >>>>>>>>> Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y >>>>>>>>> Rebuilding daily summaries in database 'weewx.sdb' ... >>>>>>>>> Daily summaries at V2.0. Patching to V4.0 >>>>>>>>> Traceback (most recent call last):0:00 CEST (1628719200) >>>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn >>>>>>>>> return fn(*args, **kwargs) >>>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute >>>>>>>>> return sqlite3.Cursor.execute(self, *args, **kwargs) >>>>>>>>> sqlite3.OperationalError: no such column: inf >>>>>>>>> >>>>>>>>> During handling of the above exception, another exception occurred: >>>>>>>>> >>>>>>>>> Traceback (most recent call last): >>>>>>>>> File "/usr/share/weewx/weectl.py", line 75, in <module> >>>>>>>>> main() >>>>>>>>> File "/usr/share/weewx/weectl.py", line 67, in main >>>>>>>>> namespace.func(namespace) >>>>>>>>> File "/usr/share/weewx/weectllib/__init__.py", line 90, in >>>>>>>>> dispatch >>>>>>>>> namespace.action_func(config_dict, namespace) >>>>>>>>> File "/usr/share/weewx/weectllib/database_cmd.py", line 329, in >>>>>>>>> rebuild_daily >>>>>>>>> weectllib.database_actions.rebuild_daily(config_dict, >>>>>>>>> File "/usr/share/weewx/weectllib/database_actions.py", line 134, >>>>>>>>> in rebuild_daily >>>>>>>>> with weewx.manager.open_manager(manager_dict, initialize=True) >>>>>>>>> as dbm: >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 924, in >>>>>>>>> open_manager >>>>>>>>> return >>>>>>>>> manager_cls.open_with_create(manager_dict['database_dict'], >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 244, in >>>>>>>>> open_with_create >>>>>>>>> dbmanager = cls(connection, table_name=table_name, >>>>>>>>> schema=schema) >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1076, in __init__ >>>>>>>>> self.patch_sums() >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1531, in >>>>>>>>> patch_sums >>>>>>>>> self.recalculate_weights(start_d=datetime.date(2020, 6, 1)) >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1451, in >>>>>>>>> recalculate_weights >>>>>>>>> self._do_tranche(mark_d, end_of_tranche_d, weight_fn, >>>>>>>>> progress_fn) >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1490, in >>>>>>>>> _do_tranche >>>>>>>>> self._set_day_sums(day_accum, cursor) >>>>>>>>> File "/usr/share/weewx/weewx/manager.py", line 1515, in >>>>>>>>> _set_day_sums >>>>>>>>> cursor.execute(update_sql, (day_accum.timespan.start,)) >>>>>>>>> File "/usr/share/weewx/weedb/sqlite.py", line 50, in guarded_fn >>>>>>>>> raise weedb.NoColumnError(e) >>>>>>>>> weedb.NoColumnError: no such column: inf >>>>>>>>> username@host:~$ >>>>>>>>> >>>>>>>>> torek, 24. december 2024 ob 15:15:16 UTC+1 je oseba Tom Keffer >>>>>>>>> napisala: >>>>>>>>> >>>>>>>>>> Let's put aside the database update and just focus on rebuilding >>>>>>>>>> the daily summaries. >>>>>>>>>> >>>>>>>>>> Could you please include the stack trace when doing "weectl >>>>>>>>>> database rebuild-daily"? Once I see where it fails, I can send you an >>>>>>>>>> instrumented version of the database code so we can see exactly what >>>>>>>>>> is >>>>>>>>>> happening. >>>>>>>>>> >>>>>>>>>> -tk >>>>>>>>>> >>>>>>>>>> On Tue, Dec 24, 2024 at 3:39 AM mihec <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Well, I've checked also the weectl database check which fails >>>>>>>>>>> at the same point as the update. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> torek, 24. december 2024 ob 12:31:36 UTC+1 je oseba >>>>>>>>>>> Auchtermuchty Weather napisala: >>>>>>>>>>> >>>>>>>>>>>> That's not what it does now. >>>>>>>>>>>> >>>>>>>>>>>> " Databases created earlier than 3.7.0 (released >>>>>>>>>>>> 11-March-2017) have a flaw that prevents them from being used with >>>>>>>>>>>> archive >>>>>>>>>>>> intervals that change with time. This utility check whether your >>>>>>>>>>>> database >>>>>>>>>>>> is affected." >>>>>>>>>>>> >>>>>>>>>>>> $ weectl database check >>>>>>>>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>>>>>>>> Checking daily summary tables version... >>>>>>>>>>>> Daily summary tables are at version 4.0. >>>>>>>>>>>> Interval Weighting Fix is not required. >>>>>>>>>>>> $ >>>>>>>>>>>> >>>>>>>>>>>> On Tuesday, 24 December 2024 at 11:13:48 UTC Graham Eddy wrote: >>>>>>>>>>>> >>>>>>>>>>>>> weewx used to include a tool for picking up things like a >>>>>>>>>>>>> string where a number should be. maybe `weectl database check’ >>>>>>>>>>>>> still does >>>>>>>>>>>>> this >>>>>>>>>>>>> *⊣GE⊢* >>>>>>>>>>>>> >>>>>>>>>>>>> On 24 Dec 2024, at 4:17 pm, mihec <[email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I think there must be an invalid entry in my database (Aug 12, >>>>>>>>>>>>> 2021). My SQLite database management knowledge is basically zero, >>>>>>>>>>>>> so I'm >>>>>>>>>>>>> looking for advice >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> >>>>>>>>>> To view this discussion visit >>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/e60483cc-3910-45f8-aff4-7b4e95939cd9n%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/e60483cc-3910-45f8-aff4-7b4e95939cd9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> >>>>>>>> To view this discussion visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/e92cf44c-d06f-4de1-ba04-440bceed755fn%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/e92cf44c-d06f-4de1-ba04-440bceed755fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion visit >>>>> https://groups.google.com/d/msgid/weewx-user/124fd290-b296-4c56-a4d4-43e7b30ae2d7n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/weewx-user/124fd290-b296-4c56-a4d4-43e7b30ae2d7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion visit >>> https://groups.google.com/d/msgid/weewx-user/8fe396c2-d6b2-4aac-a8c8-fc014e9d2df8n%40googlegroups.com >>> <https://groups.google.com/d/msgid/weewx-user/8fe396c2-d6b2-4aac-a8c8-fc014e9d2df8n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/bb0c8c81-d7b4-401f-8758-5561660d001fn%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/bb0c8c81-d7b4-401f-8758-5561660d001fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEC%3DrwPak9rKXzD7%3D2GxO67OpT1A%2Bj2n%3D8hBApK8g%2BOwkQ%40mail.gmail.com.
