Bit more progress but slight confusion.
The database did indeed have two rows in every table relating to
the exceptional date above. The last_updated stamp in the archive meta
file was also exceptional. I wrote a little python script (attached) to
correct all these and starting weewx over with the cleaned DB was
successful however then it picked up on the following record from the
logger (?) .
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] INFO __main__: Starting up weewx
version 4.1.1*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Gentle wake up of console successful*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] INFO weewx.engine: Clock error is
1.92 seconds (positive is fast)*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] INFO weewx.engine: Using binding
'wx_binding' to database 'weewx.sdb'*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] INFO weewx.manager: Starting
backfill of daily summaries*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Getting archive packets since 2021-06-16 20:16:00 BST (1623870960)*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Gentle wake up of console successful*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Retrieving 28 page(s); starting index= 0*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] ERROR weewx.engine: Internal
error detected. Catchup abandoned*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] ERROR weewx.engine: **** Unknown
archive type = 0xbf*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] INFO weewx.engine: Starting main
packet loop.*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Gentle wake up of console successful*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Requesting 200 LOOP packets.*
*Jun 17 09:54:33 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Gentle wake up of console successful*
*Jun 17 09:55:14 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Getting archive packets since 2021-06-16 20:16:00 BST (1623870960)*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Gentle wake up of console successful*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Retrieving 28 page(s); starting index= 0*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] WARNING weewx.engine: ignore
historical record: {'dateTime': 4996107120, 'usUnits': 1, 'interval': 1,
'rxCheckPercent': 100.0, 'outTemp': 1612.4, 'highOutTemp': 3257.5,
'lowOutTemp': 3263.9, 'rain': 223.9921246179, 'rainRate': 507.9606268227,
'barometer': 65.535, 'radiation': 65519.0, 'wind_samples': 65535.0,
'inTemp': -156.9, 'inHumidity': 155.0, 'outHumidity': 187.0, 'windSpeed':
127.0, 'windGust': 231.0, 'windGustDir': 2497.5, 'windDir': 2857.5, 'UV':
24.9, 'ET': 0.248, 'soilMoist1': 111.0, 'soilMoist3': 223.0, 'soilTemp3':
133.0, 'leafWet2': 251.0, 'leafWet3': 239.0, 'extraTemp2': 164.0,
'extraHumid1': 158.0, 'extraHumid2': 127.0, 'readClosed': 61951,
'readOpened': 32694}*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
DMPAFT complete: page timestamp 2127-12-28 07:35:00 GMT (4985652900) less
than final timestamp 2128-04-27 07:32:00 GMT (4996107120)*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Catch up complete.*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.reportengine: Running
reports for latest time in the database.*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Requesting 200 LOOP packets.*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.reportengine: Running
report 'StandardReport'*
*Jun 17 09:55:15 pecan-ubuntu weewx[3255] DEBUG weewx.drivers.vantage:
Gentle wake up of console successful*
So, I thought I would try one more clear-memory before factory resetting
the VUE as clearing data previously has not worked and hopefully clean the
data it has stored on board and this is the error message now!
*auser@pecan-ubuntu:/home/weewx$ sudo ./bin/wee_device --clear-memory*
*Using configuration file /home/weewx/weewx.conf*
*Using Vantage driver version 3.2.1 (weewx.drivers.vantage)*
*Traceback (most recent call last):*
* File "./bin/wee_device", line 78, in <module>*
* main()*
* File "./bin/wee_device", line 74, in main*
* device.configure(config_dict)*
* File "/home/weewx/bin/weewx/drivers/__init__.py", line 67, in configure*
* self.do_options(options, parser, config_dict, not options.noprompt)*
* File "/home/weewx/bin/weewx/drivers/vantage.py", line 2055, in
do_options*
* station = Vantage(**config_dict[DRIVER_NAME])*
* File "/home/weewx/bin/weewx/drivers/vantage.py", line 515, in __init__*
* log.debug("Hardware name: %s", self.hardware_name)*
* File "/home/weewx/bin/weewx/drivers/vantage.py", line 1292, in
hardware_name*
* raise weewx.UnsupportedFeature("Unknown hardware type %d" %
self.hardware_type)*
*weewx.UnsupportedFeature: Unknown hardware type 10*
I'm going to have to leave it for this morning and maybe you'll come back
to me with some thoughts. Thanks again for the help.
Chris
On Thursday, June 17, 2021 at 8:13:01 AM UTC+1 Chris Davies-Barnard wrote:
> So a quick update. I took a copy of the database and checked the archive
> table. I seem to have acquired these two rows at the top. I have removed
> them and copied the database back and I get a new error but believe I am
> getting somewhere:
>
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: Caught
> unrecoverable exception:*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> lastUpdate(2112-04-20 11:47:00 GMT (4490596020)) > lastRecord(2021-06-16
> 20:16:00 BST (1623870960))*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> Traceback (most recent call last):*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> File "/home/weewx/bin/weewxd", line 154, in main*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> engine.run()*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> File "/home/weewx/bin/weewx/engine.py", line 158, in run*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> self.dispatchEvent(weewx.Event(weewx.STARTUP))*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> File "/home/weewx/bin/weewx/engine.py", line 224, in dispatchEvent*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> callback(event)*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> File "/home/weewx/bin/weewx/engine.py", line 522, in startup*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> _nrecs, _ndays = dbmanager.backfill_day_summary()*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> File "/home/weewx/bin/weewx/manager.py", line 1012, in backfill_day_summary*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> timestamp_to_string(lastRecord)))*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> weewx.ViolatedPrecondition: lastUpdate(2112-04-20 11:47:00 GMT
> (4490596020)) > lastRecord(2021-06-16 20:16:00 BST (1623870960))*
> *Jun 17 08:11:01 pecan-ubuntu weewx[2098] CRITICAL __main__: ****
> Exiting.*
>
> So, my next question is where in the database will I find referrences to
> this exceptional date?
>
> Thanks again
>
> Chris
>
> [image: Capture.PNG]
>
> On Thursday, June 17, 2021 at 7:56:53 AM UTC+1 Chris Davies-Barnard wrote:
>
>> Morning Gary,
>>
>> Thanks for replying. I hope its not a faulty unit as that could be
>> the death of the weather station for now. She is over 10 years old now
>> and I replaced one of the sensor modules only last year. If she does die
>> I think I would build my own open source kit maybe? Anyway getting ahead
>> of myself.
>>
>> *1) Stop Weewx*
>> *It seems at some point in my wisdom I set up weewx as a Daemon so I have
>> removed weewx from /etc/init.d and restarted the server (Its ubuntu running
>> on QNAP's Virtualisation Station).*
>>
>> *2) Checking for remaining processes*
>> *auser@pecan-ubuntu:/home/weewx$ ps aux | grep -i weewx*
>> *auser** 1238 0.0 0.0 14428 1032 pts/0 S+ 07:34 0:00 grep
>> --color=auto -i weewx*
>> *auser**@pecan-ubuntu**:/home/weewx$ pidof weewx*
>> *auser**@pecan-ubuntu**:/home/weewx$ *
>>
>> *3) sudo ./bin/wee_device --clear-memory *
>> *Using configuration file /home/weewx/weewx.conf*
>> *Using Vantage driver version 3.2.1 (weewx.drivers.vantage)*
>> *Proceeding will erase all archive records.*
>> *Are you sure you wish to proceed (y/n)? y*
>> *Erasing all archive records ...*
>> *Archive records erased.*
>>
>> *4) set debug = 1 in weewx.conf and restart WeeWX with sudo
>> ./bin/weewxd*
>> syslog output is in attached file but run terminal output is like:
>>
>>
>> *LOOP: 2021-06-17 07:40:57 BST (1623912057) altimeter:
>> 29.86448769405689, appTemp: 54.776612632572494, barometer: 29.855,
>> beaufort: 0, cloudbase: 1583.4599268474649, consBatteryVoltage: 4.48,
>> dateTime: 1623912057, dayET: 0.0, dayRain: 0.0, dewpoint:
>> 50.709576321871154, extraAlarm1: 0, extraAlarm2: 0, extraAlarm3: 0,
>> extraAlarm4: 0, extraAlarm5: 0, extraAlarm6: 0, extraAlarm7: 0,
>> extraAlarm8: 0, forecastIcon: 6, forecastRule: 45, heatindex: 54.5,
>> humidex: 57.13247206192764, inDewpoint: 55.93846853316417, inHumidity:
>> 59.0, insideAlarm: 0, inTemp: 71.0, leafWet4: 0.0, maxSolarRad:
>> 363.4009700897632, monthET: 0.0, monthRain: 0.0, outHumidity: 87.0,
>> outsideAlarm1: 0, outsideAlarm2: 0, outTemp: 54.5, pressure: 29.09*
>>
>> The loop date is correct so the question I have here is where has
>> 2112-04-20 come from below?
>>
>> * DEBUG weewx.drivers.vantage: Getting archive packets since 2112-04-20
>> 11:47:00 GMT (4490596020)*
>>
>> *Jun 17 07:41:14 pecan-ubuntu weewx[1372] DEBUG weewx.drivers.vantage:
>> Gentle wake up of console successfulJun 17 07:41:14 pecan-ubuntu
>> weewx[1372] DEBUG weewx.drivers.vantage: Retrieving 513 page(s); starting
>> index= 3Jun 17 07:41:15 pecan-ubuntu weewx[1372] ERROR weewx.engine:
>> Internal error detected. Catchup abandonedJun 17 07:41:15 pecan-ubuntu
>> weewx[1372] ERROR weewx.engine: **** Unknown archive type = 0x37*
>> I think my next step is going to be grabbing a copy of the database and
>> seeing if I can see any problems but if you have any thoughts or solutions
>> in mind I'd be grateful.
>>
>> Thanks
>>
>> Chris
>> On Thursday, June 17, 2021 at 12:15:48 AM UTC+1 gjr80 wrote:
>>
>>> The few times this error seems to have come up the solution seems to be
>>> clearing the logger memory or failing that contacting Davis due to a faulty
>>> logger.
>>>
>>> I would not expect wee_device --dump to do anything different, it
>>> essentially is doing the same as the startup catchup just for all records
>>> in the logger memory. You say wee_device --clear-memory 'seems to
>>> succeed', what does that mean, the --clear-memory command gives
>>> appropriate output but subsequent restarting WeeWX causes the same output?
>>> or does it mean that everything cleared up but the problem came back later?
>>>
>>> What does 'clear the logger using its own functions' mean? I thought the
>>> only way to clear the logger was to use some suitable/capable software? And
>>> do you still have a phantom WeeWX process running?
>>>
>>> How about trying this:
>>> 1. stop WeeWX
>>> 2. check for any remaining WeeWX processes and if any exist kill them
>>> manually
>>> 3. run wee_device --clear-memory and post the command used and the
>>> resulting output
>>> 4. set debug = 1 in weewx.conf and restart WeeWX, post the WeeWX log
>>> output from startup through until an error occurs or data is obtained
>>> successfully from the station
>>>
>>> Gary
>>>
>>> On Thursday, 17 June 2021 at 06:25:02 UTC+10 Chris Davies-Barnard wrote:
>>>
>>>> Evening all,
>>>>
>>>> So I had to move the logger and ubuntu box today and upon reconnecting
>>>> and restarting I have encountered the following error.
>>>>
>>>> J*un 16 21:00:16 pecan-ubuntu weewx[1823] DEBUG weewx.drivers.vantage:
>>>> Getting archive packets since 2112-04-20 11:47:00 GMT (4490596020)*
>>>> *Jun 16 21:00:16 pecan-ubuntu weewx[1823] DEBUG weewx.drivers.vantage:
>>>> Gentle wake up of console successful*
>>>> *Jun 16 21:00:16 pecan-ubuntu weewx[1823] DEBUG weewx.drivers.vantage:
>>>> Retrieving 513 page(s); starting index= 2*
>>>> *Jun 16 21:00:16 pecan-ubuntu weewx[1823] ERROR weewx.engine: Internal
>>>> error detected. Catchup abandoned*
>>>> *Jun 16 21:00:16 pecan-ubuntu weewx[1823] ERROR weewx.engine: ****
>>>> Unknown archive type = 0xae*
>>>> *Jun 16 21:00:16 pecan-ubuntu weewx[1823] DEBUG weewx.reportengine:
>>>> Running reports for latest time in the database.*
>>>> *Jun 16 21:00:16 pecan-ubuntu weewx[1823] DEBUG weewx.drivers.vantage:
>>>> Requesting 200 LOOP packets.*
>>>>
>>>> I have tried
>>>>
>>>> 1) sudo ./bin/wee_device --dump which ends in same/similar error
>>>> 2) sudo ./bin/wee_device --clear-memory which seems to succeed.
>>>>
>>>> I have also cleared the logger using its own functions but thee
>>>> situation is the same..
>>>>
>>>> What's interesting is that stopping weewx seems to cause a lock and it
>>>> never completes.
>>>>
>>>> If anyone could help me solve the problem I would be very grateful.
>>>>
>>>> Thanks
>>>>
>>>> Chris
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/b03531e7-1f33-4db6-9b7c-44cc7900c300n%40googlegroups.com.
# This is a sample Python script.
import sqlite3
from datetime import datetime
import time
class WeewxDBFixer():
db_file = "weewx.sdb"
metaDataTable = "archive_day__metadata"
conn = None
table_names = []
ts_value = None
def __init__(self, auto):
print("Creating a weewxDBFixer object for " + self.db_file)
self.create_connection()
if auto==True:
self.select_all_tables()
self.select_tables_with_high_timestamps()
def create_connection(self):
""" create a database connection to the SQLite database
specified by the db_file
"""
try:
self.conn = sqlite3.connect(self.db_file)
except Error as e:
print(e)
def select_all_tables(self):
""" select all the tables in the given SQLite daatbase
and add to the table_names list
"""
cur = self.conn.cursor()
cur.execute("SELECT name FROM sqlite_master WHERE type = 'table';")
rows = cur.fetchall()
for row in rows:
self.table_names.append(row[0])
print(self.table_names)
def getTodaysTimestamp(self):
#ts = datetime.now(tz=None)
ts = int(time.time())
print("Working with " + str(ts)) #.strftime("%d-%b-%Y (%H:%M:%S.%f)"))
return ts
def select_tables_with_high_timestamps(self):
""" Selects tables with high timestamps
"""
cur = self.conn.cursor()
self.ts_value = self.getTodaysTimestamp()
sql = "SELECT * FROM {table_name} WHERE dateTime > '{ts}'"
for atable in self.table_names:
try:
tmp = sql.format(table_name=atable, ts=self.ts_value)
print("Working with: " + tmp)
cur.execute(tmp)
rows = cur.fetchall()
print(rows)
if len(rows) > 0:
self.deleteRowsFromTables(atable, rows)
self.updateMetaData(self.ts_value, "lastUpdate")
else:
print("No Rows")
except sqlite3.OperationalError as e:
print("Error:" + atable + " with " + str(e))
def deleteRowsFromTables(self, atable, rows):
print("Deleting from: " + atable + " with " + str(len(rows)) + "
row(s)")
sql = "DELETE FROM {table_name} WHERE dateTime > '{ts}'"
tmp = sql.format(table_name=atable, ts=self.ts_value)
cur = self.conn.cursor()
cur.execute(tmp)
self.conn.commit()
def updateMetaData(self, value, name):
sql = "UPDATE " + self.metaDataTable + " SET value = {avalue} WHERE
name = '{aname}'"
tmp = sql.format(avalue=value, aname=name)
print("Updating Meta Data: " + tmp)
cur = self.conn.cursor()
cur.execute(tmp)
self.conn.commit()
if __name__ == '__main__':
WeewxDBFixer(True)
