Gary Wouldn't / shouldn't wee_database give an error message if there is a permissions issue?? Andrew
On 2 August 2018 at 06:39, gjr80 <[email protected]> wrote: > This could be a permissions problem on /var/lib/weewx/weewx.sdb, from > memory a default install does not give an unprivileged user write > permission to weewx.sdb. Try prefixing your wee_database commands (ie > --drop-daily and --rebuild-daily) with sudo. > > Gary > > On Thursday, 2 August 2018 04:40:37 UTC+10, Paul_N wrote: >> >> Hi Andrew, I finally managed to find some spare time to try out your >> latest suggestion, sorry for the delay. Here's the result of running those >> two commands (please note that I used "sudo find" else I get lots of >> "permission denied" messages from numerous non-related RPi directories). >> Does this shed any new light on things ? >> Thanks again, Paul >> >> pi@raspberrypi:~ $ sudo find / -name weewx.conf >> /etc/weewx/logwatch/conf/services/weewx.conf >> /etc/weewx/logwatch/conf/logfiles/weewx.conf >> /etc/weewx/rsyslog.d/weewx.conf >> /etc/weewx/apache/conf.d/weewx.conf >> /etc/weewx/weewx.conf >> find: ‘/run/user/1000/gvfs’: Permission denied >> >> pi@raspberrypi:~ $ sudo find / -name weewx.sdb >> /var/lib/weewx/weewx.sdb >> find: ‘/run/user/1000/gvfs’: Permission denied >> pi@raspberrypi:~ $ >> >> ==================================================================== >> >> On Wednesday, August 1, 2018 at 12:08:31 PM UTC+1, Paul_N wrote: >>> >>> Thanks for that suggestion Andrew, sounds very possible ! I'll check it >>> out later today when I next get a spare moment and will let you know how I >>> get on. >>> Thanks again, Paul >>> >>> On Wednesday, August 1, 2018 at 11:57:36 AM UTC+1, Andrew Milner wrote: >>>> >>>> sorry - last one should have been >>>> find / -name weewx.sdb >>>> >>>> >>>> >>>> On Wednesday, 1 August 2018 13:56:46 UTC+3, Andrew Milner wrote: >>>>> >>>>> My suspicion is that you have either got two weewx.conf files or two >>>>> .sdb files and SQLite is looking in one place whilst drop daly is dropping >>>>> tables from a different .sdb file. >>>>> >>>>> You can check by using find >>>>> find / -name weewx.conf >>>>> and >>>>> find / -name archive.sdb >>>>> >>>>> >>>>> >>>>> On Wednesday, 1 August 2018 13:26:04 UTC+3, Paul_N wrote: >>>>>> >>>>>> Hi Andrew, thanks for this - and I think we're getting somewhere : it >>>>>> would appear that the drop-daily command is not working. I followed your >>>>>> first four steps and attach below the output from my terminal. As you'll >>>>>> see the initial query found the rogue outTemp min values but they were >>>>>> still there when I repeated that query AFTER doing the drop-daily >>>>>> command. >>>>>> So, if drop-daily is supposed to clear these values then it has not >>>>>> worked. >>>>>> >>>>>> Any suggestions please ? >>>>>> Thanks again, Paul (ps please note that all the temperature values >>>>>> below are actually in Fahrenheit because that is how my db is set up, >>>>>> hence >>>>>> the rogue min values are actually below 5 deg F). >>>>>> >>>>>> [ ok ] Stopping weewx (via systemctl): weewx.service. >>>>>> weewx stopped >>>>>> >>>>>> pi@raspberrypi:/var/lib/weewx $ sqlite3 weewx.sdb >>>>>> SQLite version 3.16.2 2017-01-06 16:32:41 >>>>>> Enter ".help" for usage hints. >>>>>> sqlite> select dateTime, min from archive_day_outTemp where min < 5; >>>>>> 1523919600|-3.46 >>>>>> 1524006000|-3.64 >>>>>> 1524438000|-3.82 >>>>>> sqlite> >>>>>> >>>>>> pi@raspberrypi:/var/lib/weewx $ cd /etc/weewx >>>>>> pi@raspberrypi:/etc/weewx $ wee_database weewx.conf --drop-daily >>>>>> Using configuration file weewx.conf >>>>>> Using database binding 'wx_binding', which is bound to database >>>>>> 'archive_sqlite' >>>>>> Proceeding will delete all your daily summaries from database >>>>>> 'weewx.sdb' >>>>>> Are you sure you want to proceed (y/n)? y >>>>>> Dropping daily summary tables from 'weewx.sdb' ... >>>>>> Daily summary tables dropped from database 'weewx.sdb' in 0.04 seconds >>>>>> >>>>>> pi@raspberrypi:/etc/weewx $ cd /var/lib/weewx >>>>>> pi@raspberrypi:/var/lib/weewx $ sqlite3 weewx.sdb >>>>>> SQLite version 3.16.2 2017-01-06 16:32:41 >>>>>> Enter ".help" for usage hints. >>>>>> sqlite> select dateTime, min from archive_day_outTemp where min < 5; >>>>>> 1523919600|-3.46 >>>>>> 1524006000|-3.64 >>>>>> 1524438000|-3.82 >>>>>> sqlite> >>>>>> pi@raspberrypi:/var/lib/weewx $ >>>>>> >>>>>> =============================================================== >>>>>> >>>>>> On Wednesday, August 1, 2018 at 11:04:08 AM UTC+1, Andrew Milner >>>>>> wrote: >>>>>>> >>>>>>> 1. stop weewx >>>>>>> >>>>>>> 2. try and see if your rogue data is in the summaries: >>>>>>> >>>>>>> select dateTime, min from archive_day_outTemp where min < -19; >>>>>>> >>>>>>> hopefully you will find the rogue day!! >>>>>>> >>>>>>> >>>>>>> 3. try and drop daily again >>>>>>> >>>>>>> 4. check that they have dropped by repeating the command: >>>>>>> select dateTime,min from archive_day_outTemp where min < -19; >>>>>>> >>>>>>> Hopefully none will be found - in fact you may even get an error. >>>>>>> >>>>>>> 5. check there is no rogue value in the archive with >>>>>>> select * from archive where outTemp < -19; >>>>>>> >>>>>>> 6. rebuild daily - this will rebuild all the archive_day tables >>>>>>> from the archive data >>>>>>> >>>>>>> 7. check the summary table no longer contains the rogue data >>>>>>> select * from archive_day_outTemp where min < -19; >>>>>>> >>>>>>> Hopefully you will have solved it. >>>>>>> >>>>>>> How long rebuild takes depends on loads of things, but should take a >>>>>>> few seconds at least I would have thought and not been instantaneous. >>>>>>> >>>>>>> DO REMEMBER THE ; ON THE END OF SQL COMMANDS >>>>>>> >>>>>>> >>>>>>> On Wednesday, 1 August 2018 11:13:01 UTC+3, Paul_N wrote: >>>>>>>> >>>>>>>> Hi Andrew, thanks for that suggestion. I have checked the database >>>>>>>> with that query and can confirm that there are no rogue values left in >>>>>>>> there. So the problem appears to be in the re-calculation of the >>>>>>>> minimum >>>>>>>> outdoor temperature for that month. I do not really understand exactly >>>>>>>> what >>>>>>>> the "rebuild-daily" command does - is that when these values get >>>>>>>> calculated >>>>>>>> ? If so, then maybe my observation that this command completed >>>>>>>> instantly >>>>>>>> might be relevant. There is a about 8 months of data in the database so >>>>>>>> should it have taken some while to run, as suggested in Tom K's wiki ? >>>>>>>> >>>>>>>> Thanks again for any advice you could give me, >>>>>>>> Regards, Paul >>>>>>>> >>>>>>>> On Wednesday, August 1, 2018 at 2:55:37 AM UTC+1, Andrew Milner >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> have you rechecked the database with >>>>>>>>> >>>>>>>>> select dateTime, outTemp from archive where outTemp < 19; >>>>>>>>> >>>>>>>>> to see if there are still any rogue temperature records in there? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, 31 July 2018 21:24:15 UTC+3, Paul_N wrote: >>>>>>>>>> >>>>>>>>>> Hi, I'd appreciate some advice on the problem I've been having >>>>>>>>>> with removing some erroneous outside temperature readings from my >>>>>>>>>> weewx >>>>>>>>>> system which occurred very early on in my set-up when the sensor was >>>>>>>>>> not >>>>>>>>>> connected - and before I discovered the value of setting StdQC >>>>>>>>>> properly ! >>>>>>>>>> The erroneous values were very low outside temperatures (-19.9 deg >>>>>>>>>> C), >>>>>>>>>> causing the minimum values to be reported for that month and for the >>>>>>>>>> year >>>>>>>>>> to be incorrect. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I followed the steps described by Tom K in his wiki guide >>>>>>>>>> <https://github.com/weewx/weewx/wiki/Cleaning-up-old-'bad'-data> >>>>>>>>>> and all seemed to go well: I was able to locate and remove (set to >>>>>>>>>> null) >>>>>>>>>> the erroneous outTemp readings in the database and then drop and >>>>>>>>>> rebuild >>>>>>>>>> the daily summaries OK. As advised elsewhere on this forum I also >>>>>>>>>> deleted >>>>>>>>>> all the PNG,HTML and NOAA text files so they were re-created after I >>>>>>>>>> did >>>>>>>>>> the database editing. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The good news is that the png chart which shows monthly Min/Max >>>>>>>>>> outside temperatures no longer shows these erroneous values and so I >>>>>>>>>> am >>>>>>>>>> encouraged to think that they really are gone from the database. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> However, the NOAA summary for that month (and also the yearly >>>>>>>>>> one) still shows this bad data so it seems that whatever calculates >>>>>>>>>> the >>>>>>>>>> minimum outside temperature has not recalculated it for some reason. >>>>>>>>>> Please >>>>>>>>>> note that these NOAA summaries have definitely been recreated since >>>>>>>>>> the >>>>>>>>>> database editing as I deleted them after doing it and before >>>>>>>>>> re-starting >>>>>>>>>> weewx. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> One thing I did note which could be indicative is that although >>>>>>>>>> the "rebuild-daily" command appeared to execute OK, it completed >>>>>>>>>> almost >>>>>>>>>> instantaneously whereas Tom's wiki guide led me to believe it would >>>>>>>>>> take >>>>>>>>>> some while. Could it be that this has not actually worked and hence >>>>>>>>>> is the >>>>>>>>>> cause of my problem ? How can I check to see if it has done what is >>>>>>>>>> expected ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Just in case it's relevant, here is what I saw on the terminal >>>>>>>>>> when dropping and rebuilding the daily summaries: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> pi@raspberrypi:/etc/weewx $ wee_database weewx.conf --drop-daily >>>>>>>>>> >>>>>>>>>> Using configuration file weewx.conf >>>>>>>>>> >>>>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>>>> 'archive_sqlite' >>>>>>>>>> >>>>>>>>>> Proceeding will delete all your daily summaries from database >>>>>>>>>> 'weewx.sdb' >>>>>>>>>> >>>>>>>>>> Are you sure you want to proceed (y/n)? y >>>>>>>>>> >>>>>>>>>> Dropping daily summary tables from 'weewx.sdb' ... >>>>>>>>>> >>>>>>>>>> Daily summary tables dropped from database 'weewx.sdb' in 0.04 >>>>>>>>>> seconds >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> pi@raspberrypi:/etc/weewx $ wee_database weewx.conf >>>>>>>>>> --rebuild-daily >>>>>>>>>> >>>>>>>>>> Using configuration file weewx.conf >>>>>>>>>> >>>>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>>>> 'archive_sqlite' >>>>>>>>>> >>>>>>>>>> All daily summaries will be rebuilt. >>>>>>>>>> >>>>>>>>>> Proceed (y/n)? y >>>>>>>>>> >>>>>>>>>> Rebuilding daily summaries in database 'weewx.sdb' ... >>>>>>>>>> >>>>>>>>>> Daily summaries up to date in 'weewx.sdb' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> With thanks in advance for any help you could give to enable me >>>>>>>>>> to track down this problem, >>>>>>>>>> >>>>>>>>>> Regards, Paul >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- > You received this message because you are subscribed to a topic in the > Google Groups "weewx-user" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/weewx-user/fZtyq_cTyZ4/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
