Re: [weewx-user] 'wee_database --calc-missing' long to execute

[William Garber]

This is a year of data.  It works much better if you process it in small 
chunks.  *I think this is a bug.  *
I changed the schema so it was necessary to add --rebuild-daily first.
The following script behaves radically differently depending on the step 
size (+4 is good).
If the step size is too large (too many days at once) it gets stuck at 6000 
records predictably.
Small chunks are effective but you have to press "y" to continue every 4 
days.

#!/bin/bash
rm end-date.log
# next line helps a lot
wee_database --rebuild-daily
for (( past=-392; past <= 0; past=past+4 ))
# for (( past=-301; past <= 0; past=++ ))
do
past1=$(( past ))
past2=$(( past + 4 ))
DATE1=$(date -d "today $past1 days" +%F)
DATE2=$(date -d "today $past2 days" +%F)
echo
echo "processing DATE=$DATE1"
# wee_database --calc-missing --date="$DATE1"
wee_database --calc-missing --from="$DATE1" --to="$DATE2"
res=$?
echo "res=$res"
[[ $res -eq 0 ]] || exit 1
echo $DATE >> end-date.log
done
# the final one runs MUCH FASTER and fixes any glitches
wee_database --calc-missing 
# eee eof


On Monday, February 20, 2023 at 4:46:29 AM UTC-8 William Garber wrote:

> I am having the same problem.  I have about 130,000 different datetimes 
> (records) in weewx.sdb.  I also tried moving it to a ramdisk.  Still 
> extremely slow.  Any help please?  Should I run calc-missing on individual 
> timeslices like one week periods?  The data covers one year of measurements.
>
> On Monday, March 14, 2022 at 7:50:36 PM UTC-7 graha...@gmail.com wrote:
>
>> i reduced the --calc-missing time from 7 *hours* to 7 *minutes* by such 
>> a simple trick that i kick myself for not seeing it earlier - i moved the 
>> database to ramdisk and symlinked it under archive, ran wee_database, then 
>> moved the database back. chalk this one under 'handy tips'
>>
>> On Thursday, 10 March 2022 at 11:09:25 am UTC+11 graha...@gmail.com 
>> wrote:
>>
>>> for gw1000 dataset, i had been using top and iotop previously to 
>>> ascertain that cpu, memory and i/o usage were extremely low, and ps-efl 
>>> showed it was spending its time waiting on interrupt. i would usually just 
>>> conclude it was slow disk and spending all its time waiting on i/o 
>>> completion *except* this is only for the smaller gw1000 dataset not the 
>>> larger vp2 dataset. it is something to do with the different nature of the 
>>> data (perhaps something as simple as different missing data being calc’ed)
>>>
>>> to give an idea of the magnitude of the difference, using built-in shell 
>>> time to run command:
>>>
>>>
>>> *recs* *real /sec* *user /sec* *sys /sec* *Idle /%* 
>>> vp2 --rebuild-daily 505,336 165 148 2 9 
>>> vp2 --calc-missing 505,336 571 525 18 5 
>>> gw1000 --rebuild-daily 162,882 86 81 1 5 
>>> gw1000 --calc-missing 162,882 23,758 301 13 99
>>>
>> [snip] 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8d420428-a941-4220-ba9e-5b73456b5539n%40googlegroups.com.

Re: [weewx-user] 'wee_database --calc-missing' long to execute

[William Garber]

I am having the same problem.  I have about 130,000 different datetimes 
(records) in weewx.sdb.  I also tried moving it to a ramdisk.  Still 
extremely slow.  Any help please?  Should I run calc-missing on individual 
timeslices like one week periods?  The data covers one year of measurements.

On Monday, March 14, 2022 at 7:50:36 PM UTC-7 graha...@gmail.com wrote:

> i reduced the --calc-missing time from 7 *hours* to 7 *minutes* by such a 
> simple trick that i kick myself for not seeing it earlier - i moved the 
> database to ramdisk and symlinked it under archive, ran wee_database, then 
> moved the database back. chalk this one under 'handy tips'
>
> On Thursday, 10 March 2022 at 11:09:25 am UTC+11 graha...@gmail.com wrote:
>
>> for gw1000 dataset, i had been using top and iotop previously to 
>> ascertain that cpu, memory and i/o usage were extremely low, and ps-efl 
>> showed it was spending its time waiting on interrupt. i would usually just 
>> conclude it was slow disk and spending all its time waiting on i/o 
>> completion *except* this is only for the smaller gw1000 dataset not the 
>> larger vp2 dataset. it is something to do with the different nature of the 
>> data (perhaps something as simple as different missing data being calc’ed)
>>
>> to give an idea of the magnitude of the difference, using built-in shell 
>> time to run command:
>>
>>
>> *recs* *real /sec* *user /sec* *sys /sec* *Idle /%* 
>> vp2 --rebuild-daily 505,336 165 148 2 9 
>> vp2 --calc-missing 505,336 571 525 18 5 
>> gw1000 --rebuild-daily 162,882 86 81 1 5 
>> gw1000 --calc-missing 162,882 23,758 301 13 99
>>
> [snip] 
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/714d7461-fb59-4b36-9350-cff877203516n%40googlegroups.com.

Re: [weewx-user] wee_database --update Error

[Joachim Puttkammer]

> Replace /usr/share/weewx/weecfg/database.py with this version.
it works without any error. thanks


 

> On Sun, Dec 25, 2022 at 9:44 AM Joachim Puttkammer  
> wrote:
>
>> Raspi4, Debian Buster, weewx 4.9.1, two instancen
>> Hello,
>> the command ended with an error:
>>
>> sudo wee_database --config=/etc/weewx/dav.conf --update --dry-run
>> Using configuration file /etc/weewx/dav.conf
>> Using database binding 'wx_binding', which is bound to database 
>> 'archive_sqlite'
>> Preparing interval weighting fix...
>> Daily summary tables are at version 4.0
>> Interval weighting fix is not required.
>> Preparing maximum windSpeed fix...
>> This is a dry run: maximum windSpeed will be calculated but not saved.
>> Traceback (most recent call last):
>>   File "/usr/share/weewx/wee_database", line 1168, in 
>> main()
>>   File "/usr/share/weewx/wee_database", line 232, in main
>> update(config_dict, db_binding, options)
>>   File "/usr/share/weewx/wee_database", line 765, in update
>> _fix_wind(config_dict, db_binding, options)
>>   File "/usr/share/weewx/wee_database", line 887, in _fix_wind
>> wind_obj.run()
>>   File "/usr/share/weewx/weecfg/database.py", line 199, in run
>> self.do_fix()
>>   File "/usr/share/weewx/weecfg/database.py", line 235, in do_fix
>> for day_span in self.genSummaryDaySpans(tr_start_ts, tr_stop_ts, 
>> 'windSpeed'):
>>   File "/usr/share/weewx/weecfg/database.py", line 102, in 
>> genSummaryDaySpans
>> yield weeutil.weeutil.archiveDaySpan(_row[0], grace=0)
>> TypeError: archiveDaySpan() got an unexpected keyword argument 'grace'
>>
>> What to do?
>>
>> Merry Christmas
>> Joachim
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/9507a9ee-4027-4e30-9a22-3c02b3218337n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/0e7123ac-70ae-4f6b-b7e4-a0fe8ed3676en%40googlegroups.com.

Re: [weewx-user] wee_database --update Error

[Tom Keffer]

My fault. The signature of archiveDaySpan() changed and I forgot to fix
weecfg/database.py so it uses the new signature.

Replace /usr/share/weewx/weecfg/database.py with this version

.

On Sun, Dec 25, 2022 at 9:44 AM Joachim Puttkammer 
wrote:

> Raspi4, Debian Buster, weewx 4.9.1, two instancen
> Hello,
> the command ended with an error:
>
> sudo wee_database --config=/etc/weewx/dav.conf --update --dry-run
> Using configuration file /etc/weewx/dav.conf
> Using database binding 'wx_binding', which is bound to database
> 'archive_sqlite'
> Preparing interval weighting fix...
> Daily summary tables are at version 4.0
> Interval weighting fix is not required.
> Preparing maximum windSpeed fix...
> This is a dry run: maximum windSpeed will be calculated but not saved.
> Traceback (most recent call last):
>   File "/usr/share/weewx/wee_database", line 1168, in 
> main()
>   File "/usr/share/weewx/wee_database", line 232, in main
> update(config_dict, db_binding, options)
>   File "/usr/share/weewx/wee_database", line 765, in update
> _fix_wind(config_dict, db_binding, options)
>   File "/usr/share/weewx/wee_database", line 887, in _fix_wind
> wind_obj.run()
>   File "/usr/share/weewx/weecfg/database.py", line 199, in run
> self.do_fix()
>   File "/usr/share/weewx/weecfg/database.py", line 235, in do_fix
> for day_span in self.genSummaryDaySpans(tr_start_ts, tr_stop_ts,
> 'windSpeed'):
>   File "/usr/share/weewx/weecfg/database.py", line 102, in
> genSummaryDaySpans
> yield weeutil.weeutil.archiveDaySpan(_row[0], grace=0)
> TypeError: archiveDaySpan() got an unexpected keyword argument 'grace'
>
> What to do?
>
> Merry Christmas
> Joachim
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/9507a9ee-4027-4e30-9a22-3c02b3218337n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEA7eoOfb0MnLEYwT73_AmucgP0L4_qBX9yDLUUw_XfmOA%40mail.gmail.com.

[weewx-user] wee_database --update Error

[Joachim Puttkammer]

Raspi4, Debian Buster, weewx 4.9.1, two instancen
Hello,
the command ended with an error:

sudo wee_database --config=/etc/weewx/dav.conf --update --dry-run
Using configuration file /etc/weewx/dav.conf
Using database binding 'wx_binding', which is bound to database 
'archive_sqlite'
Preparing interval weighting fix...
Daily summary tables are at version 4.0
Interval weighting fix is not required.
Preparing maximum windSpeed fix...
This is a dry run: maximum windSpeed will be calculated but not saved.
Traceback (most recent call last):
  File "/usr/share/weewx/wee_database", line 1168, in 
main()
  File "/usr/share/weewx/wee_database", line 232, in main
update(config_dict, db_binding, options)
  File "/usr/share/weewx/wee_database", line 765, in update
_fix_wind(config_dict, db_binding, options)
  File "/usr/share/weewx/wee_database", line 887, in _fix_wind
wind_obj.run()
  File "/usr/share/weewx/weecfg/database.py", line 199, in run
self.do_fix()
  File "/usr/share/weewx/weecfg/database.py", line 235, in do_fix
for day_span in self.genSummaryDaySpans(tr_start_ts, tr_stop_ts, 
'windSpeed'):
  File "/usr/share/weewx/weecfg/database.py", line 102, in 
genSummaryDaySpans
yield weeutil.weeutil.archiveDaySpan(_row[0], grace=0)
TypeError: archiveDaySpan() got an unexpected keyword argument 'grace'

What to do?

Merry Christmas
Joachim

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/9507a9ee-4027-4e30-9a22-3c02b3218337n%40googlegroups.com.

Re: [weewx-user] wee_database

[Mark Fraser]

On 08/11/2022 12:45, Tom Keffer wrote:

Don't know why you'd have to edit weewx.conf, but that looks reasonable.

On Tue, Nov 8, 2022 at 4:33 AM Mark Fraser > wrote:


On 08/11/2022 12:30, Tom Keffer wrote:
 > For any mutating change you should not only stop weewx, but
backup the
 > database.
 >
 > On Tue, Nov 8, 2022 at 1:24 AM Mark Fraser mailto:mfra...@gmail.com>
 > >> wrote:
 >
 >     I'm planning on renaming 2 columns in my weewx database using
 >     wee_database --rename-column --to-name.
 >
 >     Before I do this is it recommended that I stop weewx first?

Yes, I was planning on doing a backup too. So I think my plan is:
1) Edit weewx.conf
2) Stop weewx
3) Back up database
4) Rename columns
5) Restart weewx


I need to change the name of 2 of the sensors, hence why I'm renaming 
the columns in the database.


--
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/d68291c0-a0e2-3094-0af5-17bddbb1e075%40gmail.com.

Re: [weewx-user] wee_database

[Tom Keffer]

Don't know why you'd have to edit weewx.conf, but that looks reasonable.

On Tue, Nov 8, 2022 at 4:33 AM Mark Fraser  wrote:

> On 08/11/2022 12:30, Tom Keffer wrote:
> > For any mutating change you should not only stop weewx, but backup the
> > database.
> >
> > On Tue, Nov 8, 2022 at 1:24 AM Mark Fraser  > > wrote:
> >
> > I'm planning on renaming 2 columns in my weewx database using
> > wee_database --rename-column --to-name.
> >
> > Before I do this is it recommended that I stop weewx first?
>
> Yes, I was planning on doing a backup too. So I think my plan is:
> 1) Edit weewx.conf
> 2) Stop weewx
> 3) Back up database
> 4) Rename columns
> 5) Restart weewx
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/73df8efb-dd7f-ebce-dafe-877d548d216e%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEBeaMOQvBWa8Wpo7qJabubx-gXeAcPRyQnGJuA9AMY9rA%40mail.gmail.com.

Re: [weewx-user] wee_database

[Mark Fraser]

On 08/11/2022 12:30, Tom Keffer wrote:
For any mutating change you should not only stop weewx, but backup the 
database.


On Tue, Nov 8, 2022 at 1:24 AM Mark Fraser > wrote:


I'm planning on renaming 2 columns in my weewx database using
wee_database --rename-column --to-name.

Before I do this is it recommended that I stop weewx first?


Yes, I was planning on doing a backup too. So I think my plan is:
1) Edit weewx.conf
2) Stop weewx
3) Back up database
4) Rename columns
5) Restart weewx


--
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/73df8efb-dd7f-ebce-dafe-877d548d216e%40gmail.com.

Re: [weewx-user] wee_database

[Tom Keffer]

For any mutating change you should not only stop weewx, but backup the
database.

On Tue, Nov 8, 2022 at 1:24 AM Mark Fraser  wrote:

> I'm planning on renaming 2 columns in my weewx database using
> wee_database --rename-column --to-name.
>
> Before I do this is it recommended that I stop weewx first?
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/e9f83d1b-ae07-f160-8800-74a1c84cab00%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zECh4OfWYzE4sFxaNtax5jrr0T%3DQc-Jjq8Zt714abQDE7w%40mail.gmail.com.

[weewx-user] wee_database

[Mark Fraser]

I'm planning on renaming 2 columns in my weewx database using 
wee_database --rename-column --to-name.


Before I do this is it recommended that I stop weewx first?

--
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/e9f83d1b-ae07-f160-8800-74a1c84cab00%40gmail.com.

Re: [weewx-user] 'wee_database --calc-missing' long to execute

[graha...@gmail.com]

i reduced the --calc-missing time from 7 *hours* to 7 *minutes* by such a 
simple trick that i kick myself for not seeing it earlier - i moved the 
database to ramdisk and symlinked it under archive, ran wee_database, then 
moved the database back. chalk this one under 'handy tips'

On Thursday, 10 March 2022 at 11:09:25 am UTC+11 graha...@gmail.com wrote:

> for gw1000 dataset, i had been using top and iotop previously to ascertain 
> that cpu, memory and i/o usage were extremely low, and ps-efl showed it was 
> spending its time waiting on interrupt. i would usually just conclude it 
> was slow disk and spending all its time waiting on i/o completion *except* 
> this is only for the smaller gw1000 dataset not the larger vp2 dataset. it 
> is something to do with the different nature of the data (perhaps something 
> as simple as different missing data being calc’ed)
>
> to give an idea of the magnitude of the difference, using built-in shell 
> time to run command:
>
>
> *recs* *real /sec* *user /sec* *sys /sec* *Idle /%* 
> vp2 --rebuild-daily 505,336 165 148 2 9 
> vp2 --calc-missing 505,336 571 525 18 5 
> gw1000 --rebuild-daily 162,882 86 81 1 5 
> gw1000 --calc-missing 162,882 23,758 301 13 99
>
[snip] 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/f339d818-08b1-4e8b-ae4b-3e9ea1eaf9cdn%40googlegroups.com.

Re: [weewx-user] 'wee_database --calc-missing' long to execute

[Graham Eddy]

for gw1000 dataset, i had been using top and iotop previously to ascertain that 
cpu, memory and i/o usage were extremely low, and ps-efl showed it was spending 
its time waiting on interrupt. i would usually just conclude it was slow disk 
and spending all its time waiting on i/o completion *except* this is only for 
the smaller gw1000 dataset not the larger vp2 dataset. it is something to do 
with the different nature of the data (perhaps something as simple as different 
missing data being calc’ed)

to give an idea of the magnitude of the difference, using built-in shell time 
to run command:


recsreal /sec   user /sec   sys /secIdle /%
vp2 --rebuild-daily 505,336 165 148 2   9
vp2 --calc-missing  505,336 571 525 18  5
gw1000  --rebuild-daily 162,882 86  81  1   5
gw1000  --calc-missing  162,882 23,758  301 13  99

as it stands right now, for migration of production to split environment, i 
will have to
  * take a database snapshot and build the equivalent temp gw1000.sdb before 
migrating
  * do —calc-missing offline on the temp gw1000.sdb (7 hours !!)
  * dump the were-missing values in temp gw1000.sdb into a file
  * when dumped data avail, stop production system, split the databases, load 
the dumped were-missing values into gw1000.sdb
  * run —calc-missing on the interval only between dump and now ← hopefully not 
long, gw1000 data being lost!
  * start new production system on split databases

does anyone have insight into the origin of the wait-for-interrupt plaguing my 
gw1000 dataset migration? perhaps some wxxtypes in do_calculations() have a 
realtime delay built in? perhaps the yield in genBatchRecords() is not context 
switching to another thread effectively (internal python issue)? has anyone 
seen such behaviour elsewhere?

cheers
⊣ Graham Eddy ⊢

> On 9 Mar 2022, at 4:16 pm, vince  wrote:
> 
> Well I'd still try either splitting it up into pieces, or running it and 
> measuring its resource usage in another shell.   If it's not out of ram and 
> you're not pegging the CPU than waiting for i/o is the only thing left I'd 
> guess.  Run htop or the like to see.  
> 
> On Tuesday, March 8, 2022 at 7:03:37 PM UTC-8 graha...@gmail.com wrote:
> it’s an 8MB RPi 4B and RAM is abundant, though using μSD card for filesys.
> the interesting thing is that there are two datasets, one large and one 
> small, and the large one is quick but the small one is orders of magnitude 
> slower. the larger dataset (vp2-originated) gobbles >95% cpu but the smaller 
> dataset (gw1000-originated) <0.3%. it must be something to do with the nature 
> of the data, not the quantity.
> ⊣ Graham Eddy ⊢

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/28D3A6B3-A18E-4EDB-AC17-ABD2DAE1F29F%40gmail.com.

Re: [weewx-user] 'wee_database --calc-missing' long to execute

[vince]

Well I'd still try either splitting it up into pieces, or running it and 
measuring its resource usage in another shell.   If it's not out of ram and 
you're not pegging the CPU than waiting for i/o is the only thing left I'd 
guess.  Run htop or the like to see.  

On Tuesday, March 8, 2022 at 7:03:37 PM UTC-8 graha...@gmail.com wrote:

> it’s an 8MB RPi 4B and RAM is abundant, though using μSD card for filesys.
> the interesting thing is that there are two datasets, one large and one 
> small, and the large one is quick but the small one is orders of magnitude 
> slower. the larger dataset (vp2-originated) gobbles >95% cpu but the 
> smaller dataset (gw1000-originated) <0.3%. it must be something to do with 
> the nature of the data, not the quantity.
> ⊣ *Graham Eddy* ⊢
>
> On 9 Mar 2022, at 12:11 pm, vince  wrote:
>
> Need some info on which computer(s) you are doing this on.   Normally when 
> I see long run times and low cpu, it's out of RAM or waiting for i/o.  What 
> does something like 'htop' show ?
>
> Have you tried to split your input data into smaller pieces ?  Perhaps try 
> a year as a test and see if that's still massively slow ?
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/5e57077d-8e2e-4529-92f9-d70581d3f6cbn%40googlegroups.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/1bb04bdc-77f0-4a8f-8890-6eb18c9eed79n%40googlegroups.com.

Re: [weewx-user] 'wee_database --calc-missing' long to execute

[Graham Eddy]

it’s an 8MB RPi 4B and RAM is abundant, though using μSD card for filesys.
the interesting thing is that there are two datasets, one large and one small, 
and the large one is quick but the small one is orders of magnitude slower. the 
larger dataset (vp2-originated) gobbles >95% cpu but the smaller dataset 
(gw1000-originated) <0.3%. it must be something to do with the nature of the 
data, not the quantity.
⊣ Graham Eddy ⊢

> On 9 Mar 2022, at 12:11 pm, vince  wrote:
> 
> Need some info on which computer(s) you are doing this on.   Normally when I 
> see long run times and low cpu, it's out of RAM or waiting for i/o.  What 
> does something like 'htop' show ?
> 
> Have you tried to split your input data into smaller pieces ?  Perhaps try a 
> year as a test and see if that's still massively slow ?
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/5e57077d-8e2e-4529-92f9-d70581d3f6cbn%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/76C77D79-5E2D-44C2-8203-F75261F6F7E1%40gmail.com.

[weewx-user] 'wee_database --calc-missing' long to execute

[Graham Eddy]

presently my vp2 and gw1000 data are merged in one database, and i am splitting 
them.
i took the weewx.sdb and placed it in new weewx 4.7.0 environment.
i ran weewx a couple of times configured to create empty vp2.sdb and gw1000.sdb 
(with vp2.conf and gw1000.conf respectively pointing to their own schemas and 
dbs).
i then copied the relevant observations (own program) from weewx.sdb columns 
into vp2.sdb and gw1000.sdb then ran ‘wee_database —conf=__.conf —calc-missing’ 
to fill in the respective derived values.
the vp2 (with 16 years of data) ran in about 10 mins, as expected.
the gw1000 (with 2 years of data) has been running for 12 hours so far, at 
about 7000 recs/hour, and at present rate will take about 2 days in total, 
which ain’t gonna fly when i try actual migration of production environment.
note: when processing the vp2 data, cpu is typically 98%, but with gw1000 data 
wee_database dawdles on 0.3% and usually lower.

before i dive into detailed analysis on why —calc-missing is orders of 
magnitude slower on the gw1000 data than the vp2 data, can someone provide me 
some pointers, guidance where to look?

(if this doesn’t work, i’ll try again by moving columns around copies of 
weewx.sdb rather than programmatically splitting the data, and see if i get 
—calc-missing issue again…)
cheers
⊣ Graham Eddy ⊢

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CD5AE3A9-7A5B-4A1E-8D63-08B537D4C259%40gmail.com.

[weewx-user] wee_database transfer - out of memory

[StefMel]

I am trying to convert my MYSQL DB to SQLite on my RaspberryPi with the 
wee_database 
--transfer command.
The dry run was just fine, but when running in real the process fails after 
some time with a Out of Memory exception.
>From 572643 records only 10034 were transferred.
I guess the memory of my Raspi is a bit weak.
Do I have another possibility? 

pi@weatherpi:~ $  wee_database --transfer --dest-binding=dest_binding 
--dry-run
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_mysql'
Transfer 572643 records from source database 'weewx' to destination 
database 'weewx.sdb'.
Dry run, nothing done.
pi@weatherpi:~ $ wee_database --transfer --dest-binding=dest_binding
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_mysql'
Transfer 572644 records from source database 'weewx' to destination 
database 'weewx.sdb' (y/n)? y
transferring, this may take a while Out of memory (Needed 881712 bytes)
complete
10034 records transferred from source database 'weewx' to destination 
database 'weewx.sdb'.


-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/7803ed66-0c71-4d36-87a1-552833595851n%40googlegroups.com.

Re: [weewx-user] wee_database sdb --create success but --check fails

[S R]

i also get 
./bin/wee_database --update
Using configuration file /usr/local/share/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_sqlite'
The update process does not affect archive data, but does alter the 
database.
Continue (y/n)? y
Preparing Interval Weighting Fix...
Daily summary tables are at version 3.0
Interval weighting fix is not required.
Preparing Maximum windSpeed Fix...
Traceback (most recent call last):
  File "./bin/wee_database", line 1020, in 
main()
  File "./bin/wee_database", line 162, in main
update(config_dict, db_binding, options)
  File "./bin/wee_database", line 639, in update
_fix_wind(config_dict, db_binding, options)
  File "./bin/wee_database", line 742, in _fix_wind
wind_obj.run()
  File "/usr/local/share/weewx/bin/weecfg/database.py"


On Thursday, January 21, 2021 at 11:15:52 PM UTC+1 tke...@gmail.com wrote:

> Not exactly a big problem, but fixed in commit 0f67337 
> 
> .
>
>
> On Thu, Jan 21, 2021 at 12:53 PM John Kline  wrote:
>
>> Tom, the lines starting at line number 846 in wee_database need to be 
>> skipped if records == 0.  Not sure how you want to do that and whether you 
>> want a summary that says “no records in database”.  It will be easier for 
>> you to fix than to go back and forth with a PR.
>>
>> On Jan 21, 2021, at 12:42 PM, John Kline  wrote:
>>
>> 
>> I had a look.  This does not mean the database is bad.  The code needs a 
>> small fix as it’s using a variable that isn’t set if there are no records 
>> in the database.
>>
>> On Jan 21, 2021, at 12:37 PM, S R  wrote:
>>
>> 
>> if i run wee_database -- check on the sdb that was installed during the 
>> installed, it passes. 
>> if i rename the original sdb and then run wee_database --create, the file 
>> creates without error, but if i then immediately run the --check i get the 
>> below error 
>>
>> sudo wee_database --check 
>> Using configuration file /etc/weewx/weewx.conf 
>> Using database binding 'wx_binding', which is bound to database 'weewx' 
>> Checking daily summary tables version... 
>> Daily summary tables are at version 3.0 
>> Interval Weighting Fix is not required. 
>> Daily summary tables version check completed in 0.01 seconds. 
>> Preparing Null String Fix, this may take a while... 
>> Traceback (most recent call last): 
>>  File "/usr/share/weewx/wee_database", line 1020, in  
>>main() 
>>  File "/usr/share/weewx/wee_database", line 159, in main 
>>check(config_dict, db_binding, options) 
>>  File "/usr/share/weewx/wee_database", line 582, in check 
>>check_strings(config_dict, db_binding, options, fix=False) 
>>  File "/usr/share/weewx/wee_database", line 848, in check_strings 
>>% (records, timestamp_to_string(record[0])), end=' ') 
>> UnboundLocalError: local variable 'record' referenced before assignment
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/50265d72-27cc-4131-b4d7-d62bca674fe7n%40googlegroups.com
>>  
>> 
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/07907469-7084-472A-9D1E-0F81685FD244%40johnkline.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/cc4b305b-3875-4e55-87ad-e48a496e98d5n%40googlegroups.com.

Re: [weewx-user] wee_database sdb --create success but --check fails

[Tom Keffer]

Not exactly a big problem, but fixed in commit 0f67337

.


On Thu, Jan 21, 2021 at 12:53 PM John Kline  wrote:

> Tom, the lines starting at line number 846 in wee_database need to be
> skipped if records == 0.  Not sure how you want to do that and whether you
> want a summary that says “no records in database”.  It will be easier for
> you to fix than to go back and forth with a PR.
>
> On Jan 21, 2021, at 12:42 PM, John Kline  wrote:
>
> 
> I had a look.  This does not mean the database is bad.  The code needs a
> small fix as it’s using a variable that isn’t set if there are no records
> in the database.
>
> On Jan 21, 2021, at 12:37 PM, S R  wrote:
>
> 
> if i run wee_database -- check on the sdb that was installed during the
> installed, it passes.
> if i rename the original sdb and then run wee_database --create, the file
> creates without error, but if i then immediately run the --check i get the
> below error
>
> sudo wee_database --check
> Using configuration file /etc/weewx/weewx.conf
> Using database binding 'wx_binding', which is bound to database 'weewx'
> Checking daily summary tables version...
> Daily summary tables are at version 3.0
> Interval Weighting Fix is not required.
> Daily summary tables version check completed in 0.01 seconds.
> Preparing Null String Fix, this may take a while...
> Traceback (most recent call last):
>  File "/usr/share/weewx/wee_database", line 1020, in 
>main()
>  File "/usr/share/weewx/wee_database", line 159, in main
>check(config_dict, db_binding, options)
>  File "/usr/share/weewx/wee_database", line 582, in check
>check_strings(config_dict, db_binding, options, fix=False)
>  File "/usr/share/weewx/wee_database", line 848, in check_strings
>% (records, timestamp_to_string(record[0])), end=' ')
> UnboundLocalError: local variable 'record' referenced before assignment
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/50265d72-27cc-4131-b4d7-d62bca674fe7n%40googlegroups.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/07907469-7084-472A-9D1E-0F81685FD244%40johnkline.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEB7prPhG4xyEgmT7h2QSx7bzLruyCzyQ5q24A9HfrMfig%40mail.gmail.com.

Re: [weewx-user] wee_database sdb --create success but --check fails

[John Kline]

Tom, the lines starting at line number 846 in wee_database need to be skipped 
if records == 0.  Not sure how you want to do that and whether you want a 
summary that says “no records in database”.  It will be easier for you to fix 
than to go back and forth with a PR.

> On Jan 21, 2021, at 12:42 PM, John Kline  wrote:
> 
> 
> I had a look.  This does not mean the database is bad.  The code needs a 
> small fix as it’s using a variable that isn’t set if there are no records in 
> the database.
> 
>>> On Jan 21, 2021, at 12:37 PM, S R  wrote:
>>> 
>> 
>> if i run wee_database -- check on the sdb that was installed during the 
>> installed, it passes. 
>> if i rename the original sdb and then run wee_database --create, the file 
>> creates without error, but if i then immediately run the --check i get the 
>> below error 
>> 
>> sudo wee_database --check 
>> Using configuration file /etc/weewx/weewx.conf 
>> Using database binding 'wx_binding', which is bound to database 'weewx' 
>> Checking daily summary tables version... 
>> Daily summary tables are at version 3.0 
>> Interval Weighting Fix is not required. 
>> Daily summary tables version check completed in 0.01 seconds. 
>> Preparing Null String Fix, this may take a while... 
>> Traceback (most recent call last): 
>>  File "/usr/share/weewx/wee_database", line 1020, in  
>>main() 
>>  File "/usr/share/weewx/wee_database", line 159, in main 
>>check(config_dict, db_binding, options) 
>>  File "/usr/share/weewx/wee_database", line 582, in check 
>>check_strings(config_dict, db_binding, options, fix=False) 
>>  File "/usr/share/weewx/wee_database", line 848, in check_strings 
>>% (records, timestamp_to_string(record[0])), end=' ') 
>> UnboundLocalError: local variable 'record' referenced before assignment
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/50265d72-27cc-4131-b4d7-d62bca674fe7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/07907469-7084-472A-9D1E-0F81685FD244%40johnkline.com.

Re: [weewx-user] wee_database sdb --create success but --check fails

[John Kline]

I had a look.  This does not mean the database is bad.  The code needs a small 
fix as it’s using a variable that isn’t set if there are no records in the 
database.

> On Jan 21, 2021, at 12:37 PM, S R  wrote:
> 
> 
> if i run wee_database -- check on the sdb that was installed during the 
> installed, it passes. 
> if i rename the original sdb and then run wee_database --create, the file 
> creates without error, but if i then immediately run the --check i get the 
> below error 
> 
> sudo wee_database --check 
> Using configuration file /etc/weewx/weewx.conf 
> Using database binding 'wx_binding', which is bound to database 'weewx' 
> Checking daily summary tables version... 
> Daily summary tables are at version 3.0 
> Interval Weighting Fix is not required. 
> Daily summary tables version check completed in 0.01 seconds. 
> Preparing Null String Fix, this may take a while... 
> Traceback (most recent call last): 
>  File "/usr/share/weewx/wee_database", line 1020, in  
>main() 
>  File "/usr/share/weewx/wee_database", line 159, in main 
>check(config_dict, db_binding, options) 
>  File "/usr/share/weewx/wee_database", line 582, in check 
>check_strings(config_dict, db_binding, options, fix=False) 
>  File "/usr/share/weewx/wee_database", line 848, in check_strings 
>% (records, timestamp_to_string(record[0])), end=' ') 
> UnboundLocalError: local variable 'record' referenced before assignment
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/50265d72-27cc-4131-b4d7-d62bca674fe7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/D1BBE4BE-5798-49AC-B937-EC127906ACB3%40johnkline.com.

[weewx-user] wee_database sdb --create success but --check fails

[S R]

if i run wee_database -- check on the sdb that was installed during the 
installed, it passes. 
if i rename the original sdb and then run wee_database --create, the file 
creates without error, but if i then immediately run the --check i get the 
below error 

sudo wee_database --check 
Using configuration file /etc/weewx/weewx.conf 
Using database binding 'wx_binding', which is bound to database 'weewx' 
Checking daily summary tables version... 
Daily summary tables are at version 3.0 
Interval Weighting Fix is not required. 
Daily summary tables version check completed in 0.01 seconds. 
Preparing Null String Fix, this may take a while... 
Traceback (most recent call last): 
 File "/usr/share/weewx/wee_database", line 1020, in  
   main() 
 File "/usr/share/weewx/wee_database", line 159, in main 
   check(config_dict, db_binding, options) 
 File "/usr/share/weewx/wee_database", line 582, in check 
   check_strings(config_dict, db_binding, options, fix=False) 
 File "/usr/share/weewx/wee_database", line 848, in check_strings 
   % (records, timestamp_to_string(record[0])), end=' ') 
UnboundLocalError: local variable 'record' referenced before assignment


-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/50265d72-27cc-4131-b4d7-d62bca674fe7n%40googlegroups.com.

Re: [weewx-user] wee_database: ImportError

[Thomas Keffer]

For some reason, the Python MySQL client library has not been installed.
Try this:

sudo apt-get install mysql-client
sudo apt-get install python-mysqldb

-tk

On Tue, Feb 25, 2020 at 6:11 AM Eric Gammeter  wrote:

>
> Goal:
> Wanting to switch from sqlite to MySql database.
>
> I am consulting the github /weewx/wiki "Usingwee_database".
>
> The --dry-run doesn't show any error.
>
>
> pi@raspberrypi:~ $ wee_database --transfer --dest-binding=dest_binding
> --dry-run
> Using configuration file /etc/weewx/weewx.conf
> Using database binding 'wx_binding', which is bound to database
> 'archive_sqlite'
> Transfer 9177 records from source database 'weewx.sdb' to destination
> database 'weewx'.
> Dry run, nothing done.
> pi@raspberrypi:~ $ wee_database --transfer --dest-binding=dest_binding
> Using configuration file /etc/weewx/weewx.conf
> Using database binding 'wx_binding', which is bound to database
> 'archive_sqlite'
> Transfer 9177 records from source database 'weewx.sdb' to destination
> database 'weewx' (y/n)? y
> Error accessing destination database 'weewx'.
> Nothing done. Aborting.
> Traceback (most recent call last):
>   File "/usr/bin/wee_database", line 794, in 
> main()
>   File "/usr/bin/wee_database", line 159, in main
> transferDatabase(config_dict, db_binding, options)
>   File "/usr/bin/wee_database", line 410, in transferDatabase
> schema=dest_manager_dict['schema']) as dest_manager:
>   File "/usr/share/weewx/weewx/manager.py", line 128, in open_with_create
> connection = weedb.connect(database_dict)
>   File "/usr/share/weewx/weedb/__init__.py", line 86, in connect
> __import__(db_dict['driver'])
>   File "/usr/share/weewx/weedb/mysql.py", line 10, in 
> import MySQLdb
> ImportError: No module named MySQLdb
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/86fc5cde-69a0-4ab3-aa5c-f2c7f0c1559a%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zECMWNGSDs5mE1kzxQg45CS_fLWErR--75m5xinf0DAj7A%40mail.gmail.com.

[weewx-user] wee_database: ImportError

[Eric Gammeter]

Goal:
Wanting to switch from sqlite to MySql database. 

I am consulting the github /weewx/wiki "Usingwee_database".

The --dry-run doesn't show any error.  


pi@raspberrypi:~ $ wee_database --transfer --dest-binding=dest_binding 
--dry-run
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_sqlite'
Transfer 9177 records from source database 'weewx.sdb' to destination 
database 'weewx'.
Dry run, nothing done.
pi@raspberrypi:~ $ wee_database --transfer --dest-binding=dest_binding
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_sqlite'
Transfer 9177 records from source database 'weewx.sdb' to destination 
database 'weewx' (y/n)? y
Error accessing destination database 'weewx'.
Nothing done. Aborting.
Traceback (most recent call last):
  File "/usr/bin/wee_database", line 794, in 
main()
  File "/usr/bin/wee_database", line 159, in main
transferDatabase(config_dict, db_binding, options)
  File "/usr/bin/wee_database", line 410, in transferDatabase
schema=dest_manager_dict['schema']) as dest_manager:
  File "/usr/share/weewx/weewx/manager.py", line 128, in open_with_create
connection = weedb.connect(database_dict)
  File "/usr/share/weewx/weedb/__init__.py", line 86, in connect
__import__(db_dict['driver'])
  File "/usr/share/weewx/weedb/mysql.py", line 10, in 
import MySQLdb
ImportError: No module named MySQLdb

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/86fc5cde-69a0-4ab3-aa5c-f2c7f0c1559a%40googlegroups.com.

[weewx-user] wee_database --check-strings give clean report but database has NULL values

[Greg from Oz]

Hi,
I am using mqysql as my weewx database.
I run the wee_database --check-strings and it comes back clean but if I run 
mysql command to look for NULL I get a lot of results.
Anyone know why this would be the case?

Thanks

wee_database --check-strings
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_mysql'
Preparing Null String Check, this may take awhile...
Checking record: 488186; Timestamp: 2018-03-12 10:05:00 AEDT (1520809500)
No null strings found.
Completed Null String Check in 28.89 seconds.


Yet, if I run the command below it shows I do have NULL values.


MariaDB [weewx]> select dateTime, rain , rainRate from archive where rain 
IS NULL;
++--+--+
| dateTime   | rain | rainRate |
++--+--+
| 1501541984 | NULL |0 |
| 1501546184 | NULL |0 |
| 1501563583 | NULL |0 |
| 1501564184 | NULL |0 |
| 1501564490 | NULL |0 |
| 1501585184 | NULL |0 |
| 1501590884 | NULL |0 |
| 1501634684 | NULL |0 |
| 1501635284 | NULL |0 |
| 1501636784 | NULL |0 |
| 1501656884 | NULL |0 |
| 1501659584 | NULL |0 |
| 1501675645 | NULL |0 |

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] wee_database backfill only a range of dates

[Constantine Samaklis]

I am looking for a way to backfill a range of dates into my summary tables. 
When I look at the weewx website, the documentation states that you can do 
this with wee_database but it seems that the latest version of the utility 
only allows for a backfill where data is not present already in the 
aggregation tables.

I need to correct only a couple of days of bad data in my archive_day_rain 
table. 

Is there a way to do this currently with wee_database or do I need to get 
an older version of the script? (would that even be available or safe?)

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] wee_database --update fail

[Thomas Keffer]

Daily summaries without corresponding archive records is definitely an
undefined situation, so wee_database probably did the right thing by
raising an exception.

But, I'd like to know how it came to be. Did you (Eelco) delete some rows
in the archive database?

-tk



On Fri, Mar 17, 2017 at 2:27 AM, gjr80  wrote:

> Hi,
>
> Put simply --update will weight a number of daily summary fields according
> to the interval used in the archive. Judging by the error trace provided
> you appear to have daily summary records for days that have no records in
> your archive. I think you have 2 options:
>
> - The quick fix is to run wee_database with --drop-daily followed by
> --rebuild-daily to drop your daily summaries and then rebuild them from
> your archive. The rebuild should fix the situation I indicated above and
> you will not need to --update, the rebuild takes care of that. This
> approach will result in some loss of granularity of the daily summary data
> (eg max and min times) but otherwise it's non-destructive.
>
> - The alternative is to wait for Tom, he may have further insight, or he
> may say do a --drop-daily followed by a --rebuild! Of course if your
> station is offline you may be missing data if your station does not have
> memory.
>
> I would also be interested to know if you did a wee_database --update
> --dry-run beforehand, would like to know if the dry run missed the error.
>
> Gary
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] wee_database --update fail

[gjr80]

Hi,

Put simply --update will weight a number of daily summary fields according to 
the interval used in the archive. Judging by the error trace provided you 
appear to have daily summary records for days that have no records in your 
archive. I think you have 2 options:

- The quick fix is to run wee_database with --drop-daily followed by 
--rebuild-daily to drop your daily summaries and then rebuild them from your 
archive. The rebuild should fix the situation I indicated above and you will 
not need to --update, the rebuild takes care of that. This approach will result 
in some loss of granularity of the daily summary data (eg max and min times) 
but otherwise it's non-destructive.

- The alternative is to wait for Tom, he may have further insight, or he may 
say do a --drop-daily followed by a --rebuild! Of course if your station is 
offline you may be missing data if your station does not have memory.

I would also be interested to know if you did a wee_database --update --dry-run 
beforehand, would like to know if the dry run missed the error.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] wee_database --update fail

[Eelco F]

After upgrading to 3.7.0 I tried to update the database becuase of the 
max_wind issues.

Running wee_database --update results in.

Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_sqlite'
The update process does not affect archive data, but does alter the 
database.
Continue (y/n)? y
Preparing Interval Weighting Fix...
Daily summary tables are at version 1.0
Calculating interval weights...
Traceback (most recent call last):
  File "/usr/bin/wee_database", line 793, in 
main()
  File "/usr/bin/wee_database", line 165, in main
update(config_dict, db_binding, options)
  File "/usr/bin/wee_database", line 552, in update
weight_obj.run()
  File "/usr/share/weewx/weecfg/database.py", line 443, in run
self.do_fix(_next_day_to_patch_ts)
  File "/usr/share/weewx/weecfg/database.py", line 501, in do_fix
_weight = self.get_interval(_day_span) * 60
  File "/usr/share/weewx/weecfg/database.py", line 596, in get_interval
return _row[0]
TypeError: 'NoneType' object has no attribute '__getitem__'

restarting weewx doesn't work becuase the database update isn't finished.

could you help me out?

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] wee_database --drop-daily followed by --backfill-daily producing "Unit system mismatch"

[Christopher McAvaney]

G'day,

Was following the instructions to clean up a bad rain reading 
from https://github.com/weewx/weewx/wiki/Cleaning-up-old-'bad'-data

But, when I execute the --backfill-daily option I am getting a "Unit system 
mismatch" error.

$ wee_database --drop-daily
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_mysql'
Proceeding will delete all your daily summaries from database 'weewx'
Are you sure you want to proceed (y/n)? y
Dropping daily summary tables from 'weewx' ...
Dropped daily summary tables from database 'weewx'

$ wee_database --backfill-daily --dry-run
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 
'archive_mysql'
Backfilling daily summaries in database 'weewx' ...
Traceback (most recent call last):te: 2016-10-06 16:26:50 AEDT (1475731610)
  File "/usr/bin/wee_database", line 444, in 
main()
  File "/usr/bin/wee_database", line 153, in main
backfillDaily(config_dict, db_binding, int(options.trans_days))
  File "/usr/bin/wee_database", line 217, in backfillDaily
nrecs, ndays = dbmanager.backfill_day_summary(trans_days=trans_days)
  File "/usr/share/weewx/weewx/manager.py", line 1374, in 
backfill_day_summary
_day_accum.addRecord(_rec)
  File "/usr/share/weewx/weewx/accum.py", line 240, in addRecord
func(self, record, obs_type, add_hilo)
  File "/usr/share/weewx/weewx/accum.py", line 335, in check_units
self._check_units(record['usUnits'])
  File "/usr/share/weewx/weewx/accum.py", line 348, in _check_units
raise ValueError("Unit system mismatch %d v. %d" % (self.unit_system, 
new_unit_system))
ValueError: Unit system mismatch 1 v. 0


In looking at the data for that timestamp, everything looks o.k.:
"dateTime" "usUnits" "interval" "barometer" "pressure" "altimeter" "inTemp" 
"outTemp" "inHumidity" "outHumidity" "windSpeed" "windDir" "windGust" 
"windGustDir" "rainRate" "rain" "dewpoint" "windchill" "heatindex" "ET" 
"radiation" "UV" "extraTemp1" "extraTemp2" "extraTemp3" "soilTemp1" 
"soilTemp2" "soilTemp3" "soilTemp4" "leafTemp1" "leafTemp2" "extraHumid1" 
"extraHumid2" "soilMoist1" "soilMoist2" "soilMoist3" "soilMoist4" "leafWet1" 
"leafWet2" "rxCheckPercent" "txBatteryStatus" "consBatteryVoltage" "hail" 
"hailRate" "heatingTemp" "heatingVoltage" "supplyVoltage" "referenceVoltage" 
"windBatteryStatus" "rainBatteryStatus" "outTempBatteryStatus" 
"inTempBatteryStatus"
"1475731610" "1" "5" "29.43787700257" "29.3473124630833" "29.432883539847" 
"82.4" "79.34" "45" "36" "9.17143879392" "315" "12.07945597248" "315" "0" 
"0" "50.0428531173988" "79.34" "79.34" \N \N \N \N \N \N \N \N \N \N \N \N 
\N \N \N \N \N \N \N \N "100" \N \N \N \N \N \N \N \N \N \N "0" \N

I don't really understand why this error is thrown as the backfill works 
for 10's of thousands of records until that particular date.

Does anybody know why it would be throwing this unit error?

Thanks in advance,
Christopher

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.