I guess your month and year data doesn't use the daily records instead of the interval ones?
At least I know why it is so slow then. On Mon, Aug 27, 2018 at 11:47 AM Pat <p...@obrienphoto.net> wrote: > Ah, that'll do it. Explains why your database is so large and why it takes > a while to generate the graph json files. That's a lot of records to > process when generating the graphs. > > You may want to consider going to 300 seconds for archive which is the > aggregation > of the loops during those 300 seconds > <http://www.weewx.com/docs/customizing.htm#_________LOOP_packets_vs._archive_records_______>. > Up to you though, it's your data. > > Changing it now won't cure the size and slowness issues though. Suppose > you could run a database query to drop any dateTime from archive that isn't > on a 5 minute interval (0, 5, 10, 15, etc). > > Don't know if Tom or Matthew have another way. > > > > On Monday, August 27, 2018 at 12:40:15 PM UTC-4, David Hathaway wrote: >> >> [StdArchive] >> >> # If the station hardware supports data logging then the archive >> interval >> # will be downloaded from the station. Otherwise, specify it (in >> seconds). >> archive_interval = 60 >> >> On Mon, Aug 27, 2018 at 11:36 AM Pat <p...@obrienphoto.net> wrote: >> >>> The year graph don't total like the others, just shows you which days >>> had rain. That being said your year.json is HUGE at 8mb (by contrast my >>> year.json is 231kb). I wonder why yours is so big? This is why your report >>> generation is so slow. Month and Year look rather large. >>> >>> How often are you doing archive? weewx.conf StdArchive.archive_interval >>> >>> On Monday, August 27, 2018 at 12:29:26 PM UTC-4, David Hathaway wrote: >>>> >>>> Pat, >>>> >>>> If you look at my site and graphs, the year rain is very odd. >>>> >>>> http://dave77459.mynetgear.com/weewx/belchertown/graphs/ >>>> >>>> On the yearly graph, rain is far less than 0.02. Switch to month, and >>>> you can see it exceeds 1/2" a few days. I also miss the total rainfall >>>> line on the year graph. >>>> >>>> Dave >>>> >>>> >>>> On Mon, Aug 27, 2018 at 11:24 AM David Hathaway <hathaw...@gmail.com> >>>> wrote: >>>> >>>>> The .tar.gz backup of the database is 71MB. That is to say, it is >>>>> big. It is possible that it needs tuning, but running a query to create a >>>>> cumulative rain table for the past year takes less than a second. When I >>>>> get home, I can investigate further. Maybe there is a deadlock or >>>>> something. >>>>> >>>>> Meanwhile, yay! It runs and looks nice. >>>>> http://dave77459.mynetgear.com/weewx/belchertown/ >>>>> >>>>> TO-DO: >>>>> >>>>> - Figure out the wind records >>>>> - Figure out the server load issue >>>>> - NOAA data when 0.5 comes out >>>>> - Personalize the headings, etc >>>>> - Figure out how to get my smokeping graph to show >>>>> - Figure out how to get pwrstat, which indirectly tracks server >>>>> loads >>>>> - Update my URL >>>>> >>>>> Thanks for the help! Many hours of fun ahead. :-) >>>>> >>>>> >>>>> >>>>> On Mon, Aug 27, 2018 at 11:06 AM Pat <p...@obrienphoto.net> wrote: >>>>> >>>>>> > Do you mean this section of weewx.conf? >>>>>> >>>>>> Yep >>>>>> >>>>>> > Will that mess up StandardReport? >>>>>> >>>>>> Don't believe so. It will change your URL that is displayed on >>>>>> weewx.com if you have register_this_station = true >>>>>> >>>>>> > I am on a quad core Xeon CPU at 3.30Ghz, 16GB RAM, and an SSD main >>>>>> drive. >>>>>> >>>>>> So basically a server. Hmm.... the highcharts do a lot of queries so >>>>>> an extended generation time is expected but not sure why it's 2 minutes. >>>>>> Maybe your tables needs to be optimized? >>>>>> >>>>>> There are ways to tune MySQL which may be something to check out too? >>>>>> >>>>>> >>>>>> >>>>>> On Monday, August 27, 2018 at 11:58:28 AM UTC-4, David Hathaway wrote: >>>>>>> >>>>>>> Do you mean this section of weewx.conf? >>>>>>> >>>>>>> # If you have a website, you may specify an URL >>>>>>> station_url = http://dave77459.mynetgear.com/weewx/ >>>>>>> >>>>>>> Will that mess up StandardReport? Heck, I'll try it and see. Easy >>>>>>> to change back. >>>>>>> >>>>>>> I am on a quad core Xeon CPU at 3.30Ghz, 16GB RAM, and an SSD main >>>>>>> drive. Hardly anything takes time on this system. >>>>>>> >>>>>>> I'll give --rebuild-daily a try. I have a backup from yesterday, >>>>>>> but I'll make another first. >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 27, 2018 at 10:31 AM Pat <p...@obrienphoto.net> wrote: >>>>>>> >>>>>>>> This way is fine. Problems and resolutions can help everyone if >>>>>>>> it's in the public forum. >>>>>>>> >>>>>>>> I think I just resolved the error. In my highcharts Day function >>>>>>>> I'm defining an end point, yet still using the timespan (which causes >>>>>>>> the >>>>>>>> problem). Moving to my defined end point may have fixed it. Doing some >>>>>>>> testing with it now across a few lab systems to make sure. Then I'll >>>>>>>> issue >>>>>>>> 0.5 and we can give that a test. >>>>>>>> >>>>>>>> To answer some of your questions: >>>>>>>> >>>>>>>> The link to styles.css file is wrong. To fix this, change your >>>>>>>> weewx.conf's URL to " >>>>>>>> http://dave77459.mynetgear.com/weewx/belchertown"; and re-run the >>>>>>>> report. >>>>>>>> >>>>>>>> The 2 minute generation is for the highcharts section and it does >>>>>>>> seem high. Are you on a Pi? I have 3 years of data in my database and >>>>>>>> it >>>>>>>> takes 15 seconds to generate on a PC. A pi would be a bit slower. >>>>>>>> >>>>>>>> No column error with wind? Hmm. Not too sure on this. Could be >>>>>>>> database related. You can try to run sudo wee_database >>>>>>>> --rebuild-daily to rebuild the daily values and see if that >>>>>>>> helps. Make sure you have a *working backup of your database* >>>>>>>> just in case you need to revert back to it. >>>>>>>> >>>>>>>> >>>>>>>> On Monday, August 27, 2018 at 11:25:07 AM UTC-4, David Hathaway >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I feel like a problem child at this point. Maybe I should email >>>>>>>>> you directly rather than via this group? No worries if you want to >>>>>>>>> continue this way though. >>>>>>>>> >>>>>>>>> Commenting out the NOAA reports got it to run...ish. >>>>>>>>> >>>>>>>>> >>>>>>>>> - I got a "NoColumnError" in the wind section. See log >>>>>>>>> below. I commented out the Wind Records section of the >>>>>>>>> records\index.html.tmpl to continue on. >>>>>>>>> - The skin is taking 2 minutes to be generated. Is this >>>>>>>>> expected? In contrast, StandardReport takes 0.75 seconds. >>>>>>>>> - The resulting page is broken as if it is missing all >>>>>>>>> content. See it here: >>>>>>>>> http://dave77459.mynetgear.com/weewx/belchertown/ >>>>>>>>> >>>>>>>>> >>>>>>>>> Aug 27 09:53:16 localhost weewx[22819]: cheetahgenerator: >>>>>>>>> Generated 14 files for report StandardReport in 1.18 seconds >>>>>>>>> Aug 27 09:53:17 localhost weewx[22819]: imagegenerator: Generated >>>>>>>>> 16 images for StandardReport in 1.54 seconds >>>>>>>>> Aug 27 09:54:14 localhost weewx[22819]: manager: Added record >>>>>>>>> 2018-08-27 09:54:00 CDT (1535381640) to database 'weewx' >>>>>>>>> Aug 27 09:54:14 localhost weewx[22819]: manager: Added record >>>>>>>>> 2018-08-27 09:54:00 CDT (1535381640) to daily summary in 'weewx' >>>>>>>>> Aug 27 09:54:14 localhost weewx[22819]: engine: Launch of report >>>>>>>>> thread aborted: existing report thread still running >>>>>>>>> Aug 27 09:54:15 localhost weewx[22819]: restx: Wunderground-PWS: >>>>>>>>> Published record 2018-08-27 09:54:00 CDT (1535381640) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: cheetahgenerator: Generate >>>>>>>>> failed with exception '<class 'weedb.NoColumnError'>' >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: cheetahgenerator: **** >>>>>>>>> Ignoring template /etc/weewx/skins/Belchertown/records/index.html.tmpl >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: cheetahgenerator: **** >>>>>>>>> Reason: (1054, "Unknown column 'wind' in 'where clause'") >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** Traceback (most >>>>>>>>> recent call last): >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** print >> _file, >>>>>>>>> compiled_template >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in >>>>>>>>> __str__ >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** rc = >>>>>>>>> getattr(self, mainMethName)() >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "_etc_weewx_skins_Belchertown_records_index_html_tmpl.py", line 1109, >>>>>>>>> in >>>>>>>>> respond >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "_etc_weewx_skins_Belchertown_records_index_html_tmpl.py", line 385, >>>>>>>>> in >>>>>>>>> __errorCatcher46 >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File "<string>", >>>>>>>>> line 1, in <module> >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__ >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** return >>>>>>>>> self._do_query(aggregate_type) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/tags.py", line 343, in _do_query >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** val=val, >>>>>>>>> **self.option_dict) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** return >>>>>>>>> weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, >>>>>>>>> aggregateType, **option_dict) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/manager.py", line 1249, in getAggregate >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** **option_dict) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/manager.py", line 458, in getAggregate >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** _row = >>>>>>>>> self.getSql(select_stmt % interpolate_dict) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weewx/manager.py", line 394, in getSql >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** >>>>>>>>> _cursor.execute(sql, sqlargs) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** File >>>>>>>>> "/usr/share/weewx/weedb/mysql.py", line 48, in guarded_fn >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** raise klass(e) >>>>>>>>> Aug 27 09:54:35 localhost weewx[22819]: **** NoColumnError: >>>>>>>>> (1054, "Unknown column 'wind' in 'where clause'") >>>>>>>>> Aug 27 09:54:54 localhost weewx[22819]: cheetahgenerator: >>>>>>>>> Generated 13 files for report Belchertown in 96.59 seconds >>>>>>>>> Aug 27 09:54:54 localhost weewx[22819]: copygenerator: copied 22 >>>>>>>>> files to /var/www/html/weewx/belchertown >>>>>>>>> >>>>>>>>> >>>>>>>>> On Monday, August 27, 2018 at 9:44:55 AM UTC-5, David Hathaway >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> OK, I will try that and see if that gets the rest going. >>>>>>>>>> >>>>>>>>>> On Mon, Aug 27, 2018 at 9:37 AM Pat <> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks. I'm aware of this error but haven't resolved it yet. >>>>>>>>>>> It's a problem with NOAA reports being generated. >>>>>>>>>>> >>>>>>>>>>> My thought is to move the NOAA reports to another skin, but I'm >>>>>>>>>>> afraid that'll be too much extra work for everyone who installs the >>>>>>>>>>> skin. >>>>>>>>>>> >>>>>>>>>>> If you want to stop it from happening, open skin.conf and remove >>>>>>>>>>> the NOAA_month and NOAA_year sections - which will get the skin >>>>>>>>>>> working but will not produce the NOAA reports. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Monday, August 27, 2018 at 10:22:23 AM UTC-4, David Hathaway >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I updated MySQL to stop throwing that error, updated to weewx v >>>>>>>>>>>> 3.8.2, and installed Belchertown 0.4. Now I get this error: >>>>>>>>>>>> >>>>>>>>>>>> Aug 27 09:17:16 localhost weewx[4131]: imagegenerator: >>>>>>>>>>>> Generated 16 images for StandardReport in 0.75 seconds >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: reportengine: Caught >>>>>>>>>>>> unrecoverable exception in generator >>>>>>>>>>>> weewx.cheetahgenerator.CheetahGenerator >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** start time >>>>>>>>>>>> (1535346000) is greater than stop time (1420092000) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** Traceback >>>>>>>>>>>> (most recent call last): >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/reportengine.py", line 239, in run >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** >>>>>>>>>>>> obj.start() >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/reportengine.py", line 273, in start >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** >>>>>>>>>>>> self.run() >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/cheetahgenerator.py", line 158, in run >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** ngen = >>>>>>>>>>>> self.generate(gen_dict[section_name], self.gen_ts) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** ngen >>>>>>>>>>>> += self.generate(section[subsection], gen_ts) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** ngen >>>>>>>>>>>> += self.generate(section[subsection], gen_ts) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/cheetahgenerator.py", line 320, in generate >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** >>>>>>>>>>>> default_binding) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weewx/cheetahgenerator.py", line 372, in >>>>>>>>>>>> _getSearchList >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** >>>>>>>>>>>> searchList += obj.get_extension_list(timespan, db_lookup) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/user/belchertown_highchartsSearchX.py", line 88, >>>>>>>>>>>> in >>>>>>>>>>>> get_extension_list >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** >>>>>>>>>>>> (time_start_vt, time_stop_vt, outTemp_vt) = >>>>>>>>>>>> db_lookup().getSqlVectors(TimeSpan(_start_ts, timespan.stop), >>>>>>>>>>>> 'outTemp') >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** File >>>>>>>>>>>> "/usr/share/weewx/weeutil/weeutil.py", line 315, in __new__ >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** raise >>>>>>>>>>>> ValueError("start time (%d) is greater than stop time (%d)" % >>>>>>>>>>>> (args[0], >>>>>>>>>>>> args[1])) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** >>>>>>>>>>>> ValueError: start time (1535346000) is greater than stop time >>>>>>>>>>>> (1420092000) >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: **** Generator >>>>>>>>>>>> terminated >>>>>>>>>>>> Aug 27 09:17:47 localhost weewx[4131]: copygenerator: copied 1 >>>>>>>>>>>> files to /var/www/html/weewx/belchertown >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Monday, August 27, 2018 at 8:35:00 AM UTC-5, David Hathaway >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks Pat. I moved the options into weewx.conf after reading >>>>>>>>>>>>> that suggestion. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 27, 2018 at 8:03 AM Pat <> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I'd suggest using the latest release >>>>>>>>>>>>>> <https://github.com/poblabs/weewx-belchertown/releases>. No >>>>>>>>>>>>>> issues with upgrading if all your Belchertown options are within >>>>>>>>>>>>>> weewx.conf. See the GitHub readme >>>>>>>>>>>>>> <https://github.com/poblabs/weewx-belchertown> for those >>>>>>>>>>>>>> options. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "weewx-user" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> weewx-user+...@googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "weewx-user" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> weewx-user+...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "weewx-user" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> weewx-user+...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "weewx-user" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
