OK, let's see if we can clear up a few things here. just and FYI, so my level of knowledge is understood: I am not a > programmer, just a network engineer, with a fair amount of experience with > linux and internet systems, so I am fumbling thru the sqlite syntax, and > trying to look at the python code with a very basic knowledge of where > things are being called. >
That's OK, I am just an electronics engineer and an outdated one at that, when I started using WeeWX I wondered 'is that python with an ie or a y', so you will be fine. My graphs being generated have gaps, and I think it is because of the long > data? > To understand why you see gaps in plots you need to understand a little of how the WeeWX plot engine works. Assume we are plotting an observation on a 'line' type plot. When WeeWX starts plotting points for the line the current point is joined to the last point with a line. However, to provide a realistic plot when there is say a large gap in time for the observation concerned (say your sensor was offline for some time) rather than join two points quite some time apart with a straight line, WeeWX will leave out the joining line if the gap in time is sufficiently long. The gaps are controlled by the line_gap_fraction setting in skin.conf under [ImageGenerator]. A gap is considered to be *the time period covered by the entire plot* * line_gap_fraction. Any gaps in data greater than this value will be considered a gap and the connecting line omitted. So for a standard WeeWX day plot (nominally covering 27 hours) and the default line_gap_fraction value of 0.01 this value is about 16 minutes (it will actually vary as it is calculated based on the timestamps of the earliest point in the plot and the most recent point in the plot). So I would expect that you will be seeing gaps if you have successive points that are > 16 minutes apart. Remember that whilst you may have archive records every 5 or so minutes if the observation concerned is None/NULL that point/timestamp is omitted from the plot series so it is quite possible to have continuous archive records every 5 minutes but still have gaps if data is missing. A common source of 'gaps in plots' is Davis station users who initially use the default 30 minute archive period, they only have data points in the plots every 30 minutes so the plots end up being a series of sparsely spaced dots. Not sure what you mean by long data, type LONG or long as in 'this one has more digits than that one'? Either way it doesn't matter, if the data is in the archive and is not None/NULL it will be plotted without detriment. Is some of this cause by the missing_calc=True switch > All the calc-missing option does is calculate any derived observations that are missing from imported data. So for example if dewpoint is missing from an imported record and calc-missing is True then WeeWX will calculate dewpoint for that record. As usual there are a couple of provisos; firstly WeeWX can only calculate those derived obs that it knows about, these are specified in the [StdWXCalculate] [[Calculations]] stanza in weewx.conf. So for a default install that would be pressure, barometer, altimeter, windchill, heatindex, dewpoint, inDewpoint and rainRate. Secondly, WeeWX can only calculate the derived obs if the required pre-requisites exist, for example dewpoint is a function of outTemp and outHumidity, if outHumidity does not exist dewpoint cannot be calculated and it will be left as is. The one to watch out here is WeeWX field pressure (aka station pressure), if the station provides WeeWX with barometer and WeeWX is to calculate pressure the outTemp value from 12 hours previous is required, if that historical data is not available then pressure will not be calculated. This is why you sometimes see no pressure data for the first 12 hours of an import (and this is the main cause for the reworking of wee_import in v4) So unless you have some very weird intermittent data I suspect that calc-missing will have nothing to do with your gaps. Any thoughts? > Yes, for any meaningful diagnosis you need to provide the plot you believe to have issues and details of the underlying data. You have provided details of the data (I think) but without knowing when or what was plotted it is impossible to say anything more. Just a few words on wee_import. wee_import was never designed with the intent of using it as a cron job for importing WU data. In fact, there was some trepidation on Tom's behalf when a WU module was proposed for wee_import, WU is just to , well, all over the place and full of quirks to be a reliable and consistent data source. Take for example the current (new) WU API. During testing of the v4 wee_import it was found that successive API queries for the current days data for a given station would only return part of the day's data. The first query on the current day would return all records up until the current time of day but subsequent queries on the current day would return the same data as the first query of the day, any records published between the two queries would not be returned. Queries on historical days were fine. For this reason the v4 Utilities Guide has the following note warning to not rely on WU imports for the current day: *The new (2019) Weather Underground API appears to have an issue when obtaining historical data for the current day. The first time the API is queried the API returns all historical data up to and including the most recent record. However, subsequent later API queries during the same day return the same set of records rather than all records up to and including the time of the latest API query. Users importing Weather Underground data that includes data from the current day are advised to carefully check the WeeWX log to ensure that all expected records were imported. If some records are missing from the current day try running an import for the current day again using the --date option setting. If this fails then wait until the following day and perform another import for the day concerned again using the --date option setting. In all cases confirm what data has been imported by referring to the WeeWX log.* I am not 100% certain exactly how you are running wee_import but if you are using a cron job to import the current day then you could quite likely be being affected by this quirk in the WU API. If the cron job runs continuously it is quite possible that sometime in the future (of the current day) the API query returns the correct data so you could be seeing some hybrid mix of current and missing data. My suggestion is to not run wee_import as a cron job. If you must run it as a cron job then run it on historical days only. I change the trachea value from 250 to 50 thinking that had something to do > with it > Perhaps you mean tranche? If so this merely sets the number of records save to database in a single transaction. It will not affect the actual data being stored and it will only have a noticeable affect on very large imports. The odd thing is that when I do a full days import I see some PRIMARY Key > errors, but if I wait a couple hours the graphs update. I do not > understand why the graphs do not update immediately like they did before > the api_key was required back in May? > What graphs? WeeWX generated I presume. Remember the WeeWX generated plots update at different rates dependent on the aggregate period being used. Daily plots (no aggregate period) update every time the reports are run. Weekly plots use a one hour aggregate so update every hour, month use a three hour aggregate so update every three hours and year use a 24 hour aggregate (and you can guess when they update). The best approach when checking plots for changes is to delete all plots before the report cycle occurs, this forces all plots to be regenerated. Checking plot file timestamps via the operating system is a quick way to see the plot was last generated. The log also gives a good clue as well. I might mention that this box is running simulator driver, and I pull the > data from WU with wee_importv4 / wee-reportsv3.8.0 to populate the graphs > via cheetah and the Standard template. > Soooo, simulator writes data to archive and then you use wee_import to import WU data, is that WU data that covers the same times as the simulator was running? If so the WU data will not be saved to archive. Is the WU data the simulator data that has been posted to WU or data from another station? If the latter I expect there may be a mix of simulated and real life data. Not a problem in itself but I expect it may confuse the issue when seeing what was/was not imported, especially when judging by plots. One more hint, when providing an SQLite query output use the .headers on command before executing the query and then post the resulting output with the headers info and the data, that way we can see what each field contains. Gary -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a4a81250-0a53-4e15-96f3-a3fd9405b893%40googlegroups.com.
