Trying to figure out what happens if there are packets from previous
archive interval that somehow have not been processed yet and arrive in the
current archive interval.
In StdArchive.new_loop_packet it would raise weewx.accum.OutOfSpan; but
then it would create a new accumulator assuming it had moved on to the next
interval
(not it was still stuck in the last interval). This could not happen
unless the packets arrived out of sequence, with old timestamp after new
timestamp which is possible with two data sources like what I have.
You can "cheat" and instead of using an RTC (real time clock) to get the
timestamp just have the driver timestamp each packet with the time it
received it. The drift on my RTC is usually only a couple msec per hour.
On Tuesday, April 25, 2023 at 8:22:12 AM UTC-7 William Garber wrote:
> jjj.2.edited.a.grep_packet.log shows all the new packet information
> between 6:45am and 6:55am.
> typescript.4.edited shows the database dump of weewx_atlas.sdb at 6:50am
> when there was a gap (None / NULL values for data).
> you can see that weewx should have got a full set of data for that archive
> interval. The archive interval is 300 = 5 minutes.
> Could it be that there is just too much data? About half of the data is
> from the Atlas and half from the indoor sensors.
> Weewx can handle the data from the atlas. The only difference is that the
> weewx-sdr driver removes duplicate data from the Atlas
> but there are no duplicates from the indoor sensors.
>
> On Monday, April 24, 2023 at 11:34:22 PM UTC-7 William Garber wrote:
>
>> The modification to AsyncReader is to add a write mode so weewx acts as
>> a timeserver. Triggered by sentinel "request_datetime".
>> The timeserver was not called during the gap at 9:15pm in the previous
>> post attachment and log. Search the log for "AsyncReader"
>>
>>
>> def run_read_mode(self):
>> # fixme I think this should be longer than the loop time for the
>> arduino
>> timeout = 12
>> # select(rlist,wlist,xlist,timeout); wait until ready for reading
>> (rlist);
>> ready,_,_ = select.select([self._fd], [], [], timeout)
>> if not ready:
>> return
>> line = self._fd.readline()
>> # fixme verbose
>>
>> logdbg("read line = '%s'" % (line))
>> line = line.rstrip()
>> # all our files are binary
>>
>> line = line.decode()
>> if not line:
>> return
>> if self._write_mode and (line == "request_datetime"):
>> self._read_mode = False
>> self._write_attempt = 0
>> return
>> # garberw begin -----------------------
>>
>> # we are not processing any lines other than json so just keep
>> only json;
>> # this saves a little work until we get to PacketFactory.create;
>>
>> # to see the debugging info comment this line out;
>>
>> # PacketFactory.parse_json handles this line;
>>
>> # PacketFactory.parse_text handles the other lines; we never use
>> that;
>> #if not line.startswith("{"):
>>
>> # return
>>
>> # garberw end -----------------------
>>
>> # line = fudge_time()
>>
>> self._queue.put(line)
>>
>> def run_write_mode(self):
>> # fixme check this
>>
>> timeout = 12
>> # select(rlist,wlist,xlist,timeout); wait until ready for writing
>> (wlist);
>> _,ready,_ = select.select([], [self._fd], [], timeout)
>> if not ready:
>> self._write_attempt += 1
>> if self._write_attempt >= self.WRITE_ATTEMPT_MAX:
>> logdbg("AsyncReader write timeout")
>> self._read_mode = True
>> return
>> # int(1677178029.12345) seconds since epoch
>>
>> time_now_epoch = int(time.time())
>> # '1677178029\n'
>>
>> time_str_epoch = '{}\n'.format(time_now_epoch)
>> # b'1677178029\n'
>>
>> buf = bytes(time_str_epoch, 'ascii')
>> self._fd.write(buf)
>> self._fd.flush()
>> loginf("AsyncReader wrote timestamp={}
>> buf={}".format(time_now_epoch, buf))
>> self._read_mode = True
>>
>> Maybe the problem is that the queue in AsyncReader gets too full before
>> ProcessManager.get_stdout()
>> yields all the lines to PacketFactory.create() which parses the json in
>> them and
>> pops each line off the list "lines". Sort of like flushing the input
>> buffer.
>> ProcessManager.get_stdout() yields the lines immediately if they contain
>> a timestamp matching
>> TS = re.compile('^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d[\s]+')
>> maybe this would "flush the input buffer sooner" but this doesn't work
>> for me since I am using unix epoch time as an integer
>> not formatted like TS.
>> On Monday, April 24, 2023 at 10:31:21 PM UTC-7 William Garber wrote:
>>
>>> jjj.log.gz is journalctl -b -u weewx_atlas.service
>>> typescript is an sql query of the database weewx_atlas.sdb showing a gap
>>> at 9:15pm.
>>> could it be garbage collecting caused the gap ?
>>> On Monday, April 24, 2023 at 10:12:15 PM UTC-7 William Garber wrote:
>>>
>>>> I have a weewx-sdr driver receiving data over usb from rtl-sdr from an
>>>> rtl-sdr radio receiver dongle. The radio receiver gets outdoor data from
>>>> an Acurite Atlas. The inside weather data comes from an arduino over usb.
>>>>
>>>> The weewx server is on a raspberry pi. I modified the weewx-sdr driver
>>>> AsyncReader (in the beginning of the file) and ProcessManager. There is
>>>> now an AsyncReader thread for both the rtl433 ttyACM0 connection receiving
>>>> data over USB and for the arduino reading data over usb. These both share
>>>> the same queue where they save both indoor and outdoor lines of input data.
>>>> The driver only accepts data formatted as json and rejects the other
>>>> lines which are for debugging. There is a lot of extra discarded data.
>>>>
>>>> It looks like there simply be so much data that the weewx server skips
>>>> some of it sometimes. I get some records with nothing but None except for
>>>> rain.
>>>>
>>>> The archive interval is 5 minutes. The Atlas and the arduino both
>>>> generate data every 7 seconds or so. The arduino was much faster and I
>>>> got
>>>> more gaps. When I slowed the arduino down to the speed of the Atlas there
>>>> were fewer gaps.
>>>>
>>>> Maybe I should just increase the archive interval? Or decrease it?
>>>> I don't know how to change the Atlas logging interval.
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/b4277ff5-e035-4824-9356-9c31a03b4871n%40googlegroups.com.
