Hi Gary,
You were correct!! The issue was with errors in my input file. The main
problem was how the data was being written to the file. It was being
committed to disk in chunks of what amount to about 360 lines, but it did
not always write complete lines at the last record in the chunk. I added a
file.flush() command after each file.write() command to ensure that the
data is written line-by-line. I was also able to add some code to check the
incoming data and skip any faulty records. Together those changes seem to
resolve the issue. It has been up and running for hours now. Thanks again
for your help! You rock!
On Thursday, June 20, 2019 at 7:31:31 PM UTC-4, gjr80 wrote:
>
> File access contention could be occurring, one would like to think RPi
> file system would handle that and not allow one or the other to access the
> file if it is in use. How likely is it that contention could occur, I guess
> it would depend on how often you write the file and how often you read it.
> I don't know anything of how arduinos write to a file or how your arduino
> code works but you might want to look at the structure of the code to make
> sure the file is 'written' as quickly as possible, eg rather than opening
> the file, polling a sensor, processing a one data point, writing that data
> point, polling the next sensor, etc do all the data gathering/polling up
> front and then open the file and write the data in one go. Another
> technique is to do an atomic write whereby you write the data to a
> temporary file and then once the temporary file is complete you copy the
> temporary file over the real data file.
>
> You say your changes 'seem to have eliminated the issue of writing
> incomplete or malformed records', you might want to check that is indeed
> the case. You at least want to be able to see what the file was like when
> you get the error (what error do you get, 'cannot read value' or WeeWX
> exiting?). If the data file is written by the arduino so frequently that
> you cannot see the state of the file when the error occurs alter your
> arduino code to append the same data to another file. That way you can
> backtrack and find the state of the data file as it would have been when
> the error occurred. Remember the fileparse drive is fairly basic, whilst it
> will detect and deal with lines that have no data, malformed field names or
> a missing '=' it can still be tripped up.
>
> One other thing, don't be afraid to post log extracts to show errors or
> other activity. It would have been helpful to see a log extract from the
> leadup to the 'approximately 560 records' error to the actual error. Often
> the logs may look the same or may appear to have no useful information but
> they often do. The good thing about the logs is their content lacks human
> interpretation/description.
>
> Gary
>
> On Friday, 21 June 2019 07:22:50 UTC+10, Ron Walker wrote:
>>
>> I have an update on this issue. I modified the code that I'm using to
>> retrieve the info from the arduino and write it to the file that fileparse
>> is reading. It now checks the length of the retrieved line and discards it
>> if it is less than a certain length. That seems to have eliminated the
>> issue of writing incomplete or malformed records. This has not eliminated
>> the errors. If I run the code to write to the file and then start weewx,
>> approximately 560 so records are written to the file before the error
>> starts showing up. Is it possible that the reading and writing are somehow
>> colliding?
>>
>> On Thursday, June 20, 2019 at 2:53:55 PM UTC-4, Ron Walker wrote:
>>>
>>> Hi Gary,
>>>
>>> You are correct as far as the data file being the issue. I confirmed
>>> this by creating a data file and then making sure that every record was
>>> complete. Ran weewx using this data file and it proceeded with no issues.
>>> I am using an arduino uno to collect the sensor data and then sending it
>>> out to a raspberry pi via serial port and saving it to a file.I have
>>> noticed that when the arduino starts sending data, it may send two or three
>>> lines of 'garbage" to start off. Is there a way to ensure that those lines
>>> are discarded? I am a newbie to python.
>>>
>>> Also, I put that line into the fileparse.py file, but I did not notice
>>> any more info being given regarding the error in the syslog.
>>>
>>> Ron
>>>
>>> On Thursday, June 20, 2019 at 11:55:59 AM UTC-4, gjr80 wrote:
>>>>
>>>> Ron,
>>>>
>>>> Since your data file field names are in fact WeeWX field names there is
>>>> no need for a field map, I only asked because if you had one and it had
>>>> some issues then it could be messing things up.
>>>>
>>>> On looking again at the fileparse code I realise I mis-read it, though
>>>> I am still sure that the issue is an occasional malformed data file. The
>>>> fileparse driver is pretty basic, it splits each line at the '=', the
>>>> string on the left of the '=' is the field name and the string on the
>>>> right
>>>> is the data. The data is converted to a float. If the data cannot be
>>>> converted to a float WeeWX logs the offending field name and the error
>>>> message. So in the case of this line:
>>>>
>>>> Jun 19 16:55:30 WeatherPi weewx[32206]: fileparse: cannot read value
>>>> for '': could not convert string to float: l
>>>>
>>>> it tells us that fileparse tried to process the field '' (ie an empty
>>>> string) but it could not convert the field data to a float with the error
>>>> message being 'could not convert string to float: l'. That particular
>>>> error message displays the offending data after the : and for some reason
>>>> it has been truncated. Since it is a l (lower case L) I suspect that
>>>> somehow the data file has been malformed and 'lightLevel' has somehow
>>>> ended up on the right hand side of an '=' and since the corresponding
>>>> field
>>>> name is an empty string. Either that or the data file is badly mangled. I
>>>> would say that at some stage your data file has a line:
>>>>
>>>> =lightLevel
>>>>
>>>> Have you been monitoring the data file when the error occurs? You
>>>> could try adding some debug code to fileparse.py, edit fileparse.py
>>>> and add the highlighted line shown below:
>>>>
>>>> def _get_as_float(d, s):
>>>> v = None
>>>> if s in d:
>>>> try:
>>>> v = float(d[s])
>>>>
>>>> except ValueError as e:
>>>> loginf("d=%s" %(d, ))
>>>> logerr("cannot read value for '%s': %s" % (s, e))
>>>> return v
>>>>
>>>> save the file then restart WeeWX. Next time WeeWX encounters that
>>>> particular error it will dump the entire dictionary of data that it read
>>>> from the file. Ideally we would like to see the data file but to do so
>>>> would take a bit more code, this is a simple one liner that will give us
>>>> he
>>>> next best thing and may give us a clue as to what is going on with the
>>>> data
>>>> file.
>>>>
>>>> Gary
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/d27f7947-46e5-4e2e-974e-e18b299e0ea6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
