You should terminate your lines with '\n', not '\r' when you write your
file.
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "Received message:", data
with open(fname, 'w') as f:
for ds in data.split(','):
f.write(ds + "\n")
-tk
On Tue, May 7, 2019 at 3:35 AM Colin Larsen <[email protected]> wrote:
> Thanks Gary appreciate the thoughts, I'll try those out tomorrow
>
> Colin
>
> On Tue, 7 May 2019, 20:23 gjr80, <[email protected]> wrote:
>
>> A few thoughts. If it works sometimes and not others there must be a
>> problem with either the consistency of the format of the data or the
>> algorithm being used to parse the data. Can't say I am a big fan of slicing
>> in a situation like this, I think it can be a bit rigid at times. Maybe
>> consider using split. To to obtain your name, value pair maybe something
>> like:
>>
>> ...
>> for line in fd:
>> data = line.split("=")
>> if len(data) != 2:
>> continue
>> name,value = data
>> new_record_data[self.label_map.get(name, name)] =
>> to_float(value)
>> ...
>>
>> or cut out the middle man if you like:
>>
>> ...
>> for line in fd:
>> data = line.split("=")
>> if len(data) != 2:
>> continue
>> new_record_data[self.label_map.get(data[0], data[0])] =
>> to_float(data[1])
>> ...
>>
>> Gary
>>
>> On Tuesday, 7 May 2019 17:05:27 UTC+10, Colin Larsen wrote:
>>>
>>> And this is the code the writes the txt file - capturing a string via UDP
>>>
>>> *#!/usr/bin/env python*
>>>
>>> *import* socket
>>>
>>> *from* shutil *import* copyfile
>>>
>>>
>>> UDP_IP = *"0.0.0.0"*
>>>
>>> UDP_PORT = 9886
>>>
>>> fname = *"/home/pi/AQIData.txt"*
>>>
>>>
>>> sock = socket.socket(socket.AF_INET,* # Internet*
>>>
>>> socket.SOCK_DGRAM)* # UDP*
>>>
>>> sock.bind((UDP_IP, UDP_PORT))
>>>
>>>
>>>
>>> *while* True:
>>>
>>>
>>> data, addr = sock.recvfrom(1024)* # buffer size is 1024 bytes*
>>>
>>> *print* *"Received message:"*, data
>>>
>>> dataSplit = data.split(*','*)
>>>
>>> f = open(fname,*'w'*)
>>>
>>> *for* ds *in* dataSplit:
>>>
>>> f.write(ds + *"\r"*)
>>>
>>> f.write(*'\n'*)
>>>
>>> f.close()
>>>
>>> On Tue, May 7, 2019 at 7:01 PM Colin Larsen <[email protected]> wrote:
>>>
>>>> Apparently #015 is a numeric representation of \r or 'return' to we
>>>> can rule that out?
>>>>
>>>> On Tue, May 7, 2019 at 6:56 PM Colin Larsen <[email protected]> wrote:
>>>>
>>>>> Well this is a little strange. This went fine for a while then stopped
>>>>> again with pretty much the same error. It appears to be missing the
>>>>> beginning of the first line in the file which is AQI25 = then gets the
>>>>> reading which is 4.55
>>>>>
>>>>> invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex =
>>>>> 18#015AQICO2 = 863
>>>>>
>>>>> Is the #015 a clue of any sorts?
>>>>>
>>>>> The file looks ok
>>>>>
>>>>> AQI25 = 4.55
>>>>>
>>>>> AQI100 = 9.09
>>>>>
>>>>> AQIIndex = 18
>>>>>
>>>>> AQICO2 = 863
>>>>>
>>>>>
>>>>> Error;
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: engine: Main loop exiting.
>>>>> Shutting engine down.
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: engine: Caught
>>>>> unrecoverable exception in engine:
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** invalid literal
>>>>> for float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** Traceback (most
>>>>> recent call last):
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** engine.run()
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>>> callback(event)
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>>> self._software_catchup()
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>>>>> record=record, origin='software'))
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>>> callback(event)
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>>> new_record_data[self.label_map.get(name,
>>>>> name)] = to_float(value)
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>>> "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** return
>>>>> float(x) if x is not None else None
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** ValueError:
>>>>> invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex =
>>>>> 18#015AQICO2 = 863
>>>>>
>>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** Exiting.
>>>>>
>>>>> On Tue, May 7, 2019 at 3:23 PM Colin Larsen <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi Thomas
>>>>>>
>>>>>> Thanks for the quick reply. It was just the format, it needs the
>>>>>> spaces so each line in the text file is
>>>>>>
>>>>>>
>>>>>> AQI25 = 1.60
>>>>>>
>>>>>> AQI100 = 3.30
>>>>>>
>>>>>> AQIIndex = 6
>>>>>>
>>>>>> AQICO2 = 683
>>>>>>
>>>>>>
>>>>>> Once I did that and checked the database after an archive period the
>>>>>> fields were populated
>>>>>>
>>>>>> Does my section with the units look valid? I've borrowed that from a
>>>>>> similar service but I can't wrap my head around how the relationship
>>>>>> between those and the data in the txt file get joined together
>>>>>>
>>>>>>
>>>>>> Thanks again
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 7, 2019 at 3:13 PM Thomas Keffer <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, Colin
>>>>>>>
>>>>>>> Does your text file use newlines ('\n') as line delineators? Perhaps
>>>>>>> it is using the MS-DOS standard of '\n\r'? Examine the file carefully
>>>>>>> and
>>>>>>> make sure it contains what you think it contains.
>>>>>>>
>>>>>>> -tk
>>>>>>>
>>>>>>> On Mon, May 6, 2019 at 8:05 PM Colin Larsen <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> I'm trying to get Filepile working but have come across an error
>>>>>>>> that I don't understand. Any help appreciated. The database has been
>>>>>>>> extended with fields to match those below in the data file - am I just
>>>>>>>> missing spaces in the layout? Should it be AQI25 = 1.60 etc etc
>>>>>>>>
>>>>>>>> Many thanks
>>>>>>>>
>>>>>>>> This is my data file;
>>>>>>>>
>>>>>>>> AQI25=1.60
>>>>>>>>
>>>>>>>> AQI100=3.30
>>>>>>>>
>>>>>>>> AQIIndex=6
>>>>>>>>
>>>>>>>> AQICO2=683
>>>>>>>>
>>>>>>>>
>>>>>>>> This is my stanza in weewx.conf
>>>>>>>>
>>>>>>>>
>>>>>>>> ##############################################################################
>>>>>>>>
>>>>>>>>
>>>>>>>> # This section is for FilePile
>>>>>>>>
>>>>>>>>
>>>>>>>> [FilePile]
>>>>>>>>
>>>>>>>>
>>>>>>>> filename = /home/pi/AQIData.txt
>>>>>>>>
>>>>>>>> unit_system = METRIC
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ##############################################################################
>>>>>>>>
>>>>>>>>
>>>>>>>> This is my filepile.py (modified from original to suit)
>>>>>>>>
>>>>>>>>
>>>>>>>> *import* syslog
>>>>>>>>
>>>>>>>> *import* weewx
>>>>>>>>
>>>>>>>> *import* weewx.units
>>>>>>>>
>>>>>>>> *from* weewx.wxengine *import* StdService
>>>>>>>>
>>>>>>>> *from* weeutil.weeutil *import* to_float
>>>>>>>>
>>>>>>>>
>>>>>>>> weewx.units.USUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>>>
>>>>>>>> weewx.units.MetricUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>>>
>>>>>>>> weewx.units.MetricWXUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>>>
>>>>>>>> weewx.units.default_unit_format_dict[*'ppm'*] = *'%.0f'*
>>>>>>>>
>>>>>>>> weewx.units.default_unit_label_dict[*'ppm'*] = *' ppm'*
>>>>>>>>
>>>>>>>>
>>>>>>>> weewx.units.USUnits[*'group_dust'*] =
>>>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>>>
>>>>>>>> weewx.units.MetricUnits[*'group_dust'*] =
>>>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>>>
>>>>>>>> weewx.units.MetricWXUnits[*'group_dust'*] =
>>>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>>>
>>>>>>>> weewx.units.default_unit_format_dict[*'microgramm_per_meter_cubic'*
>>>>>>>> ] = *'%.1f'*
>>>>>>>>
>>>>>>>> weewx.units.default_unit_label_dict[*'microgramm_per_meter_cubic'*]
>>>>>>>> = *' \xce\xbcg/m\xc2\xb3'*
>>>>>>>>
>>>>>>>>
>>>>>>>> *class* FilePile(StdService):
>>>>>>>>
>>>>>>>> *"""WeeWX service for augmenting a record with data parsed
>>>>>>>> from a file."""*
>>>>>>>>
>>>>>>>>
>>>>>>>> *def** __init__*(self, engine, config_dict):
>>>>>>>>
>>>>>>>> * # Initialize my superclass:*
>>>>>>>>
>>>>>>>> super(FilePile, self).__init__(engine, config_dict)
>>>>>>>>
>>>>>>>> * # Extract our stanza from the configuration dicdtionary*
>>>>>>>>
>>>>>>>> filepile_dict = config_dict.get(*'FilePile'*, {})
>>>>>>>>
>>>>>>>> * # Get the location of the file ...*
>>>>>>>>
>>>>>>>> self.filename = filepile_dict.get(*'filename'*,
>>>>>>>> *'/var/tmp/filepile.txt'*)
>>>>>>>>
>>>>>>>> * # ... and the unit system it will use*
>>>>>>>>
>>>>>>>> unit_system_name = filepile_dict.get(*'unit_system'*,
>>>>>>>> *'METRICWX'*).strip().upper()
>>>>>>>>
>>>>>>>> * # Make sure we know about the unit system. If not, raise
>>>>>>>> an exception.*
>>>>>>>>
>>>>>>>> *if* unit_system_name *not* *in*
>>>>>>>> weewx.units.unit_constants:
>>>>>>>>
>>>>>>>> *raise* ValueError(*"FilePile: Unknown unit system:
>>>>>>>> %s"* % unit_system_name)
>>>>>>>>
>>>>>>>> * # Use the numeric code for the unit system*
>>>>>>>>
>>>>>>>> self.unit_system =
>>>>>>>> weewx.units.unit_constants[unit_system_name]
>>>>>>>>
>>>>>>>>
>>>>>>>> * # Mapping from variable names to weewx names*
>>>>>>>>
>>>>>>>> self.label_map = filepile_dict.get(*'label_map'*, {})
>>>>>>>>
>>>>>>>> syslog.syslog(syslog.LOG_INFO, *"filepile: Using %s with
>>>>>>>> the '%s' unit system"*
>>>>>>>>
>>>>>>>> % (self.filename, unit_system_name))
>>>>>>>>
>>>>>>>> syslog.syslog(syslog.LOG_INFO, *"filepile: Label map is
>>>>>>>> %s"* % self.label_map)
>>>>>>>>
>>>>>>>>
>>>>>>>> * # Bind to the NEW_ARCHIVE_RECORD event*
>>>>>>>>
>>>>>>>> self.bind(weewx.NEW_ARCHIVE_RECORD,
>>>>>>>> self.new_archive_record)
>>>>>>>>
>>>>>>>>
>>>>>>>> *def** new_archive_record*(self, event):
>>>>>>>>
>>>>>>>> new_record_data = {}
>>>>>>>>
>>>>>>>> *try*:
>>>>>>>>
>>>>>>>> *with* open(self.filename, *'r'*) *as* fd:
>>>>>>>>
>>>>>>>> *for* line *in* fd:
>>>>>>>>
>>>>>>>> eq_index = line.find(*'='*)
>>>>>>>>
>>>>>>>> * # Ignore all lines that do not have an equal sign*
>>>>>>>>
>>>>>>>> *if* eq_index == -1:
>>>>>>>>
>>>>>>>> *continue*
>>>>>>>>
>>>>>>>> name = line[:eq_index].strip()
>>>>>>>>
>>>>>>>> value = line[eq_index + 1:].strip()
>>>>>>>>
>>>>>>>> new_record_data[self.label_map.get(name,
>>>>>>>> name)] = to_float(value)
>>>>>>>>
>>>>>>>> * # Supply a unit system if one wasn't included in
>>>>>>>> the file*
>>>>>>>>
>>>>>>>> *if* *'usUnits'* *not* *in* new_record_data:
>>>>>>>>
>>>>>>>> new_record_data[*'usUnits'*] = self.unit_system
>>>>>>>>
>>>>>>>> * # Convert the new values to the same unit system
>>>>>>>> as the record*
>>>>>>>>
>>>>>>>> target_data =
>>>>>>>> weewx.units.to_std_system(new_record_data, event.record[*'usUnits'*
>>>>>>>> ])
>>>>>>>>
>>>>>>>> * # Add the converted values to the record:*
>>>>>>>>
>>>>>>>> event.record.update(target_data)
>>>>>>>>
>>>>>>>> *except* IOError *as* e:
>>>>>>>>
>>>>>>>> syslog.syslog(syslog.LOG_ERR, *"FilePile: Cannot open file.
>>>>>>>> Reason: %s"* % e)
>>>>>>>>
>>>>>>>> This is the syslog error;
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: engine: Caught
>>>>>>>> unrecoverable exception in engine:
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** invalid literal
>>>>>>>> for float(): 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** Traceback (most
>>>>>>>> recent call last):
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** engine.run()
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>>> callback(event)
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>>> self._software_catchup()
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>>>>>>>> record=record, origin='software'))
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>>> callback(event)
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>>> new_record_data[self.label_map.get(name,
>>>>>>>> name)] = to_float(value)
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>>> "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** return
>>>>>>>> float(x) if x is not None else None
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** ValueError:
>>>>>>>> invalid literal for float():
>>>>>>>> 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683
>>>>>>>>
>>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** Exiting.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "weewx-user" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "weewx-user" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/0f6edb07-8759-4ca4-8b91-d0745db22848%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-user/0f6edb07-8759-4ca4-8b91-d0745db22848%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/CACjxfUtAX4SeZxkJWPZdtOSKFAOVaZY6-xJhUPmaAmTth5ATDA%40mail.gmail.com
> <https://groups.google.com/d/msgid/weewx-user/CACjxfUtAX4SeZxkJWPZdtOSKFAOVaZY6-xJhUPmaAmTth5ATDA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/CAPq0zEA%3DdCv_XjMyW_9%2BmhX-gTmfH6FPwU7w5SpRMCZTLNQJMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
