Re: Async serial communication/threads sharing data
On Wed, 25 Mar 2009 22:23:25 -0700, John Nagle na...@animats.com wrote: Jean-Paul Calderone wrote: On Tue, 24 Mar 2009 22:20:49 -0700, John Nagle na...@animats.com wrote: Jean-Paul Calderone wrote: On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood n...@craig-wood.com wrote: Jean-Paul Calderone exar...@divmod.com wrote: [snip] After bringing in all the heavy machinery of Twisted, you're still polling at 10Hz. That's disappointing. Hmm, no? There's no polling in the latest version of the code I saw. Jean-Paul if too_full: reactor.callLater(0.1, self.throttle) Hi John, You're looking at a different version of the code than what I was talking about. After discussing things with Nick, we came up with a version which includes no polling. See Nick's message of March 23, at around 1:30 PM for the code I was referring to - a version of the app which does no polling. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: These days, serial ports are on the way out, I think. I don't see generic USB and bluetooth serial devices disappear that fast... E.g. AFAIK (almost?) all GPS receivers have to be accessed as serial devices (mine even looks like a generic USB-to-serial device to the kernel). -- JanC -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Tue, 24 Mar 2009 22:20:49 -0700, John Nagle na...@animats.com wrote: Jean-Paul Calderone wrote: On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood n...@craig-wood.com wrote: Jean-Paul Calderone exar...@divmod.com wrote: [snip] After bringing in all the heavy machinery of Twisted, you're still polling at 10Hz. That's disappointing. Hmm, no? There's no polling in the latest version of the code I saw. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: On Tue, 24 Mar 2009 22:20:49 -0700, John Nagle na...@animats.com wrote: Jean-Paul Calderone wrote: On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood n...@craig-wood.com wrote: Jean-Paul Calderone exar...@divmod.com wrote: [snip] After bringing in all the heavy machinery of Twisted, you're still polling at 10Hz. That's disappointing. Hmm, no? There's no polling in the latest version of the code I saw. Jean-Paul if too_full: reactor.callLater(0.1, self.throttle) John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Hendrik van Rooyen wrote: Nick Craig-Wood ni...g-wood.com wrote: I wrote a serial port to TCP proxy (with logging) with twisted. The problem I had was that twisted serial ports didn't seem to have any back pressure. Not sure if this is Twisted's fault - do python sockets have automatic back pressure? Do Linux sockets have back pressure? Yes, and yes. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood n...@craig-wood.com wrote: Jean-Paul Calderone exar...@divmod.com wrote: [snip] After bringing in all the heavy machinery of Twisted, you're still polling at 10Hz. That's disappointing. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Nick Craig-Wood ni...g-wood.com wrote: I wrote a serial port to TCP proxy (with logging) with twisted. The problem I had was that twisted serial ports didn't seem to have any back pressure. By that I mean I could pump data into a 9600 baud serial port at 10 Mbit/s. Twisted would then buffer the data for me using 10s or 100s or Megabytes of RAM. No data would be lost, but there would be hours of latency and my program would use up all my RAM and explode. What I wanted to happen was for twisted to stop taking the data when the serial port buffer was full and to only take the data at 9600 baud. I never did solve that problem :-( Not sure if this is Twisted's fault - do python sockets have automatic back pressure? Do Linux sockets have back pressure? - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone exar...@divmod.com wrote:
On Sun, 22 Mar 2009 12:30:04 -0500, Nick Craig-Wood n...@craig-wood.com
wrote:
I wrote a serial port to TCP proxy (with logging) with twisted. The
problem I had was that twisted serial ports didn't seem to have any
back pressure. By that I mean I could pump data into a 9600 baud
serial port at 10 Mbit/s. Twisted would then buffer the data for me
using 10s or 100s or Megabytes of RAM. No data would be lost, but
there would be hours of latency and my program would use up all my RAM
and explode.
What I wanted to happen was for twisted to stop taking the data when
the serial port buffer was full and to only take the data at 9600
baud.
This is what Twisted's producers and consumers let you do. There's a
document covering these features:
http://twistedmatrix.com/projects/core/documentation/howto/producers.html
In the case of a TCP to serial forwarder, you don't actually have to
implement either a producer or a consumer, since both the TCP connection
and the serial connection are already both producers and consumers. All
you need to do is hook them up to each other so that when the send buffer
of one fills up, the other one gets paused, and when the buffer is empty
again, it gets resumed.
I eventually came up with this which seems to work, but I'm not sure
it is the best way of doing it as it had to mess about with the
twisted internals to get the number of bytes in the serial port
output buffer.
class SerialPort(protocol.Protocol):
Create a serial port connection and pass data from it to a known list of
TCP ports.
def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port, baudrate,
rtscts=0)
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')
def write(self, data):
Write data to the serial port.
self.serial.write(data)
if self.log:
self.log.write(data)
self.throttle()
def throttle(self):
Pause the inputs if there is too much data in the output buffer
bytes_in_buffer = len(self.serial.dataBuffer) + self.serial._tempDataLen
too_full = bytes_in_buffer 1024
for tcp_port in self.tcp_ports:
if too_full:
tcp_port.transport.pauseProducing()
else:
tcp_port.transport.resumeProducing()
if too_full:
reactor.callLater(0.1, self.throttle)
--
Nick Craig-Wood n...@craig-wood.com -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood n...@craig-wood.com wrote:
Jean-Paul Calderone exar...@divmod.com wrote:
[snip]
In the case of a TCP to serial forwarder, you don't actually have to
implement either a producer or a consumer, since both the TCP connection
and the serial connection are already both producers and consumers. All
you need to do is hook them up to each other so that when the send buffer
of one fills up, the other one gets paused, and when the buffer is empty
again, it gets resumed.
I eventually came up with this which seems to work, but I'm not sure
it is the best way of doing it as it had to mess about with the
twisted internals to get the number of bytes in the serial port
output buffer.
This is sort of on the right track. Here's how to do it without
touching implementation details:
class SerialPort(protocol.Protocol):
Create a serial port connection and pass data from it to a known list of TCP
ports.
def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port, baudrate,
rtscts=0)
Here, register this object to receive buffer full and not-full events from
the serial port:
self.serial.registerProducer(self, True)
And an attribute to keep track of our state:
self.paused = False
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')
I'm not exactly sure where `self.tcp_ports´ gets populated, so I'll make a
method to do it:
def addPort(self, port):
if self.paused:
port.transport.pauseProducing()
self.tcp_ports.append(port)
Now a method to handle serial port buffer full events:
def pauseProducing(self):
self.paused = True
for port in self.tcp_ports:
port.transport.pauseProducing()
And a method to handle buffer no longer full events:
def resumeProducing(self):
self.paused = False
for port in self.tcp_ports:
port.transport.resumeProducing()
With these, you can get rid of `throttle´ entirely.
def write(self, data):
Write data to the serial port.
self.serial.write(data)
if self.log:
self.log.write(data)
self.throttle()
def throttle(self):
Pause the inputs if there is too much data in the output buffer
bytes_in_buffer = len(self.serial.dataBuffer) + self.serial._tempDataLen
too_full = bytes_in_buffer 1024
for tcp_port in self.tcp_ports:
if too_full:
tcp_port.transport.pauseProducing()
else:
tcp_port.transport.resumeProducing()
if too_full:
reactor.callLater(0.1, self.throttle)
Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone exar...@divmod.com wrote:
On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood n...@craig-wood.com
wrote:
Jean-Paul Calderone exar...@divmod.com wrote:
[snip]
In the case of a TCP to serial forwarder, you don't actually have to
implement either a producer or a consumer, since both the TCP connection
and the serial connection are already both producers and consumers. All
you need to do is hook them up to each other so that when the send buffer
of one fills up, the other one gets paused, and when the buffer is empty
again, it gets resumed.
I eventually came up with this which seems to work, but I'm not sure
it is the best way of doing it as it had to mess about with the
twisted internals to get the number of bytes in the serial port
output buffer.
This is sort of on the right track. Here's how to do it without
touching implementation details:
Thank you for that!
See below for the complete prog with your suggested modifications.
That seems to solve the problem - I can see the writer pausing and
unpausing at the serial port rate.
write 16446
write 6430
pause producing
resume producing
write 65536
write 56724
pause producing
resume producing
write 65536
write 65536
pause producing
It has exposed a problem with the sender not throttling properly now,
but I guess that is progress!
Thanks for your help
Here is the complete program FYI with your suggested mods.
#!/usr/bin/python
Transfer data between a serial port and one (or more) TCP
connections.
options:
-h, --help:this help
-p, --port=PORT: port, a number, default = 0 or a device name
-b, --baud=BAUD: baudrate, default 115200
-t, --tcp=PORT: TCP port number, default 1234
-l, --log: log data streams to 'snifter-0', 'snifter-1'
-L, --log_name=NAME: log data streams to 'NAME-0', 'NAME-1'
import sys
import getopt
from twisted.internet import reactor, protocol, serialport
from zope.interface import implements
from twisted.internet import protocol, interfaces
# FIXME set serial buffer size? SEND_LIMIT
class SerialPort(protocol.Protocol):
Create a serial port connection and pass data from it to a
known list of
TCP ports.
def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port,
baudrate, rtscts=0)
self.serial.registerProducer(self, True)
self.paused = False
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')
def add_tcp(self, tcp_port):
Add a TCPPort to those receiving serial data.
if self.paused:
tcp_port.transport.pauseProducing()
self.tcp_ports.append(tcp_port)
def del_tcp(self, tcp_port):
Remove a TCPPort from the those receiving serial data.
self.tcp_ports.remove(tcp_port)
def write(self, data):
Write data to the serial port.
self.serial.write(data)
if self.log:
self.log.write(data)
def pauseProducing(self):
Pause producing event
print pause producing
self.paused = True
for port in self.tcp_ports:
port.transport.pauseProducing()
def resumeProducing(self):
Resume producing event
print resume producing
self.paused = False
for port in self.tcp_ports:
port.transport.resumeProducing()
def stopProducing(self):
Stop producing event
print stop producing
def dataReceived(self, data):
Pass any received data to the list of TCPPorts.
for tcp_port in self.tcp_ports:
tcp_port.write(data)
class TCPPort(protocol.Protocol):
Create a TCP server connection and pass data from it to the
serial port.
def __init__(self, serial, log_name, index):
Add this TCPPort to the SerialPort.
self.serial = serial
self.serial.add_tcp(self)
self.log = None
if log_name is not None:
self.log = file('%s-%d' % (log_name, index+1), 'w')
def __del__(self):
Remove this TCPPort from the SerialPort.
self.serial.del_tcp(self)
def dataReceived(self, data):
Pass received data to the SerialPort.
print write, len(data)
self.serial.write(data)
def write(self, data):
Write data to the TCP port.
self.transport.write(data)
if self.log is not None:
self.log.write(data)
class TCPPortFactory(protocol.ServerFactory):
Factory to create TCPPort protocol instances, an instanced
SerialPort must be passed in.
def __init__(self, serial, log_name=None):
self.serial = serial
self.log_name = log_name
self.index = 0
def buildProtocol(self, addr):
Build a TCPPort, passing in the instanced SerialPort.
p = TCPPort(self.serial, self.log_name, self.index)
Re: Async serial communication/threads sharing data
Nick Timkovich prom@gmail.com wrote: I've been working on a program that will talk to an embedded device over the serial port, using some basic binary communications with messages 4-10 bytes long or so. Most of the nuts and bolts problems I've been able to solve, and have learned a little about the threading library to avoid blocking all action while waiting for responses (which can take 50 ms to 10 s). Ultimately, this program will test the device on the COM port by sending it messages and collecting responses for 10k-100k cycles; a cycle being: 1. tell it to switch a relay, 2. get it's response from the event, 3. ask it for some measurements, 4. get measurements, 5. repeat. Later I would like to develop a GUI as well, but not a big issue now (another reason to use threads? not sure). The overall structure of what I should do is not very apparent to me on how to efficiently deal with the communications. Have the main loop handle the overall timing of how often to run the test cycle, then have a thread deal with all the communications, and within that another thread that just receives data? My main issue is with how to exchange data between different threads; can I just do something like have a global list of messages, appending, modifying, and removing as needed? Does the threading.Lock object just prevent every other thread from running, or is it bound somehow to a specific object (like my list)? What I have been doing for similar stuff is to modularize using processes and pipes instead of queues. This allows you to have the communications in one process, in which you hide all the protocol issues, and allows you to deal with the resulting data at a slightly higher level of abstraction. Such a structure also makes it easier to have different user interfaces - command line or GUI, it does not matter as the commands and responses to the comms module can be standardised via an input and output pipe to and from the comms module. HTH - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Mar 21, 9:19 pm, Jean-Paul Calderone exar...@divmod.com wrote: On Sat, 21 Mar 2009 13:52:21 -0700 (PDT), Nick Timkovich prometheus...@gmail.com wrote: I've been working on a program that will talk to an embedded device over the serial port, using some basic binary communications with messages 4-10 bytes long or so. Most of the nuts and bolts problems I've been able to solve, and have learned a little about the threading library to avoid blocking all action while waiting for responses (which can take 50 ms to 10 s). Ultimately, this program will test the device on the COM port by sending it messages and collecting responses for 10k-100k cycles; a cycle being: 1. tell it to switch a relay, 2. get it's response from the event, 3. ask it for some measurements, 4. get measurements, 5. repeat. Later I would like to develop a GUI as well, but not a big issue now (another reason to use threads? not sure). Twisted includes serial port support and will let you integrate with a GUI toolkit. Since Twisted encourages you to write programs which deal with things asynchronously in a single thread, if you use it, your concerns about data exchange, locking, and timing should be addressed as a simple consequence of your overall program structure. Jean-Paul I've looked at Twisted a little bit because of some searches on serial port comm turning up advice for it. However, there seems to be no/ minimal documentation for the serial portions, like they are some old relic that nobody uses from this seemingly massive package. Do you have any examples or somewhere in particular you could point me? Thanks for the responses, Nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Sun, 22 Mar 2009 03:13:36 -0700 (PDT), Nick Timkovich prometheus...@gmail.com wrote: On Mar 21, 9:19 pm, Jean-Paul Calderone exar...@divmod.com wrote: On Sat, 21 Mar 2009 13:52:21 -0700 (PDT), Nick Timkovich prometheus...@gmail.com wrote: I've been working on a program that will talk to an embedded device over the serial port, using some basic binary communications with messages 4-10 bytes long or so. Most of the nuts and bolts problems I've been able to solve, and have learned a little about the threading library to avoid blocking all action while waiting for responses (which can take 50 ms to 10 s). Ultimately, this program will test the device on the COM port by sending it messages and collecting responses for 10k-100k cycles; a cycle being: 1. tell it to switch a relay, 2. get it's response from the event, 3. ask it for some measurements, 4. get measurements, 5. repeat. Later I would like to develop a GUI as well, but not a big issue now (another reason to use threads? not sure). Twisted includes serial port support and will let you integrate with a GUI toolkit. Since Twisted encourages you to write programs which deal with things asynchronously in a single thread, if you use it, your concerns about data exchange, locking, and timing should be addressed as a simple consequence of your overall program structure. Jean-Paul I've looked at Twisted a little bit because of some searches on serial port comm turning up advice for it. However, there seems to be no/ minimal documentation for the serial portions, like they are some old relic that nobody uses from this seemingly massive package. Do you have any examples or somewhere in particular you could point me? It's true that the serial port support in Twisted isn't the most used feature. :) These days, serial ports are on the way out, I think. That said, much of the way a serial port is used in Twisted is the same as the way a TCP connection is used. This means that the Twisted documentation for TCP connections is largely applicable to using serial ports. The only major difference is how you set up the connection. You can see examples of using the serial port support here (one of them seems to suggest that it won't work on Windows, but I think this is a mistake): http://twistedmatrix.com/projects/core/documentation/examples/gpsfix.py http://twistedmatrix.com/projects/core/documentation/examples/mouse.py The 2nd to last line of each example shows how to connect to the serial port. These basic documents describe how to implement a protocol. Though in the context of TCP, the protocol implementation ideas apply to working with serial ports as well: http://twistedmatrix.com/projects/core/documentation/howto/servers.html http://twistedmatrix.com/projects/core/documentation/howto/clients.html You can ignore the parts about factories, since they're not used with serial ports. Hope this helps, Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone exar...@divmod.com wrote: It's true that the serial port support in Twisted isn't the most used feature. :) These days, serial ports are on the way out, I think. That said, much of the way a serial port is used in Twisted is the same as the way a TCP connection is used. This means that the Twisted documentation for TCP connections is largely applicable to using serial ports. The only major difference is how you set up the connection. You can see examples of using the serial port support here (one of them seems to suggest that it won't work on Windows, but I think this is a mistake): http://twistedmatrix.com/projects/core/documentation/examples/gpsfix.py http://twistedmatrix.com/projects/core/documentation/examples/mouse.py The 2nd to last line of each example shows how to connect to the serial port. These basic documents describe how to implement a protocol. Though in the context of TCP, the protocol implementation ideas apply to working with serial ports as well: http://twistedmatrix.com/projects/core/documentation/howto/servers.html http://twistedmatrix.com/projects/core/documentation/howto/clients.html You can ignore the parts about factories, since they're not used with serial ports. I wrote a serial port to TCP proxy (with logging) with twisted. The problem I had was that twisted serial ports didn't seem to have any back pressure. By that I mean I could pump data into a 9600 baud serial port at 10 Mbit/s. Twisted would then buffer the data for me using 10s or 100s or Megabytes of RAM. No data would be lost, but there would be hours of latency and my program would use up all my RAM and explode. What I wanted to happen was for twisted to stop taking the data when the serial port buffer was full and to only take the data at 9600 baud. I never did solve that problem :-( -- Nick Craig-Wood n...@craig-wood.com -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Sun, 22 Mar 2009 12:30:04 -0500, Nick Craig-Wood n...@craig-wood.com wrote: [snip] I wrote a serial port to TCP proxy (with logging) with twisted. The problem I had was that twisted serial ports didn't seem to have any back pressure. By that I mean I could pump data into a 9600 baud serial port at 10 Mbit/s. Twisted would then buffer the data for me using 10s or 100s or Megabytes of RAM. No data would be lost, but there would be hours of latency and my program would use up all my RAM and explode. What I wanted to happen was for twisted to stop taking the data when the serial port buffer was full and to only take the data at 9600 baud. I never did solve that problem :-( This is what Twisted's producers and consumers let you do. There's a document covering these features: http://twistedmatrix.com/projects/core/documentation/howto/producers.html In the case of a TCP to serial forwarder, you don't actually have to implement either a producer or a consumer, since both the TCP connection and the serial connection are already both producers and consumers. All you need to do is hook them up to each other so that when the send buffer of one fills up, the other one gets paused, and when the buffer is empty again, it gets resumed. Hope this helps! Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Nick Timkovich prometheus...@gmail.com writes: My main issue is with how to exchange data between different threads; can I just do something like have a global list of messages, appending, modifying, and removing as needed? Does the threading.Lock object just prevent every other thread from running, or is it bound somehow to a specific object (like my list)? The favored approach to this is to use the Queue module, which gives a thread-safe double ended queue. This is cleaner than messing with lower level synchronization primitives like Lock. You use Queues as communications channels between threads. -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Nick Timkovich wrote: I've been working on a program that will talk to an embedded device over the serial port, using some basic binary communications with messages 4-10 bytes long or so... Ultimately, this program ... [send] messages and collecting responses for 10k-100k cycles ... Here's a simple structure: One thread talks to the device, it passes commands to the device, collects the response, and (when a full response is collected), sends the response back. Queues have all the locking needed, so most of us just use Queues to communicate between threads (as Paul Rubin has suggested). For example: def thread_body(command_queue, result_queue): for command, reps in iter(command_queue.get, None): result = [] try: for n in range(reps): device.send(command) result.append(device.read(reply_length(command)) finally: result_queue.put(result) Then you just do something like: import queue commands = queue.Queue() replies = queue.Queue() start_some_thread(thread_body, commands, replies) ... some loop: commands.put(fetch_voltage, 50) print(replies.get()) ... commands.put(None) Here the device-talker works away while it can, waiting until it has a command, and waiting for all replies or some failure to get back to the caller. The caller may either block for thread replies, or use replies.get(timeout=7.25) or replies.get(block=False) to wait (or not) and get a queue.Empty exception to signal no data yet. Does the threading.Lock object just prevent every other thread from running, or is it bound somehow to a specific object (like my list)? A Lock instance can be used to protect as many resources as you like; it really only decides who gets the lock; you decide what you only do while holding the lock (the other threads are running, except those waiting for the lock). --Scott David Daniels scott.dani...@acm.org -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Sat, 21 Mar 2009 13:52:21 -0700 (PDT), Nick Timkovich prometheus...@gmail.com wrote: I've been working on a program that will talk to an embedded device over the serial port, using some basic binary communications with messages 4-10 bytes long or so. Most of the nuts and bolts problems I've been able to solve, and have learned a little about the threading library to avoid blocking all action while waiting for responses (which can take 50 ms to 10 s). Ultimately, this program will test the device on the COM port by sending it messages and collecting responses for 10k-100k cycles; a cycle being: 1. tell it to switch a relay, 2. get it's response from the event, 3. ask it for some measurements, 4. get measurements, 5. repeat. Later I would like to develop a GUI as well, but not a big issue now (another reason to use threads? not sure). Twisted includes serial port support and will let you integrate with a GUI toolkit. Since Twisted encourages you to write programs which deal with things asynchronously in a single thread, if you use it, your concerns about data exchange, locking, and timing should be addressed as a simple consequence of your overall program structure. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
