Hi -
I still regard myself as a newbie, and may have mangled some of
the terminology, but ...
A question I can answer! I had the same problem...
Make your protocol class be an Int16StringReceiver (or
Int<size>StringReceiver, I think there is an 8 and a 32 available
as well), and Twisted will do it for you. When writing, it will
prepend the size to each message, and on the receiving side it
will accumulate pieces until it gets the full message, and then
call your stringReceived() method with the full message (length
stripped off.)
It you lose the connection in the middle of a message, I think
you'll get a connectionLost event, rather than a partial message
or an indefinite hang.
So your stringReceived() and sendString() usage looks exactly
the same, but behind the scenes, Twisted does all the work.
(TCP always delivers all the data you send in the correct order,
but the fragmentation can be random. A single message can get
split into multiple chunks, and I think multiple messages could
get appended into a single chunk. But the byte order seen by the
receiver is preserved, so the receiver can tell when a complete
message has arrived if it lays some sort of protocol on top of
the TCP stream. It could be message lengths (this is what the
Int*StringReceiver protocols do), or it could be "start" and
"Stop" characters embedded in the stream (for example, FTP in
text mode or SMTP use <CR><LF> to indicate ends of lines), or
it could be an agreed-upon count (I think FTP in binary mode
expects a series of 512-byte blocks, though I could be wrong
about this, and the receiver just partitions the incoming TCP
stream into groups of 512 bytes.) I'm sure there are lots of
other ways to do it as well. But the Twisted Int* protocol
should work fine for you.)
OBTW, the "16" is the number of bits in the message size
value; you should select a size large enough to hold your
largest message: "8" for up to 255 bytes, 16 allows for 65535
bytes and 32 allows something like 4MB.
The only two lines I had to change from my original code:
> from twisted.protocols.basic import Int16StringReceiver
> class MyProtocol(Int16StringReceiver):
[...]
Hope this helps.
--
John Santos
Evans Griffiths & Hart, Inc.
781-861-0670 ext 539
I spoke too fast. But pardon my noobiness.
Ok, so I am using a simple protocol that is listening on a TCP port.
One the client side, I write 4096 bytes using
self.transport.write(bytes)
on dataReceived side, I get only 1448.
Now, what I "want" to happen is when I issue a write of a known number
of bytes.
I "want" those bytes to arrive in total because they represent a pickled
object.
The server has no idea if the bytes are split and scattered (again, I
want the
control protocol to take affect).
1) Am I doing something wrong here?
2) Can I force twisted to send ALL the bytes I issue in the write
without
re-thinking TCP or forcing me to re-implement TCP?
thanks!
Darren
On Wed, 2010-02-10 at 18:34 -0500, Darren Govoni wrote:
> Thanks for that explanation David. Makes sense!
>
> On Wed, 2010-02-10 at 16:01 -0500, David Bolen wrote:
>
> > Darren Govoni <dar...@ontrenet.com> writes:
> >
> > >>From what I learned in other posts, the dataReceived(self, data): in the
> > > Echo server
> > > will get called with out-of-order data/bytes from the client. Of course,
> > > I could be misinformed,
> > > but what I understood before was that in this type of Protocol, I would
> > > have to re-order
> > > and re-assemble the bytes. And I'm trying to avoid that, since of
> > > course, TCP already does it.
> >
> > Data being received out of order can't happen, as long as the Protocol
> > is layered on top of TCP, since as you say, TCP already provides that
> > guarantee. The dataReceived() method is really just how the data
> > being received from TCP is handed to the Protocol object.
> >
> > UDP can be out of order, as it provides very few guarantees above and
> > beyond IP itself. But I'm not sure you can layer an IProtocol over
> > UDP with Twisted.
> >
> > There is a general issue where you may receive the data in differently
> > sized chunks in dataReceived() than it might have been transmitted
> > originally, which is a common source of confusion to people new to
> > stream protocols, so perhaps you were thinking of that issue?
> >
> > The stream nature (and possibility for early disconnect from the
> > client) is why having some internal length information for bulk
> > transfers is sensible. For your original question, I was going to
> > suggest an older posting of mine for a similar situation where I needed
> > a bulk upload to augment a PB-based server, but it appears that you've
> > located it in the archives yourself.
> >
> > > But like I said, I could have been misinformed because it seems pretty
> > > basic to write 1,2,3
> > > to a server and have it received 1,2,3, guaranteed.
> >
> > Yes - TCP guarantees that what you transmit at one end will be
> > received in order at the other end or not at all (e.g., outages,
> > disconnects, etc...). It has a weaker guarantee in terms of no
> > corruption, but one that is, in combination with typical link layer
> > protections, generally more than sufficient for the vast majority of
> > connections using it each day.
> >
> > -- David
> >
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
