On Wednesday 28 Aug 2002 2:19 pm, Jeremy Hylton wrote:
I'd like to change this bit of code in handle_write():
if n len(v):
# XXX It's unfortunate that we end up making many
# slices of a large string.
output.insert(0, v[n:])
break # we can't write any more
If we send a very large message, which could be caused by a big pickle
or just a transaction that touches a lot of objects, the message gets
sent in little chunks. The current implementation uses string slicing
to save the rest of the string, but that ends up making many copies of
the large string -- an O(n^2) proposition.
A possible solution is to store an index into the first string in
__output and just increment the index. The logic will be a bit tricky
to get right.
There is an O(n) option with easy logic... fragment big strings inside
message_output.
diff attached.
[hmmm. Its use of a python list as a fifo queue is O(n2) worst case, but I
doubt we will be affected by that in practice]
Index: smac.py
===
RCS file: /cvs-repository/Packages/ZEO/smac.py,v
retrieving revision 1.19
diff -c -4 -r1.19 smac.py
*** smac.py 22 Aug 2002 18:29:13 - 1.19
--- smac.py 29 Aug 2002 09:40:44 -
***
*** 133,140
--- 133,145
return 0
else:
return 1
+ # We chose 6 as the socket limit by looking at the
+ # largest strings that we could pass to send() without
+ # blocking.
+ _write_chunk_size = 6
+
def handle_write(self):
output = self.__output
while output:
# Accumulate output into a single string so that we avoid
***
*** 144,159
# unfortunate interaction between the Nagle algorithm and
# delayed acks. If we send a very large string, only a
# portion of it will actually be delivered at a time.
- # We chose 6 as the socket limit by looking at the
- # largest strings that we could pass to send() without
- # blocking.
-
l = 0
for i in range(len(output)):
l += len(output[i])
! if l 6:
break
i += 1
# It is very unlikely that i will be 1.
--- 149,160
# unfortunate interaction between the Nagle algorithm and
# delayed acks. If we send a very large string, only a
# portion of it will actually be delivered at a time.
l = 0
for i in range(len(output)):
l += len(output[i])
! if l = self._write_chunk_size:
break
i += 1
# It is very unlikely that i will be 1.
***
*** 189,198
This action is temporarily unavailable.
p
)
# do two separate appends to avoid copying the message string
! self.__output.append(struct.pack(i, len(message)))
! self.__output.append(message)
def close(self):
if self.__closed is None:
self.__closed = 1
--- 190,206
This action is temporarily unavailable.
p
)
# do two separate appends to avoid copying the message string
! l = len(message)
! self.__output.append(struct.pack(i, l))
! if l = self._write_chunk_size:
! self.__output.append(message)
! else:
! # fragment large strings now, to avoid O(n2) behavior when
! # dealing with huge strings.
! for start in range(0,l,self._write_chunk_size):
! self.__output.append(message[start:start+self._write_chunk_size])
def close(self):
if self.__closed is None:
self.__closed = 1
