[
https://issues.apache.org/jira/browse/DERBY-867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dag H. Wanvik updated DERBY-867:
--------------------------------
Derby Categories: [Performance]
> Performance issues with large object handling in DDMWriter
> ----------------------------------------------------------
>
> Key: DERBY-867
> URL: https://issues.apache.org/jira/browse/DERBY-867
> Project: Derby
> Issue Type: Improvement
> Components: Network Server
> Affects Versions: 10.1.1.0, 10.2.1.6
> Reporter: Bryan Pendleton
> Priority: Minor
>
> There are some performance issues related to the way that DDMWriter handles
> extremely large network messages, such as those that arise in DERBY-125 and
> DERBY-492. The performance problems involve both memory usage and CPU cycles.
> Among the problems that could be investigated and improved:
> 1) The DDMWriter internal byte array grows, but never shrinks (except when
> the DDMWriter as a whole is collected). This means that there may be some
> memory wastage when the buffer grows to a large value, then is later used for
> only small buffers.
> 2) When the buffer grows, it tends to double in size. When you start gettting
> to moderately large buffer sizes, this may mean that the buffer over-expands
> during a growth event, and grows to a larger size than is needed, again
> leading to memory wastage.
> 3) I'm not sure what would happen if memory were unavailable when the buffer
> wanted to grow; I think it would be a very hard error. There may be ways to
> handle this better, such as: waiting a while to see if memory becomes
> available; writing previously chained messages out of the buffer to free up
> space, etc. At the very least, it would be nice if an out-of-memory problem
> were handled cleanly, for example by terminating this connection but leaving
> the rest of the Network Server online and available for other users.
> 4) Perhaps we could eliminate the expansion all together by instead of
> expanding have a pool of buffers available to all the DDMWriters. If a
> DDMWriter needs to write more, it picks up a new buffer and returns it to
> the pool when no longer needed.
> 5) Large DDM objects which require Layer B segmentation are written as a
> single object, then later the segmentation is added after the payload data is
> complete. This processing is handled by endDdm() and finalizeDssLength().
> This code currently copies the bytes around in the buffer two "extra" times:
> once to insert the extended length bytes at the front of the large DDM
> object, and a second time to insert the DSS Continuation Headers every 32K
> bytes. Given the API between DDMWriter and its callers, it may be impossible
> to totally remove this memory copying, but it does seem like we could reduce
> from 2 full copies to 1, by combining the insertion of the extended length
> bytes with the continuation header segmentation. It might also be possible to
> detect a large object once it spills out from one DSS block into the second
> block, and hence avoid copying for subsequent blocks, even if those first two
> blocks do have to be copied once.
> There may be other opportunities in this part of the code; this issue is just
> a collection of the ideas that came up while the developers were discussing
> bugs DERBY-125, DERBY-170, DERBY-491, and DERBY-492. A review of the
> processing that occurs in those test cases should be performed as part of
> working on this bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
