Hello all,
it's 2 months since my last message, but finally we have got to be able to conduct new load tests in production (customer's machines were too busy in this period). Results are good as expected and customer is satisfied (view it at fixed width, ramp-up 200ms):
n.thread Average Min Max Error Rate Client Max heap
5 3692 2556 4505 0.00% 1.1/sec 512Mb
5 4929 4819 5107 0.00% 58.7/min 512Mb
10 6439 6117 6738 0.00% 1.5/sec 512Mb
10 6026 5381 6734 0.00% 1.5/sec 512Mb
20 11017 9175 13134 0.00% 1.5/sec 512Mb
20 9657 6915 11465 0.00% 1.7/sec 512Mb
30 12004 7670 15437 0.00% 1.6/sec 512Mb
30 12815 9539 16167 0.00% 1.7/sec 512Mb
40 18497 11049 25532 0.00% 1.5/sec 512Mb
40 14276 9369 18820 0.00% 1.7/sec 512Mb
50 18711 10665 26607 0.00% 1.6/sec 512Mb
50 18896 9441 25013 0.00% 1.6/sec 512Mb
Comparing these results (custom buffer implementation) with byte[] parameter shows an improvement in performance between 3 and 5 times, while comparing with unbuffered attachments is between 7 and 12 times. You should also note that the unbuffered version is in reality buffered: when asked for returning a stream to read attachments, axis implements a stream that is basically buffered, but implementing a custom buffer shows an order of magnitude of improvement.
Ciao, Giuliano
Piergiuliano Bossi wrote:
After some more thoughts it looks that we are not doing buffered I/O while reading the attachment from disk. As suggested in many places (for example http://java.sun.com/docs/books/performance/1st_edition/html/JPIOPerformance.fm.html) I/O operations in java are by default unbuffered.
Shame on us! :(
Preliminary measures taken on development workstation show an order of magnitude of improvement.
Hopefully tomorrow we will get better results on servers and under load as well.
Cheers, Giuliano
Piergiuliano Bossi wrote:
Hello all,
one year ago we have developed a web service that has a few parameters, one of which is a byte[]. At that time this parameter was used to get a binary file (for example midi or gif) with size up to 150 KB (max). We have had excellent results, both in terms of reliability and speed.
Customer's needs have grown since then and today we are working with files that can be 4 MB in size. Of course, we have looked at attachments for this (check thread http://marc.theaimsgroup.com/?l=axis-user&m=109707110706202&w=2).
Now our web service looks for the binary file in the byte array parameter first, and then into attachments. The idea is that if the file is small (ie: 100 KB) then using the parameter is ok, otherwise the attachment got to be used.
The problem is that we observe unexpected performance in production. We have built some jmeter load tests, trying to send 4 MB content with or without using attachments, 5 or 10 concurrent client threads (no ramp-up period) and we get the following results (please watch it at fixed width):
thread Average Min Max Error Attachments
5 14889 13373 16205 0.00% no
5 32170 31282 33438 0.00% yes
10 31673 31547 31857 0.00% no
10 45556 41563 51975 0.00% yes
These measures are "round-trip", taken from client-side.
As you can see it seems that without attachments performances are better! This contradicts everything we have read so far and even logic too.
Please, note that production architecture is structured in this way:
*) there are 2 servers, each with Red Hat Linux rel 8.0 (kernel 2.4.18-14smp), 4 microproc. Intel Xeon 550 MHz, 1 GB RAM
*) each server is using a 1.4.2_06 jvm, tomcat 4.1.30 and axis 1.1
*) in front of the servers there is an Alteon balancer
Load tests were performed using a fast client machine, a Sun Solaris 220, double-processor with 2 GB RAM, therefore we expect that the problem doesn't lie on client side.
Before implementing attachments we have profiled a single request with a 4 MB file (passed as a byte[]) to our web service (with OptimizeIT) and we have discovered that our business logic was consuming 19.2% of cpu time: the remining part was consumed by axis in various ways, for example retrieving the Envelope (org.apache.axis.Message.getSOAPEnvelope) and getting parameters in java objects (org.apache.axis.message.RPCElement.getParams)
We have not profiled current implementation with attachments yet, but we will do it soon. Let's see if there will be any surprise.
Real problem is that we should be able to have a round-trip max period well under 10 seconds under more than 10 concurrent threads on client side.
What do you think about it? How can we explain those numbers? Is there any way to improve significantly these performance results?
We are considering passing the file over the filesystem, getting path only into the web service, but this is a last chance that we prefer to avoid, because it assumes that client and server share a filesystem somewhere.
TIA Giuliano
PS: a couple of snippets of code (client side is used in load tests)
********************CUT HERE - CLIENT SIDE********************
/**This class should store all attachment data in memory */
static class MemoryOnlyDataSource extends org.apache.axis.attachments.ManagedMemoryDataSource{
MemoryOnlyDataSource( byte [] in, String contentType) throws java.io.IOException{
super( new java.io.ByteArrayInputStream(in) , Integer.MAX_VALUE -2, contentType, true);
}
MemoryOnlyDataSource( String in, String contentType)throws java.io.IOException{
this( in.getBytes() , contentType);
}
}
service._setProperty(Call.ATTACHMENT_ENCAPSULATION_FORMAT, Call.ATTACHMENT_ENCAPSULATION_FORMAT_DIME); service.setTimeout(60*60*1000);
DataHandler attachment = new DataHandler(new MemoryOnlyDataSource(aContentAsAttachment, mimeType));
service.addAttachment(attachment);
byte[] contentData = null; // if null, then attachments are used
service.insertContent(param1, param2, param3, param4, param5, contentData, param6, param7, param8);
********************CUT HERE - CLIENT SIDE********************
********************CUT HERE - SERVER SIDE********************
public int insertContent(String param1, String param2, String param3, String param4, String param5, byte[] contentData, int param6, boolean param7, StringHolder param8)
{
if (contentData == null)
{
try
{
AttachmentPart messageAttachment = getmessageAttachment();
if (messageAttachment != null)
contentData = toByteArray(messageAttachment);
}
catch (Exception e)
{
// ...
}
}
// ... business logic here ...
}
private AttachmentPart getmessageAttachment() throws AxisFault { MessageContext msgContext = MessageContext.getCurrentContext(); Message reqMsg = msgContext.getRequestmessage(); Attachments allAttachments = reqMsg.getAttachmentsImpl();
AttachmentPart attachment = null;
if (allAttachments == null)
// ...
else if (allAttachments.getAttachmentCount() > 0)
attachment = (AttachmentPart)allAttachments.getAttachments().iterator().next();
return attachment;
} private byte[] toByteArray(AttachmentPart attachment) throws SOAPException, IOException
{
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(attachment.getSize());
InputStream inputStream = (InputStream) attachment.getContent();
int currentByte = -1;
while ((currentByte = inputStream.read()) != -1)
outputStream.write(currentByte);
inputStream.close(); outputStream.close();
return outputStream.toByteArray(); } ********************CUT HERE - SERVER SIDE********************
