Re: Limits to byte[] size in RPC?
On Mon, Nov 17, 2008 at 11:15 AM, Axel <[EMAIL PROTECTED]> wrote:
> Here's the code that seems to be doing the serialization of a byte[]
> object, from
> com.google.gwt.user.client.rpc.core.java.lang.Byte_CustomFieldSerializer:
>
> public static void serialize(SerializationStreamWriter streamWriter,
> Byte instance) throws SerializationException {
>streamWriter.writeByte(instance.byteValue());
> }
Nope. That's the serialization code for a java.lang.Byte, which is
the wrapper type for bytes. byte[] (an array of primitive bytes) is
serialized somewhere else, although I'm not sure exactly where.
(Maybe in a class with Array in its name?)
> and then
> com.google.gwt.user.client.rpc.impl.AbstractSerializationStreamWriter:
>
> public void writeByte(byte fieldValue) {
>append(String.valueOf(fieldValue));
> }
>
> I take it that this creates a new String object for each byte in the
> byte[]. Furthermore, the encoding is a decimal numeric string,
> presumably prefixed with the type code used for the byte type during
> serialization which seems to be the class name ("java.lang.Byte").
The string "java.lang.Byte" should only be in the stream once and then
only if there's an instance of java.lang.Byte in the stream. Each
reference to the string is by index. I think, though, that a byte[]
is probably serialized as some kind of type identifier (it might be
"[" followed by the one-character shorthand for a primitive byte
followed by ";"), followed by the length of the array as a decimal
integer, followed by the elements of the array as decimal integers.
Ian
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google Web Toolkit" group.
To post to this group, send email to Google-Web-Toolkit@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/Google-Web-Toolkit?hl=en
-~--~~~~--~~--~--~---
Re: Limits to byte[] size in RPC?
Hi Axel,
I understand your frustration not being able to return a Route object
from a file upload servlet, but I cannot see why, having just slogged
through downloading a 1MB binary, you want to struggle through orders
of magnitude greater slog trying to upload it again to the client from
whence it came. I'm not surprised FF objects!
What I would try is to generate an XML string in my file upload
servlet to represent my Route object (easy using, say, dom4j) which
you can stream directly back to client via
HttpServletResponse .getOutputStream(). Then, back on client side...
public void onSubmitComplete(FormSubmitCompleteEvent event) {
// When the form submission is successfully completed, this
event is
// fired. Assuming the service returned a response of type
text/html,
// we can get the result text here
plug said text into the GWT XMLParser to produce a GWT Document from
which you can read your Route object properties. Job done.
Yes I know it's not as convenient and clean as RPC, but hey, when it
comes to HTTP and you're in doubt, sling a bit of XML about. It
doesn't have to conform to anyone's standards but yours.
regards
gregor
On Nov 17, 7:15 pm, Axel <[EMAIL PROTECTED]> wrote:
> Here's the code that seems to be doing the serialization of a byte[]
> object, from
> com.google.gwt.user.client.rpc.core.java.lang.Byte_CustomFieldSerializer:
>
> public static void serialize(SerializationStreamWriter streamWriter,
> Byte instance) throws SerializationException {
> streamWriter.writeByte(instance.byteValue());
> }
>
> and then
> com.google.gwt.user.client.rpc.impl.AbstractSerializationStreamWriter:
>
> public void writeByte(byte fieldValue) {
> append(String.valueOf(fieldValue));
> }
>
> I take it that this creates a new String object for each byte in the
> byte[]. Furthermore, the encoding is a decimal numeric string,
> presumably prefixed with the type code used for the byte type during
> serialization which seems to be the class name ("java.lang.Byte").
> This at least would explain some of the bloat happening. I'll follow
> Matt's advice and will file this as a bug report.
>
> Thanks for all the comments,
> -- Axel
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google Web Toolkit" group.
To post to this group, send email to Google-Web-Toolkit@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/Google-Web-Toolkit?hl=en
-~--~~~~--~~--~--~---
Re: Limits to byte[] size in RPC?
Here's the code that seems to be doing the serialization of a byte[]
object, from
com.google.gwt.user.client.rpc.core.java.lang.Byte_CustomFieldSerializer:
public static void serialize(SerializationStreamWriter streamWriter,
Byte instance) throws SerializationException {
streamWriter.writeByte(instance.byteValue());
}
and then
com.google.gwt.user.client.rpc.impl.AbstractSerializationStreamWriter:
public void writeByte(byte fieldValue) {
append(String.valueOf(fieldValue));
}
I take it that this creates a new String object for each byte in the
byte[]. Furthermore, the encoding is a decimal numeric string,
presumably prefixed with the type code used for the byte type during
serialization which seems to be the class name ("java.lang.Byte").
This at least would explain some of the bloat happening. I'll follow
Matt's advice and will file this as a bug report.
Thanks for all the comments,
-- Axel
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google Web Toolkit" group.
To post to this group, send email to Google-Web-Toolkit@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/Google-Web-Toolkit?hl=en
-~--~~~~--~~--~--~---
Re: Limits to byte[] size in RPC?
If you have time, please prepare a bug report. Regardless of the real- world use case, this should be addressed. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group. To post to this group, send email to Google-Web-Toolkit@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/Google-Web-Toolkit?hl=en -~--~~~~--~~--~--~---
Re: Limits to byte[] size in RPC?
> That's a good guess, but you have to remember one thing: Javascript's > only number type is equivalent to Java's double. It's been a while > since I've looked at the RPC internals, so things may have changed, > but it has been true that a byte[] would be converted into the > equivalent of a double[], and then serialized. Ouch, right, I faintly remember having read about these implicit conversions. > I'm curious about what you're doing with a byte[] on the client, Axel. I'm trying to write a file importer for GPS tracks where the user can use an upload form to submit the file. At the same time I'd really like to make use of a convenient RPC because the result will be a Route object. Now unfortunately, the form submit results are just a formatted HTML string returned by onSubmitComplete callback in the FormSubmitCompleteEvent. Since I'm not aware of any good way of assembling the Route object in a non-RPC servlet and serializing it through the servlet response so that I can easily deserialize it from the FormSubmitCompleteEvent, for the time I chose a rather inefficient but easy-to-write approach. I more or less "echo" the file and then have its contents in the client from where I can then submit them to an RPC doing the actual conversion, returning the Route object. It's expensive for the additional round-trip, but then again I wanted something working now that I may optimize later. I then encountered that the FormSubmitCompleteEvent contents are somehow altered by the browser. I therefore decided to Base64-encode and put it in between to force the browser to leave the Base64 string alone. I then originally decided to decode on the client into a byte[] which would be the Java abstraction of arbitrary file contents (think of binary GPS track formats) and then submit the byte [] to the RPC doing the actual track decoding. After the trouble with the byte[] I now just keep the Base64 string and submit that to the RPC which then does the Base64-decoding on the server. All I need to do on the client now is string the from the string. Although I like the clever trick with the double encoding you describe, I think I may stay with Base64 right now because it makes encoding/decoding more or less transparent to the client. Yes, it creates a hidden dependency between the echo servlet and the RPC implementation, but that's all on the server and I feel I can live with that for now. If someone can recommend a way to access the RPC serialization protocol as an API, I could perhaps use that in my echo servlet and do the conversion to a Route object in one round trip. I may still need to Base64-encode and wrap in to avoid distortion of the form submit results by the browser, but I could then Base64-decode and afterwards RPC-deserialize on the client. That, I think, would be a good solution because it would save the extra round trip. The byte[] problem, at last, seems to he the price we have to pay with Java vs. JavaScript incompatibilities. Still, better performance in the RPC implementation for byte[] (de-)serialization I find desirable. Best, -- Axel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group. To post to this group, send email to Google-Web-Toolkit@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/Google-Web-Toolkit?hl=en -~--~~~~--~~--~--~---
Re: Limits to byte[] size in RPC?
On Sun, Nov 16, 2008 at 12:01 PM, Shawn Pearce <[EMAIL PROTECTED]> wrote:
> My initial guess is that the array serialization during RPC is packing each
> byte into its own element in a JSON array. So your 400kB byte array turns
> into a 409,600 element integer array, which is probably using at least 16
> bytes per entry, resulting in a fairly large (several megabyte) footprint in
> Firefox. Doesn't explain why FF grabs several GB for this, but I think you
> are using a lot more memory than you expect.
That's a good guess, but you have to remember one thing: Javascript's
only number type is equivalent to Java's double. It's been a while
since I've looked at the RPC internals, so things may have changed,
but it has been true that a byte[] would be converted into the
equivalent of a double[], and then serialized.
I'm curious about what you're doing with a byte[] on the client, Axel.
Besides filing an issue to see if the GWT developers want to support
your use case, I'd suggest the following: try sending a double[]
instead of a byte[] by packing 6 bytes into each double. Code like
the following might work.
public static double[] bytesToDoubles(byte[] bytes) {
double[] ret = new double[(bytes.length + 5) / 6];
for (int i = 0, n = r.length; i < n; ++i) {
ret[i] = bytesToDouble(bytes, i * 6);
}
return ret;
}
private static double bytesToDouble(byte[] bytes, int offset) {
if (offset + 5 < bytes.length) {
double highTwo = 4294967296.0 * (bytes[0] << 8 + bytes[1]);
int lowFour = bytes[2] << 24 + bytes[3] << 16 + bytes[4] << 8 + bytes[5];
return highTwo + lowFour;
}
double highTwo = 4294967296.0 * (safeGet(bytes, 0) << 8 + safeGet(bytes, 1));
int lowFour = safeGet(bytes, 2) << 24 + safeGet(bytes, 3) << 16 +
safeGet(bytes, 4) << 8 + safeGet(bytes, 5);
return highTwo + lowFour;
}
private static byte safeGet(byte[] bytes, int index) {
return (index >= bytes.length ? 0 : bytes[index]);
}
I'm in a rush, so I'll leave the unpacking code as an "exercise for
the reader". Note also that, since Java's String is a counted data
type, it might be safe to pack bytes into a String without base64
encoding them first, which would save on bytes, too.
Ian
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google Web Toolkit" group.
To post to this group, send email to Google-Web-Toolkit@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/Google-Web-Toolkit?hl=en
-~--~~~~--~~--~--~---
Re: Limits to byte[] size in RPC?
My initial guess is that the array serialization during RPC is packing each byte into its own element in a JSON array. So your 400kB byte array turns into a 409,600 element integer array, which is probably using at least 16 bytes per entry, resulting in a fairly large (several megabyte) footprint in Firefox. Doesn't explain why FF grabs several GB for this, but I think you are using a lot more memory than you expect. On Sun, Nov 16, 2008 at 04:21, Axel <[EMAIL PROTECTED]> wrote: > > Hi, > > I've been trying to handle binary data with one or more MB in size, > passing it through RPC. In Firefox (tested with 3.0.4) I can reproduce > what to me seems like a memory leak. No matter how I assemble the byte > [] contents: as soon as the size approaches or exceeds ~500kB, Firefox > allocates >1GB of memory, bringing my 2GB client machine to swap so > heavily I have to kill FF. > > I tested with different byte[] sizes. 400kB seems to be ok, regardless > the contents. 500kB rarely works and 600kB I never managed to get > through. Assembling the byte[] before actually calling the RPC works > fine. It is the RPC call itself where memory consumption in FF goes up > so drastically. > > My workaround for now is to Base64-encode the byte[] into a String and > send that along, decoding it on the server again. This lets me suspect > that something may be wrong with the byte[] serialization code. > > In the hosted Java environment this problem does not occur at all, and > I can flawlessly send byte[]s with several MB in size no problem. > > Best, > -- Axel > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group. To post to this group, send email to Google-Web-Toolkit@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/Google-Web-Toolkit?hl=en -~--~~~~--~~--~--~---
Limits to byte[] size in RPC?
Hi, I've been trying to handle binary data with one or more MB in size, passing it through RPC. In Firefox (tested with 3.0.4) I can reproduce what to me seems like a memory leak. No matter how I assemble the byte [] contents: as soon as the size approaches or exceeds ~500kB, Firefox allocates >1GB of memory, bringing my 2GB client machine to swap so heavily I have to kill FF. I tested with different byte[] sizes. 400kB seems to be ok, regardless the contents. 500kB rarely works and 600kB I never managed to get through. Assembling the byte[] before actually calling the RPC works fine. It is the RPC call itself where memory consumption in FF goes up so drastically. My workaround for now is to Base64-encode the byte[] into a String and send that along, decoding it on the server again. This lets me suspect that something may be wrong with the byte[] serialization code. In the hosted Java environment this problem does not occur at all, and I can flawlessly send byte[]s with several MB in size no problem. Best, -- Axel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group. To post to this group, send email to Google-Web-Toolkit@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/Google-Web-Toolkit?hl=en -~--~~~~--~~--~--~---
