I see. So in fact your code is quite possibly slower in non-ASCII cases? In fact, it sounds like having even one non-ASCII character would force extra copies to occur, which I would guess would defeat the benefit, but we'd need benchmarks to tell for sure.
On Fri, May 7, 2010 at 6:21 PM, Evan Jones <[email protected]> wrote: > On May 7, 2010, at 18:54 , Kenton Varda wrote: > >> I'd be very interested to hear why the JDK is not optimal here. >> > > I dug into this. I *think* the problem is that the JDK ends up allocating a > huge temporary array for the UTF-8 data. Hence, the garbage collection cost > is higher for the JDK's implementation, rather than my implementation. > Basically the code does this: > > > * allocate a new byte[] array that is string length * max bytes per > character ( = 4 for the UTF encoder) > * use the java.nio.charset.CharsetEncoder to encode the char[] into the > byte[] (wrapped in CharBuffer / ByteBuffer). > * copy the exact number of bytes out of the byte[] into a new byte[], and > return that. > > The only "trick" the JDK gets to use that "normal" Java code can't is that > they can access the string's char[] buffer directly, whereas I need to copy > it out into a char[] array. > > > Hence, I think what is happening is that the JDK allocates 4-5 times as > much memory per encode than I do. In the cases where the data is ASCII, my > code is faster, since it allocates exactly the right amount of space, and > doesn't need an extra copy. When the data is not ASCII, my code may still be > faster, since it doesn't overallocate quite as much (in exchange my code > does many copies). > > > Conclusion: there is a legitimate reason for this code to be faster than > the JDK's code. But it still may not be worth including this patch in the > main line protocol buffer code. > > > Evan > > -- > Evan Jones > http://evanjones.ca/ > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
