Hi Godmar,
sorry for the delay, but I was on holidays last week, and away from my
mail.
Am Sam, 19 Aug 2000 schrieben Sie:
> From what I understand, and someone correct me if I'm wrong,
> there shouldn't be any reason not to include the change you suggest -
> if someone implements it, of course.
Done. I have a patched version of Encode.java. I 'll
clean it up when a definite solution has stabilized.
> If I understand your proposal right, you'd use an array for
> the first 256 values and a hashtable or something like that
> for the rest. I don't think there would be a problem with changing
> it so that it would both serialize an array and a hashtable.
> One or two objects in *.ser shouldn't make a difference.
Yes. It should work nicely for ISO-8859 based encodings, and
then for some.
Actually, for byte to char conversion you don't even
need a hash table, since all ISO-8859-X assign unicode chars
(simply speaking) to byte values in the range 0-255.
For the reverse way (char to byte conversion) I'd need to do some
experiments to figure out a better way. In most character to byte
encodings, there is no single range from character x to character y
where all characters are mapped from. So the array based approach is
space-inefficient. A combination of arrays and hasmaps might be
interesting. But for the time being, I'm playing around with
java.io.InputStreamReader, so I'm trying to fix byte to char conversion
first.
> You could even stick a flag at the beginning if the array shouldn't
> pay off for some encodings.
I'd prefer a more class hierarchy based approach. We already have
kaffe.io.ByteToCharHashBased. We could have ByteToCharArrayBased, too.
Something like this (warning: untested code ahead):
abstract public class ByteToCharArrayBased extends ByteToCharConverter {
// map is used to map characters from bytes to chars. A byte
// code b is mapped to character map[b & 0xFF].
private final char [] map;
public ByteToCharArrayBased ( char [] chars) {
map = chars;
}
public final int convert (byte[] from, int fpos, int flen,
char[] to, int tpos, int tlen) {
// Since it's a one to one encoding assume that
// flen == tlen.
for (int i = flen; i > 0; i --) {
to[ tpos++] = convert( from [ fpos++ ]);
}
return flen;
}
public final char convert (byte b) {
return map[b & 0xFF ];
}
public final int getNumberOfChars(byte [] from, int fpos, int
flen) {
return flen;
}
}
Now a (byte to char) conversion class has three choices:
a) it uses all byte values from 0-255 -> it extends
ByteToCharArrayBased, and makes the constructor use the
appropriate char array.
b) the encoded byte values are sparsely distributed through the range
of all legal byte values -> it extends ByteToCharHashBased due to
its space efficiency.
c) there is a huge block of bytes used in the encoding, but there are
also many bytes outside that block's range used in the encoding -> it
extends ByteToCharConverter and uses fields for ArrayBased as well
HashBased conversion. The convert method checks whether a byte is
within the block and uses the array, or uses the hash table otherwise.
>From my experience (converting ISO-8859-X encodings from being hash
based to being array based), for byte to char conversion option (a)
takes little memory (256 chars for the table) and is very fast. As I
explained in my previous post, it beats option (b) in time-efficiency.
I suppose it beats it in space-efficiency as well, as long as most
bytes are convertable into characters.
When there are only a few lagal byte values, which can be encoded into
characters, the hash based conversion could be more space-efficient. On
the other hand, the array based implementation doesn't waste much
memory in that case. Even in the worst case, a fictive encoding that
solely uses a specific single byte to encode some character, there are
255 * 2 = 510 bytes wasted. That's not much, and can be improved upon,
by introducing range checks and similar techniques.
The choices really start to matter when you're going the other way
round, from chars to bytes. Take a look at ISO-8859-8 (a.k.a. hebrew).
It encodes 220 characters. Of these 220, only 32 are *not*
mapping a byte value to itself. They are either mappings into the
range between \u05D0 and \u05EA, mappings within the first 256
characters, or mappings to a few special characters like LEFT-TO-RIGHT
MARK.
> One would have to see what the actual sizes of the
.ser files would be; > keeping those small is certainly desirable. From
what I understand, > they're more compact than any Java code
representation. > Edouard would know more since he wrote that code, I
think. > > > On a related note, this whole conversion thing stinks. >
Why can't people stick to 7-bit ASCII? > For instance, the JVM98 jack
benchmark calls PrintStream.print > a whopping 296218 times in a single
run. Every call results in a new > converter object being
newinstanced, just to convert a bunch of bytes. > (The new converter
was one of the changes done to make the > charset conversion
thread-safe.) This is one of the reasons > why we're on this test some
7 or 8 times slower than IBM. > And that's not even using any of the
serialized converters, just > the default one (which is written in
JNI). >
> - Godmar
>
> >
> >
> > Hi,
> >
> > I wrote a simple program to show a Java charmap (
> > something like Encode.java in developers directory).
> > It essentially creates a byte array with size 1, and
> > creates a string with the appropriate Unicode char
> > using the encoding in question for every value a byte
> > can take.
> >
> > When displaying a serialized converter like 8859_2,
> > the performance is very bad. Comparing current kaffe
> > from CVS running on SuSE Linux 6.4 with jit3 and IBM's
> > JRE 1.3 running in interpreted mode, kaffe is about 10
> > times slower.
> >
> > While I consider the idea to use serialized encoders
> > based on hashtables a great one, it is very
> > inefficient for ISO-8859-X and similar byte to char
> > encodings. These encodings use most of the 256
> > possible values a byte can take to encode characters,
> > so I tried using an array instead. I achieved
> > comparable running times to JRE 1.3.
> >
> > Why was the hashtable based conversion chosen over
> > alternatives (switch based lookup, array based
> > lookup)?
> >
> > Dali
> >
> > =====
> > "Success means never having to wear a suit"
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Send instant messages & get email alerts with Yahoo! Messenger.
> > http://im.yahoo.com/
> >
> -------------------------------------------------------
>
> _________________________________________________________
> Do You Yahoo!?
> Get your free @yahoo.com address at http://mail.yahoo.com
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
