Hello,
I just discovered that we are having problems with the length less
overloads in the embedded driver. Before I add any Jiras, I would like
some feedback from the community. There are for sure problems in
SQLBinary.readFromStream(). I would also appreciate if someone with
knowledge of the storage layer can tell me if we are facing trouble
there as well.
SQL layer
=========
SQLBinary.readFromStream()
1) The method does not support streaming.
It will either grow the buffer array to twice its size, or possibly
more if the available() method of the input stream returns a
non-zero value, until all data is read. This approach causes an
OutOfMemoryError if the stream data cannot fit into memory.
2) Might enter endless loop.
If the available() method of the input stream returns 0, and the
data in the stream is larger than the initial buffer array, an
endless loop will be entered. The problem is that the length
argument of read(byte[],int,int) is set to 0. We don't read any
more data and the stream is never exhausted.
To me, relying on available() to determine if the stream is exhausted
seems wrong. Also, subclasses of InputStream will return 0 if they don't
override the method.
I wrote a simple workaround for 2), but then the OutOfMemoryError
comes into play for large data.
Store layer
===========
I haven't had time to study the store layer, and know very little about
it. I hope somebody can give me some quick answers here.
3) Is it possible to stream directly to the store layer if you don't
know the length of the data?
Can meta information (page headers, record headers etc.) be updated
"as we go", or must the size be specified when the insert is
started?
I started looking at stuff in the store layer, but got a bit
overwhelmed, so instead of studying all the code, I'll post a few
questions and go to bed :)
I observed that inserts that go through the JDBC calls with length
arguments, go to BasePage.insertAllowOverflow if the column is
long. The application stream is finally read (through various wrapper
streams) from MemByteHolder.write(InputStream,long).
Can we get there without information about the data length?
I'll dig further another day, but as this is getting pretty
complex I'd like to get some help and pieces of advice from the experts.
Thanks,
--
Kristian
