Le 11/10/13 10:25 AM, Emmanuel Lécharny a écrit :
> Just quick heads up :
>
> Im tryinh to fix a bug that forbid us to inject a few thousands entries
> in the server (the injection test fails after aound 3000 thousands of
> entries added). The problem lies in teh way we serialize/deserialize
> ParentIdAndRdn. For some reason, we add the length of the byte[]
> containing the ParentIdAndRdn before the serialized value. The
> deserializer does not expect to find this length, thought the next
> position is not good. If I remove this length, now I have a pb because
> the number of expected RDNs is too big, and I get an
> ArrayOutofCoundException while tryingt to create an array of RDN[] with
> zillions of values.
>
> Obviously, we don't serialize/deserialize correctly, although the
> ser/deser methods have been tested and work. It's more about the way we
> call them : when we reach a number of stored elements, the cache is
> full, and we need to read back some pages from disk. This is when we
> have the issue. I have to understand why a Page deserialization fails.
> It should not be such a big issue, but the failure is quite random
> (something I don't understand, though... Not sure that the cache is not
> responsible for this randomness).
>
> I'll keep the list updated.
>
Ok, found the issue.
We have to store the length of a serialized key before the serialized
key, otherwise we have no way to know where to start the deserialization
of the next value. The code that deserialize a page is like :
// Read each value and key
for ( int i = 0; i < nbElems; i++ )
{
// Read the value (page offsets)
long offset = OFFSET_SERIALIZER.deserialize( byteBuffer );
long lastOffset = OFFSET_SERIALIZER.deserialize( byteBuffer );
...
// And the key
K key = btree.getKeySerializer().deserialize( byteBuffer );
}
Here, we deserialize the key passing a ByteBuffer, though the position
in this ByteBuffer should be changed after the deserialization.
This is correctly done in the MavibotParentIdAndRdnSerializer :
@Override
public ParentIdAndRdn deserialize( ByteBuffer buffer ) throws
IOException
{
int len = buffer.getInt();
ParentIdAndRdn parentIdAndRdn = fromBytes( buffer.array(),
buffer.position() );
buffer.position( buffer.position() + len );
return parentIdAndRdn;
}
Although in the same class, the serialization does not correctly set
this value :
public byte[] serialize( ParentIdAndRdn parentIdAndRdn )
{
try
{
int bufferSize = 1024;
while ( bufferSize < Integer.MAX_VALUE )
{
// allocate a big enough buffer for most of the cases
byte[] buffer = new byte[bufferSize];
try
{
// The current position. Start at 4, as we will add
the length at pos 0 <<<----------------- Here we are missing something
int pos = 0;
<----------------------------------------------------------------------------
It should be 4
...
byte[] result = new byte[pos];
<------------------------------------------------------------- We don't
store the byte[] length
System.arraycopy( buffer, 0, result, 0, pos );
return result;
So we just have to fix that.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
