Ok, pb fixed.
It was slightly more complex that I expected. The key length wasn't
stored for nodes, as shown in my previous mail, but the impact was a bit
more than just what I thought (ie, the fix was not what I suggested).
Le 11/11/13 8:14 AM, Emmanuel Lécharny a écrit :
> Le 11/10/13 10:25 AM, Emmanuel Lécharny a écrit :
>> Just quick heads up :
>>
>> Im tryinh to fix a bug that forbid us to inject a few thousands entries
>> in the server (the injection test fails after aound 3000 thousands of
>> entries added). The problem lies in teh way we serialize/deserialize
>> ParentIdAndRdn. For some reason, we add the length of the byte[]
>> containing the ParentIdAndRdn before the serialized value. The
>> deserializer does not expect to find this length, thought the next
>> position is not good. If I remove this length, now I have a pb because
>> the number of expected RDNs is too big, and I get an
>> ArrayOutofCoundException while tryingt to create an array of RDN[] with
>> zillions of values.
>>
>> Obviously, we don't serialize/deserialize correctly, although the
>> ser/deser methods have been tested and work. It's more about the way we
>> call them : when we reach a number of stored elements, the cache is
>> full, and we need to read back some pages from disk. This is when we
>> have the issue. I have to understand why a Page deserialization fails.
>> It should not be such a big issue, but the failure is quite random
>> (something I don't understand, though... Not sure that the cache is not
>> responsible for this randomness).
>>
>> I'll keep the list updated.
>>
> Ok, found the issue.
>
> We have to store the length of a serialized key before the serialized
> key, otherwise we have no way to know where to start the deserialization
> of the next value. The code that deserialize a page is like :
>
> // Read each value and key
> for ( int i = 0; i < nbElems; i++ )
> {
> // Read the value (page offsets)
> long offset = OFFSET_SERIALIZER.deserialize( byteBuffer );
> long lastOffset = OFFSET_SERIALIZER.deserialize( byteBuffer );
>
> ...
>
> // And the key
> K key = btree.getKeySerializer().deserialize( byteBuffer );
> }
>
> Here, we deserialize the key passing a ByteBuffer, though the position
> in this ByteBuffer should be changed after the deserialization.
>
> This is correctly done in the MavibotParentIdAndRdnSerializer :
>
> @Override
> public ParentIdAndRdn deserialize( ByteBuffer buffer ) throws
> IOException
> {
> int len = buffer.getInt();
>
> ParentIdAndRdn parentIdAndRdn = fromBytes( buffer.array(),
> buffer.position() );
>
> buffer.position( buffer.position() + len );
>
> return parentIdAndRdn;
> }
>
> Although in the same class, the serialization does not correctly set
> this value :
>
> public byte[] serialize( ParentIdAndRdn parentIdAndRdn )
> {
> try
> {
> int bufferSize = 1024;
>
> while ( bufferSize < Integer.MAX_VALUE )
> {
> // allocate a big enough buffer for most of the cases
> byte[] buffer = new byte[bufferSize];
>
> try
> {
> // The current position. Start at 4, as we will add
> the length at pos 0 <<<----------------- Here we are missing something
> int pos = 0;
> <----------------------------------------------------------------------------
> It should be 4
>
> ...
>
> byte[] result = new byte[pos];
> <------------------------------------------------------------- We don't
> store the byte[] length
> System.arraycopy( buffer, 0, result, 0, pos );
>
> return result;
>
> So we just have to fix that.
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
