Hi all,
Posting it with more elaborately.
Also if someone deletes a Property from Class. It would need to delete the
property i.e. field from all the records in the database for that class.
If we keep mapping as suggested above, the deleted property can be removed
straight-away from the mapping (integer identifier and property name) and
the actual deletion of the property/field from all the records for that
class can happen in the background with some cleanup/defrag tool.
Regards,
Ameer
On Thursday, 20 February 2014 19:22:54 UTC+5:30, Ameer Tamboli wrote:
>
> Hi all,
>
> I am not sure whether this is already implemented/planned.
>
> Also if someone deletes a Property from Class, the field can be removed
> from the field names array. Actual deletion of the field can happen in the
> background with some cleanup/defrag tool.
>
> Regards,
> Ameer
>
> On Thursday, 20 February 2014 17:54:31 UTC+5:30, Steve Coughlan wrote:
>>
>> Hi Andrey,
>>
>> I forked orient-core today and spent most of the day playing around with
>> the source trying to work out how to change over my pseudo schema,
>> property, type classes into OSchema, OProperty, OType.
>> ORecordSerializerDocument2Binary was very useful for understanding things.
>> Is it actually in use? I can't find any references to it.
>>
>> Could you explain *"We have many third party drivers for binary
>> protocol"* a bit more? Are there any examples?
>>
>> I also have a question about ORID and whether it can be considered fixed
>> length. It contains OClusterPosition which has two implementations. One
>> is 8 bytes long and the other is 24 bytes long. For the purposes of
>> serialization we can't consider the ORID to be fixed length unless we
>> guaruntee that every instance of ORID within a DB is only one of these
>> implementations. Is this the case?
>>
>> At the moment I'm also wrestling with what to do about null fixed length
>> fields and whether to reserves space inside a record. Whilst headers are
>> ordered by schema_fixed_length, schema_variable_length, schema_less fields
>> there's no reason data needs to follow the same order. But by default it
>> probably would. Consider an object schema like this:
>> class SomeClass {
>> update_time: DateTime //fixed length
>> short_string: String
>> massive_string: String
>> }
>>
>> If we first write the record and update_time is null we'd have something
>> like this
>> update_time:0 bytes|short_string: 10 bytes|massive_string:100kbytes
>>
>> Then we update it to add update_time we have a few options.
>> 1/ When originally writing the object reserve space even though the value
>> is null (wasted space)
>> 2/ Search for a hole. e.g. if short_string has been set to null we could
>> steal it's space.
>> 3/ Write the update_time field after massive_string (If there is space
>> before the beginning of the next record). Potentially we are writing into a
>> different disk block so for future reads when we aren't interested in
>> massive_string we still have to load the block into memory)
>> 4/ Rewrite the entire record.
>>
>> I suppose it is worth considering whether there's a benefit to reserving
>> partial holes. i.e. if we have 10 * 4 byte nullable fixed length fields
>> (all null on initial write) should we take a guess and reserve say 10 out
>> of the 40 possible bytes for future updates? But I'm probably getting
>> ahead of myself. I'll work on a simple implementation first before trying
>> to be too clever ;)
>>
>>
>> On 20/02/14 20:12, Andrey Lomakin wrote:
>>
>> Hi Steve,
>> Good that you are going to help us.
>> Few additional information:
>> 1. We already have binary serialization support you can see it here
>> com.orientechnologies.common.serialization.types.OBinarySerializer so
>> obviously we should not have several version of the same. Also I think it
>> will be interesting for you to look at this issue and discussion here
>> https://github.com/orientechnologies/orientdb/issues/681#issuecomment-28466948.
>> We discussed serialization of single record (sorry had no time to analyze
>> it deeply because a lot of events) but in case of SQL query you have to
>> process millions of them.
>> 2. We are working on binary compatibility mechanics too (I mean
>> compatibility between storage formats), without it current users will not
>> be able to accomplish new features especially binary serialization.
>> 3. We have many third party drivers for binary protocol (which pass
>> serialized records on client;s side) so we have to think how to not break
>> functionality of this drivers.
>>
>>
>>
>> On Wed, Feb 19, 2014 at 1:53 PM, Steve <[email protected]> wrote:
>>
>>> Hi Luca,
>>>
>>> I'll give it a go with the real ODB code. The reason I didn't is
>>> because I'm actually quite new to ODB even as an end user but your
>>> instructions will set me in the right direction. Most of my experience
>>> with data serialization formats has been with Bitcoin which was mostly for
>>> network protocol use cases rather than big-data storage. But that was also
>>> a high performance scenario so I guess there are a lot of parallels.
>>>
>>>
>>> On 19/02/14 21:33, Luca Garulli wrote:
>>>
>>> Hi Steve,
>>> sorry for such delay.
>>>
>>> I like your ideas, I think this is the right direction. varint8 e
>>> varint16 could be a good way to save space, but we should consider when
>>> this slows down some use cases, like partial field loading.
>>>
>>> About the POC you created I think it would be much more useful if you
>>> play with real documents. It's easy and you could push it to a separate
>>> branch to let to us and other developers to contribute & test. WDYT?
>>>
>>> Follow these steps:
>>>
>>> (1) create your serializer
>>>
>>> This is the skeleton of the class to implement:
>>>
>>> public class BinaryDocumentSerializer implements ORecordSerializer {
>>> public static final String NAME = "binarydoc";
>>>
>>> // UN-MARSHALLING
>>> public ORecordInternal<?> fromStream(final byte[] iSource) {
>>> }
>>>
>>> // PARTIAL UN-MARSHALLING
>>> public ORecordInternal<?> fromStream(final byte[] iSource, final
>>> ORecordInternal<?> iRecord, String[] iFields) {
>>> }
>>>
>>> // MARSHALLING
>>> public byte[] toStream(final ORecordInternal<?> iSource, boolean
>>> iOnlyDelta) {
>>> }
>>> }
>>>
>>> (2) register your implementation
>>>
>>>
>>> ORecordSerializerFactory.instance().register(BinaryDocumentSerializer.NAME,
>>> new BinaryDocumentSerializer());
>>>
>>> (3) create a new ODocument subclass
>>>
>>> Then create a new class that extends ODocument but uses your
>>> implementation:
>>>
>>> public class BinaryDocument extends ODocument {
>>> protected void setup() {
>>> super.setup();
>>> _recordFormat =
>>> ORecordSerializerFactory.instance().getFormat(BinaryDocumentSerializer.NAME);
>>> }
>>> }
>>>
>>> (4) Try it!
>>>
>>> And now try to create a BinaryDocument, set fields and call .save().
>>> The method BinaryDocumentSerializer.toStream() will be called.
>>>
>>>
>>>
>>> Lvc@
>>>
>>>
>>>
>>> On 18 February 2014 06:08, Steve <[email protected]> wrote:
>>>
>>>>
>>>> The point is: why should I store the field name when I've declared
>>>> that a class has such names?
>>>>
>>>>
>>>> Precisely. But I don't think you need to limit it to the declarative
>>>> case... i.e. schema-full. By using a numbered field_id you cover
>>>> schema-full, schema-mixed and schema-free cases with a single solution.
>>>> There are two issues here... Performance and storage space. Arguably
>>>> improving storage space also improves performance in a bigdata context
>>>> because it allows caches to retain more logical units in memory.
>>>>
>>>>
>>>> I've been having a good think about this and I think I've come up with
>>>> a viable plan that solves a few problems. It requires schema versioning.
>>>>
>>>> I was hesitant to make this suggestion as it introduces more complexity
>>>> in order to improve compactness and unnecessary reading of metadata.
>>>> However I see from you original proposal that the problem exists there as
>>>> well.:
>>>>
>>>> *Cons:*
>>>>
>>>> - *Every time the schema changes, a full scan and update of record
>>>> is needed*
>>>>
>>>> The proposal is that record metadata is made of 3 parts + a meta-header
>>>> (which in most cases would be 2-3 bytes. Fixed length schema declared
>>>> fields, variable length schema declared fields and schema-less fields.
>>>> The
>>>> problem as you point out with a single schema per class is that if you
>>>> change the schema you have to update every record. If you insert a field
>>>> before the last field you would likely have to rewrite every record from
>>>> scratch.
>>>>
>>>> First a couple of definitions:
>>>>
>>>> Definitions:
>>>>
>>>> varint8: a standard varint that is built from any number of 1 byte
>>>> segments. The first bit of each segment is set to 1 if there is a
>>>> subsequent segment. A number is constructed by concatenating the last 7
>>>> bits of each byte. This allows for the following value ranges:
>>>> 1 byte : 127
>>>> 2 bytes: 16k
>>>> 3 bytes: 2m
>>>> 4 bytes: 268m
>>>>
>>>> varint16: same as varint8 but the first segment is 16 bits and all
>>>> subsequent are 8 bits
>>>> 2 bytes: 32k
>>>> 3 bytes: 4m
>>>> 4 bytes: 536m
>>>>
>>>> nameId: an int (or long) index from a field name array. This index
>>>> could be one per JVM or one per class. Getting the field name using the
>>>> nameId is a single array lookup. This is stored on disk as a varint16
>>>> allowing 32k names before we need to use a 3rd byte for name storage.
>>>>
>>>> I propose a record header that looks like this:
>>>>
>>>> version:varint8|header_length:varint8|variable_length_declared_field_headers|undeclared_field_headers
>>>>
>>>> Version is the schema version and would in most cases be only 1 byte.
>>>> You would need 128 schema changes to make it 2 bytes. This proposal would
>>>> require a cleanup tool that could scan all record and reset them all to
>>>> most recent schema version (at which point version is reset to 0). But it
>>>> would be necessary on every schema change. The user could choose if and
>>>> when to run it. The only time you would need to do a full scan would be
>>>> if
>>>> you are introducing some sort of constraint and needed to validate that
>>>> existing records don't violate the constraint.
>>>>
>>>> When a new schema is generated the user defined order of fields is
>>>> stored in each field's Schema entry. Internally the fields are rearranged
>>>> so that all fixed length fields come first. Because the order and length
>>>> of fields is known by the schema there is no need to store offset/length
>>>> in
>>>> the record header.
>>>>
>>>> Variable length declared fields need only a length and offset and the
>>>> rest of the field meta data is determined by the schema.
>>>>
>>>> Finally undeclared (schema-less) fields require additional header data:
>>>> nameId:varint16|dataType:byte?|offset:varint8|length:varint8
>>>>
>>>> I've attached a very rough partial implementation to try and
>>>> demonstrate the concept. It won't run because a number of low level
>>>> functions aren't implemented but if you start at the Record class you
>>>> should be able to follow the code through from the read(int nameId)
>>>> method. It demonstrates how you would read a schema/fixed,
>>>> schema/variable
>>>> and non-schema field from the record using random access.
>>>>
>>>> I think I've made one significant mistake in demo code. I've used
>>>> varints to store offset/length for schema-variable-length fields. This
>>>> means you cannot find the header for one of those field without scanning
>>>> that entire section of the header. The same is true for schema-less
>>>> however in this case it doesn't matter since we don't know what fields are
>>>> there (or the order) from the schema we have no option but to scan that
>>>> part of the header to find the field metadata we are looking for.
>>>>
>>>> The advantage though of storing length as a varint is that perhaps in a
>>>> majority of cases field length is going to be less than 127 bytes which
>>>> means you can store it in a single byte rather than 4 or 8 for an int or
>>>> long.
>>>>
>>>> We have a couple of potential tradeoffs to consider here (only relavent
>>>> to the schema declared variable length fields). By doing a full scan of
>>>> the header we can use varints with impunity and can gain storage benefits
>>>> from it. We can also dispense with storing the offset field altogether as
>>>> it can be calculated during the header scan. So potentially reducing the
>>>> header entry for each field from 8 bytes (if you use int) to as little as
>>>> 1. Also we remove a potential constraint on maximum field length. On the
>>>> other hand if we use fixed length fields (like int or long) to store
>>>> offset/length we gain random access in the header.
>>>>
>>>> I can see two edge cases where this sort of scheme would run into
>>>> difficulties or potentially create a storage penalty. 1) a dataset that
>>>> has a vast number of different fields. Perhaps where the user is for some
>>>> reason using the field name as a kind of meta-data which would increase
>>>> the
>>>> in-memory field_name table and 2) Where a user has adopted the (rather
>>>> hideous) mongoDB solution of abbreviating field names and taken it to the
>>>> extreme of a single character field name. In this case my proposed 16 bit
>>>> minimum nameIndex size would be 8 bits over what could be achieved.
>>>>
>>>> The first issue could be dealt with by only by making the tokenised
>>>> field name feature available only in the case where the field is declared
>>>> in schema (basically your proposal). But would also require a flag on
>>>> internally stored field_name token to indicate if it's a schema token or
>>>> schema-less full field name. It could be mitigated by giving an option
>>>> for
>>>> full field_name storage (I would imagine this would be a rare use case).
>>>>
>>>> The second issue (if deemed important enough to address) could also be
>>>> be dealt with by a separate implementation of something like
>>>> IFieldNameDecoder that uses an 8 bit segment and asking the user to
>>>> declare
>>>> a cluster/class as using that if they have a use case for it.
>>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "OrientDB" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "OrientDB" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey Lomakin.
>>
>> Orient Technologies
>> the Company behind OrientDB
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "OrientDB" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
--
---
You received this message because you are subscribed to the Google Groups
"OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
