I know, but that's hardly a specification - not enough to reference for an implementation.
For now, I will try to port your implementation... On Fri, Dec 12, 2014 at 12:49 PM, GoorMoon <[email protected]> wrote: > > You can look hear > https://github.com/orientechnologies/orientdb/wiki/Record-Schemaless-Binary-Serialization#shortintegerlong > > > On Friday, December 12, 2014 1:44:01 PM UTC+2, mindplay.dk wrote: >> >> Glad to hear that, thanks :-) >> >> So on a related note - the "varint" type used in the OrientDB binary >> protocol, what specification does it follow precisely? Because apparently >> there are lots >> <http://vpri.org/fonc_wiki/index.php/Variable_Length_Integer> of ways to >> encode a variable-size integer. >> >> I sort of wish there was an option for the client to disable >> variable-length integers in the protocol, instead encoding them with a >> fixed size. >> >> I can implement UTF-8 style reading/writing of variable-size integers in >> PHP, but this is going to add considerable CPU overhead - in the case of a >> PHP client (probably other scripting languages too) a small amount of >> bandwidth overhead is likely preferable to CPU overhead. What we want is a >> fast client - whether that means using a little more bandwidth is probably >> secondary, as is the ability to support more than 2 billion records for >> most projects. >> >> Just putting that out there :-) >> >> But for the time being, can you point me to a specification or (better) a >> reference implementation (in any language) of the VLI encoding used by >> OrientDB? >> >> I can reference the one in GoorMoon's .NET driver >> <https://github.com/orientechnologies/OrientDB-NET.binary/blob/binary.serialization/src/Orient/Orient.Client/Protocol/Serializers/RecordBinarySerializer.cs#L502>, >> or the one on wikipedia <http://en.wikipedia.org/wiki/UTF-8>, but >> neither of them appear to have tests, and I'm unsure how to test them. Go >> has a nice implementation >> <http://golang.org/src/encoding/binary/varint.go> with tests >> <http://golang.org/src/encoding/binary/varint_test.go>, but since there >> are so many types of VLI which don't appear to have any official names or >> standardization, I can't be sure it's the same type of encoding... >> >> >> On Fri, Dec 12, 2014 at 11:45 AM, Luca Garulli <[email protected]> wrote: >>> >>> Authors of drivers have such kind of high priority on requests :-) >>> >>> Lvc@ >>> >>> >>> On 11 December 2014 at 23:29, GoorMoon <[email protected]> wrote: >>> >>>> Glad to hear !!! >>>> >>>> On Thursday, December 11, 2014 11:41:05 PM UTC+2, mindplay.dk wrote: >>>>> >>>>> Never mind, spotted it :-) >>>>> >>>>> >>>>> On Thu, Dec 11, 2014 at 10:03 PM, Rasmus Schultz <[email protected]> >>>>> wrote: >>>>> >>>>>> That was a fast decision - very happy to see them reacting to this >>>>>> issue so quickly and slating it for the 2.0 release! :-) >>>>>> >>>>>> So I'm referencing a bunch of your code for my client now, and I'm >>>>>> hung up on a small issue, maybe you can point me in the right >>>>>> direction... >>>>>> >>>>>> The so-called "varint" type in the binary serialization - I >>>>>> understand it's a variable-size integer, encoded like UTF-8 character >>>>>> codes. Where or how do you handle this in your implementation? >>>>>> >>>>>> >>>>>> On Thu, Dec 11, 2014 at 7:31 PM, Rasmus Schultz <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Posted here >>>>>>> >>>>>>> https://github.com/orientechnologies/orientdb/issues/3175 >>>>>>> >>>>>>> >>>>>>> On Thu, Dec 11, 2014 at 7:10 PM, GoorMoon <[email protected]> wrote: >>>>>>> >>>>>>>> I am agree with you, but this not our decision. >>>>>>>> I suggest you to open issue here https://github.com/orientechno >>>>>>>> logies/orientdb that describe your purpose. >>>>>>>> >>>>>>>> >>>>>>>> On Thursday, December 11, 2014 10:55:46 AM UTC+2, mindplay.dk >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> This looks great, thanks! So much simpler than the CSV serializer. >>>>>>>>> >>>>>>>>> I see that you do have to buffer the record in memory still, as I >>>>>>>>> suspected. I really do wish they would make the change I suggested >>>>>>>>> below, >>>>>>>>> putting the record format before the actual record - I think then you >>>>>>>>> wouldn't need to buffer records in memory before you can deserialize. >>>>>>>>> Thoughts? >>>>>>>>> On Dec 10, 2014 1:09 PM, "GoorMoon" <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hey, >>>>>>>>>> I don't know about PHP driver but, i contribute to .NET Driver >>>>>>>>>> https://github.com/orientechnologies/OrientDB-NET.binary >>>>>>>>>> i implemented a lot of features of Binary Serializer, and may >>>>>>>>>> help you with your question. >>>>>>>>>> About RECORD_LOAD i don't have any problem and get document in >>>>>>>>>> binary format. >>>>>>>>>> >>>>>>>>>> On Tuesday, December 9, 2014 6:15:13 PM UTC+2, mindplay.dk wrote: >>>>>>>>>>> >>>>>>>>>>> We are well aware that this is no small undertaking, but we >>>>>>>>>>> believe in OrientDB and we think it's worthwhile. >>>>>>>>>>> >>>>>>>>>>> > one big step is actually implement the serialize/deserialize >>>>>>>>>>> of hte document correctly from the binary serialization >>>>>>>>>>> >>>>>>>>>>> To my knowledge, that has not been done in php yet, by anyone? >>>>>>>>>>> All existing implementations, including the fork by Ostico, support >>>>>>>>>>> only >>>>>>>>>>> the CSV style serialization. The binary serialization format >>>>>>>>>>> actually ought >>>>>>>>>>> to be a lot easier to implement, as it won't require a state >>>>>>>>>>> machine/parser >>>>>>>>>>> like the CSV format - and also should be a lot more CPU friendly, >>>>>>>>>>> memory >>>>>>>>>>> efficient, and less bandwidth overhead, so we're targeting that >>>>>>>>>>> exclusively. >>>>>>>>>>> >>>>>>>>>>> We're also targeting the most recent protocol, which already >>>>>>>>>>> differs substantially from what we were able to reference from >>>>>>>>>>> existing >>>>>>>>>>> implementations, which are based on older versions of the protocol. >>>>>>>>>>> We hope >>>>>>>>>>> to support the final version of the protocol when OrientDB 2.0 is >>>>>>>>>>> released >>>>>>>>>>> - we do not want this client library to only support a legacy >>>>>>>>>>> protocol from >>>>>>>>>>> inception. >>>>>>>>>>> >>>>>>>>>>> As said though, it doesn't appear that REQUEST_RECORD_LOAD >>>>>>>>>>> respects the serializer setting - it appears to always return >>>>>>>>>>> records in >>>>>>>>>>> the CSV format. If this a missing feature or server-side issue, we >>>>>>>>>>> won't >>>>>>>>>>> get very far with our client anytime soon... Either way, we need >>>>>>>>>>> someone >>>>>>>>>>> who can at least answer the question and help set us on the right >>>>>>>>>>> path. >>>>>>>>>>> >>>>>>>>>>> At this moment, we are stalled, since we don't even know if the >>>>>>>>>>> server is behaving correctly, or whether we need to support the CSV >>>>>>>>>>> format >>>>>>>>>>> or not. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 9, 2014 at 4:57 PM, Emanuel <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> We have also other drivers php this one https://github.com/ >>>>>>>>>>>> orientechnologies/php-orientdb that also already have a few >>>>>>>>>>>> forks (example this : https://github.com/Ostico/PhpOrient ). >>>>>>>>>>>> >>>>>>>>>>>> i would like to say that is better have less drivers more >>>>>>>>>>>> update and i warn you, write a driver from scratch is not so easy >>>>>>>>>>>> as it >>>>>>>>>>>> seems :) >>>>>>>>>>>> >>>>>>>>>>>> anyway you are free to do so ;) >>>>>>>>>>>> >>>>>>>>>>>> one big step is actually implement the serialize/deserialize >>>>>>>>>>>> of hte document correctly from the binary serialization, that is >>>>>>>>>>>> quite >>>>>>>>>>>> complex and can be also target of evolution/optimization in not to >>>>>>>>>>>> far >>>>>>>>>>>> future. >>>>>>>>>>>> >>>>>>>>>>>> Here in orient we are evaluating to give an easier way to >>>>>>>>>>>> read/write the document on the binary protocol, but i will open >>>>>>>>>>>> another >>>>>>>>>>>> thread on this :) >>>>>>>>>>>> >>>>>>>>>>>> bye >>>>>>>>>>>> >>>>>>>>>>>> Emanuel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 12/09/2014 11:48 AM, Rasmus Schultz wrote: >>>>>>>>>>>> >>>>>>>>>>>> Doctrine is the only one of those projects that still have any >>>>>>>>>>>> traction - and it's a full scale data mapper, what we need is a >>>>>>>>>>>> simple >>>>>>>>>>>> driver/client. >>>>>>>>>>>> >>>>>>>>>>>> We are of course referencing those projects for lots of >>>>>>>>>>>> implementation details, but we're shooting for something much >>>>>>>>>>>> simpler and >>>>>>>>>>>> more low-level, something people can use to build their own >>>>>>>>>>>> mappers/DAO/AR >>>>>>>>>>>> implementations on top of. >>>>>>>>>>>> >>>>>>>>>>>> We're also designing the whole thing using very basic OOP >>>>>>>>>>>> patterns (no traits) in the hopes of porting this to a native >>>>>>>>>>>> extension >>>>>>>>>>>> (e.g. Zephir) eventually. >>>>>>>>>>>> >>>>>>>>>>>> We're also designing the whole thing with zero dependencies >>>>>>>>>>>> on other libraries. >>>>>>>>>>>> >>>>>>>>>>>> So we have somewhat different objectives from the other >>>>>>>>>>>> projects, and more of a minimalist mindset, I think. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 9, 2014 at 12:36 PM, 'Curtis Mosters' via OrientDB >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Well there is no other Google Group. But why not use the >>>>>>>>>>>>> Github already existing PHP OrientDB projects? >>>>>>>>>>>>> >>>>>>>>>>>>> https://github.com/AntonTerekhov/OrientDB-PHP >>>>>>>>>>>>> https://github.com/doctrine/orientdb-odm >>>>>>>>>>>>> https://packagist.org/packages/orientdb-php/orientdb-php >>>>>>>>>>>>> >>>>>>>>>>>>> I don't know but this would be way better to do it there. WDYT? >>>>>>>>>>>>> >>>>>>>>>>>>> Am Dienstag, 9. Dezember 2014 11:38:07 UTC+1 schrieb >>>>>>>>>>>>> mindplay.dk: >>>>>>>>>>>>> >>>>>>>>>>>>>> Is there a different group for developers with more technical >>>>>>>>>>>>>> questions? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I want to help bring OrientDB to php - is this the right >>>>>>>>>>>>>> place for that? Or is nobody interested? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Monday, December 8, 2014 5:19:07 PM UTC+1, mindplay.dk >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm trying to tackle REQUEST_RECORD_LOAD as the first useful >>>>>>>>>>>>>>> function in my PHP client. (I have the basics like connect and >>>>>>>>>>>>>>> open, error >>>>>>>>>>>>>>> handling, etc. working so far.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This being a PHP client, one major concern for me, is to >>>>>>>>>>>>>>> avoid parsing (with a state machine, as was necessary with the >>>>>>>>>>>>>>> old format) >>>>>>>>>>>>>>> since this is extremely inefficient in PHP - this is one reason >>>>>>>>>>>>>>> I'm >>>>>>>>>>>>>>> targeting OrientDB 2.0 and the new binary format exclusively, >>>>>>>>>>>>>>> as this >>>>>>>>>>>>>>> appears to make that possible (?) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Unfortunately, the response format of REQUEST_RECORD_LOAD >>>>>>>>>>>>>>> itself appears to make that impossible. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [(payload-status:byte)[(record-content:bytes)(record- >>>>>>>>>>>>>>> version:int)(record-type:byte)]*]+ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In order to read sequentially over "record-content", I >>>>>>>>>>>>>>> need to know the "record-type" in advance, so the order of this >>>>>>>>>>>>>>> data >>>>>>>>>>>>>>> appears to be wrong? I believe the record format of each >>>>>>>>>>>>>>> payload chunk >>>>>>>>>>>>>>> would need to backwards, basically: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [(payload-status:byte)[(record-type:byte)(record-version: >>>>>>>>>>>>>>> int)(record-content:bytes)]*]+ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Otherwise, I am forced to load the whole record-content >>>>>>>>>>>>>>> into memory first, before I can know how to interpret the data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Or am I missing something here? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also, it appears the "record-content" is in the old CSV >>>>>>>>>>>>>>> format, regardless of my having selected the new binary >>>>>>>>>>>>>>> serialization >>>>>>>>>>>>>>> format? Does the REQUEST_RECORD_LOAD command not support the >>>>>>>>>>>>>>> new binary >>>>>>>>>>>>>>> serialization format? Is it not supported everywhere yet? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I really do not want a client that has to load and then >>>>>>>>>>>>>>> parse in two stages - this adds considerable complexity, >>>>>>>>>>>>>>> run-time overhead, >>>>>>>>>>>>>>> and duplicates everything in-memory while loading. I'm probably >>>>>>>>>>>>>>> doing >>>>>>>>>>>>>>> something wrong or missing something obvious? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> --- >>>>>>>>>>>>> You received this message because you are subscribed to a >>>>>>>>>>>>> topic in the Google Groups "OrientDB" group. >>>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>>> https://groups.google.com/d/topic/orient-database/9CKEun_Wrr >>>>>>>>>>>>> A/unsubscribe. >>>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "OrientDB" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "OrientDB" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/orient-database/9CKEun_Wrr >>>>>>>>>>>> A/unsubscribe. >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "OrientDB" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/orient-database/9CKEun_Wrr >>>>>>>>>> A/unsubscribe. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to [email protected]. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "OrientDB" group. >>>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>>> topic/orient-database/9CKEun_WrrA/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "OrientDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "OrientDB" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>> topic/orient-database/9CKEun_WrrA/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "OrientDB" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/orient-database/9CKEun_WrrA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
