You can look hear https://github.com/orientechnologies/orientdb/wiki/Record-Schemaless-Binary-Serialization#shortintegerlong
On Friday, December 12, 2014 1:44:01 PM UTC+2, mindplay.dk wrote: > > Glad to hear that, thanks :-) > > So on a related note - the "varint" type used in the OrientDB binary > protocol, what specification does it follow precisely? Because apparently > there are lots > <http://vpri.org/fonc_wiki/index.php/Variable_Length_Integer> of ways to > encode a variable-size integer. > > I sort of wish there was an option for the client to disable > variable-length integers in the protocol, instead encoding them with a > fixed size. > > I can implement UTF-8 style reading/writing of variable-size integers in > PHP, but this is going to add considerable CPU overhead - in the case of a > PHP client (probably other scripting languages too) a small amount of > bandwidth overhead is likely preferable to CPU overhead. What we want is a > fast client - whether that means using a little more bandwidth is probably > secondary, as is the ability to support more than 2 billion records for > most projects. > > Just putting that out there :-) > > But for the time being, can you point me to a specification or (better) a > reference implementation (in any language) of the VLI encoding used by > OrientDB? > > I can reference the one in GoorMoon's .NET driver > <https://github.com/orientechnologies/OrientDB-NET.binary/blob/binary.serialization/src/Orient/Orient.Client/Protocol/Serializers/RecordBinarySerializer.cs#L502>, > > or the one on wikipedia <http://en.wikipedia.org/wiki/UTF-8>, but neither > of them appear to have tests, and I'm unsure how to test them. Go has a nice > implementation <http://golang.org/src/encoding/binary/varint.go> with > tests <http://golang.org/src/encoding/binary/varint_test.go>, but since > there are so many types of VLI which don't appear to have any official > names or standardization, I can't be sure it's the same type of encoding... > > > On Fri, Dec 12, 2014 at 11:45 AM, Luca Garulli <[email protected] > <javascript:>> wrote: >> >> Authors of drivers have such kind of high priority on requests :-) >> >> Lvc@ >> >> >> On 11 December 2014 at 23:29, GoorMoon <[email protected] <javascript:>> >> wrote: >> >>> Glad to hear !!! >>> >>> On Thursday, December 11, 2014 11:41:05 PM UTC+2, mindplay.dk wrote: >>>> >>>> Never mind, spotted it :-) >>>> >>>> >>>> On Thu, Dec 11, 2014 at 10:03 PM, Rasmus Schultz <[email protected]> >>>> wrote: >>>> >>>>> That was a fast decision - very happy to see them reacting to this >>>>> issue so quickly and slating it for the 2.0 release! :-) >>>>> >>>>> So I'm referencing a bunch of your code for my client now, and I'm >>>>> hung up on a small issue, maybe you can point me in the right direction... >>>>> >>>>> The so-called "varint" type in the binary serialization - I understand >>>>> it's a variable-size integer, encoded like UTF-8 character codes. Where >>>>> or >>>>> how do you handle this in your implementation? >>>>> >>>>> >>>>> On Thu, Dec 11, 2014 at 7:31 PM, Rasmus Schultz <[email protected]> >>>>> wrote: >>>>> >>>>>> Posted here >>>>>> >>>>>> https://github.com/orientechnologies/orientdb/issues/3175 >>>>>> >>>>>> >>>>>> On Thu, Dec 11, 2014 at 7:10 PM, GoorMoon <[email protected]> wrote: >>>>>> >>>>>>> I am agree with you, but this not our decision. >>>>>>> I suggest you to open issue here https://github.com/ >>>>>>> orientechnologies/orientdb that describe your purpose. >>>>>>> >>>>>>> >>>>>>> On Thursday, December 11, 2014 10:55:46 AM UTC+2, mindplay.dk wrote: >>>>>>>> >>>>>>>> This looks great, thanks! So much simpler than the CSV serializer. >>>>>>>> >>>>>>>> I see that you do have to buffer the record in memory still, as I >>>>>>>> suspected. I really do wish they would make the change I suggested >>>>>>>> below, >>>>>>>> putting the record format before the actual record - I think then you >>>>>>>> wouldn't need to buffer records in memory before you can deserialize. >>>>>>>> Thoughts? >>>>>>>> On Dec 10, 2014 1:09 PM, "GoorMoon" <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hey, >>>>>>>>> I don't know about PHP driver but, i contribute to .NET Driver >>>>>>>>> https://github.com/orientechnologies/OrientDB-NET.binary >>>>>>>>> i implemented a lot of features of Binary Serializer, and may help >>>>>>>>> you with your question. >>>>>>>>> About RECORD_LOAD i don't have any problem and get document in >>>>>>>>> binary format. >>>>>>>>> >>>>>>>>> On Tuesday, December 9, 2014 6:15:13 PM UTC+2, mindplay.dk wrote: >>>>>>>>>> >>>>>>>>>> We are well aware that this is no small undertaking, but we >>>>>>>>>> believe in OrientDB and we think it's worthwhile. >>>>>>>>>> >>>>>>>>>> > one big step is actually implement the serialize/deserialize >>>>>>>>>> of hte document correctly from the binary serialization >>>>>>>>>> >>>>>>>>>> To my knowledge, that has not been done in php yet, by anyone? >>>>>>>>>> All existing implementations, including the fork by Ostico, support >>>>>>>>>> only >>>>>>>>>> the CSV style serialization. The binary serialization format >>>>>>>>>> actually ought >>>>>>>>>> to be a lot easier to implement, as it won't require a state >>>>>>>>>> machine/parser >>>>>>>>>> like the CSV format - and also should be a lot more CPU friendly, >>>>>>>>>> memory >>>>>>>>>> efficient, and less bandwidth overhead, so we're targeting that >>>>>>>>>> exclusively. >>>>>>>>>> >>>>>>>>>> We're also targeting the most recent protocol, which already >>>>>>>>>> differs substantially from what we were able to reference from >>>>>>>>>> existing >>>>>>>>>> implementations, which are based on older versions of the protocol. >>>>>>>>>> We hope >>>>>>>>>> to support the final version of the protocol when OrientDB 2.0 is >>>>>>>>>> released >>>>>>>>>> - we do not want this client library to only support a legacy >>>>>>>>>> protocol from >>>>>>>>>> inception. >>>>>>>>>> >>>>>>>>>> As said though, it doesn't appear that REQUEST_RECORD_LOAD >>>>>>>>>> respects the serializer setting - it appears to always return >>>>>>>>>> records in >>>>>>>>>> the CSV format. If this a missing feature or server-side issue, we >>>>>>>>>> won't >>>>>>>>>> get very far with our client anytime soon... Either way, we need >>>>>>>>>> someone >>>>>>>>>> who can at least answer the question and help set us on the right >>>>>>>>>> path. >>>>>>>>>> >>>>>>>>>> At this moment, we are stalled, since we don't even know if the >>>>>>>>>> server is behaving correctly, or whether we need to support the CSV >>>>>>>>>> format >>>>>>>>>> or not. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Dec 9, 2014 at 4:57 PM, Emanuel <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> We have also other drivers php this one https://github.com/ >>>>>>>>>>> orientechnologies/php-orientdb that also already have a few >>>>>>>>>>> forks (example this : https://github.com/Ostico/PhpOrient ). >>>>>>>>>>> >>>>>>>>>>> i would like to say that is better have less drivers more update >>>>>>>>>>> and i warn you, write a driver from scratch is not so easy as it >>>>>>>>>>> seems :) >>>>>>>>>>> >>>>>>>>>>> anyway you are free to do so ;) >>>>>>>>>>> >>>>>>>>>>> one big step is actually implement the serialize/deserialize of >>>>>>>>>>> hte document correctly from the binary serialization, that is quite >>>>>>>>>>> complex >>>>>>>>>>> and can be also target of evolution/optimization in not to far >>>>>>>>>>> future. >>>>>>>>>>> >>>>>>>>>>> Here in orient we are evaluating to give an easier way to >>>>>>>>>>> read/write the document on the binary protocol, but i will open >>>>>>>>>>> another >>>>>>>>>>> thread on this :) >>>>>>>>>>> >>>>>>>>>>> bye >>>>>>>>>>> >>>>>>>>>>> Emanuel >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 12/09/2014 11:48 AM, Rasmus Schultz wrote: >>>>>>>>>>> >>>>>>>>>>> Doctrine is the only one of those projects that still have any >>>>>>>>>>> traction - and it's a full scale data mapper, what we need is a >>>>>>>>>>> simple >>>>>>>>>>> driver/client. >>>>>>>>>>> >>>>>>>>>>> We are of course referencing those projects for lots of >>>>>>>>>>> implementation details, but we're shooting for something much >>>>>>>>>>> simpler and >>>>>>>>>>> more low-level, something people can use to build their own >>>>>>>>>>> mappers/DAO/AR >>>>>>>>>>> implementations on top of. >>>>>>>>>>> >>>>>>>>>>> We're also designing the whole thing using very basic OOP >>>>>>>>>>> patterns (no traits) in the hopes of porting this to a native >>>>>>>>>>> extension >>>>>>>>>>> (e.g. Zephir) eventually. >>>>>>>>>>> >>>>>>>>>>> We're also designing the whole thing with zero dependencies on >>>>>>>>>>> other libraries. >>>>>>>>>>> >>>>>>>>>>> So we have somewhat different objectives from the other >>>>>>>>>>> projects, and more of a minimalist mindset, I think. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 9, 2014 at 12:36 PM, 'Curtis Mosters' via OrientDB < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Well there is no other Google Group. But why not use the Github >>>>>>>>>>>> already existing PHP OrientDB projects? >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/AntonTerekhov/OrientDB-PHP >>>>>>>>>>>> https://github.com/doctrine/orientdb-odm >>>>>>>>>>>> https://packagist.org/packages/orientdb-php/orientdb-php >>>>>>>>>>>> >>>>>>>>>>>> I don't know but this would be way better to do it there. WDYT? >>>>>>>>>>>> >>>>>>>>>>>> Am Dienstag, 9. Dezember 2014 11:38:07 UTC+1 schrieb >>>>>>>>>>>> mindplay.dk: >>>>>>>>>>>> >>>>>>>>>>>>> Is there a different group for developers with more technical >>>>>>>>>>>>> questions? >>>>>>>>>>>>> >>>>>>>>>>>>> I want to help bring OrientDB to php - is this the right >>>>>>>>>>>>> place for that? Or is nobody interested? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Monday, December 8, 2014 5:19:07 PM UTC+1, mindplay.dk >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm trying to tackle REQUEST_RECORD_LOAD as the first useful >>>>>>>>>>>>>> function in my PHP client. (I have the basics like connect and >>>>>>>>>>>>>> open, error >>>>>>>>>>>>>> handling, etc. working so far.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> This being a PHP client, one major concern for me, is to >>>>>>>>>>>>>> avoid parsing (with a state machine, as was necessary with the >>>>>>>>>>>>>> old format) >>>>>>>>>>>>>> since this is extremely inefficient in PHP - this is one reason >>>>>>>>>>>>>> I'm >>>>>>>>>>>>>> targeting OrientDB 2.0 and the new binary format exclusively, as >>>>>>>>>>>>>> this >>>>>>>>>>>>>> appears to make that possible (?) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unfortunately, the response format of REQUEST_RECORD_LOAD >>>>>>>>>>>>>> itself appears to make that impossible. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [(payload-status:byte)[(record-content:bytes)(record-version >>>>>>>>>>>>>> :int)(record-type:byte)]*]+ >>>>>>>>>>>>>> >>>>>>>>>>>>>> In order to read sequentially over "record-content", I need >>>>>>>>>>>>>> to know the "record-type" in advance, so the order of this data >>>>>>>>>>>>>> appears to >>>>>>>>>>>>>> be wrong? I believe the record format of each payload chunk >>>>>>>>>>>>>> would need to >>>>>>>>>>>>>> backwards, basically: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [(payload-status:byte)[(record-type:byte)(record-version:int >>>>>>>>>>>>>> )(record-content:bytes)]*]+ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Otherwise, I am forced to load the whole record-content >>>>>>>>>>>>>> into memory first, before I can know how to interpret the data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Or am I missing something here? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, it appears the "record-content" is in the old CSV >>>>>>>>>>>>>> format, regardless of my having selected the new binary >>>>>>>>>>>>>> serialization >>>>>>>>>>>>>> format? Does the REQUEST_RECORD_LOAD command not support the new >>>>>>>>>>>>>> binary >>>>>>>>>>>>>> serialization format? Is it not supported everywhere yet? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I really do not want a client that has to load and then >>>>>>>>>>>>>> parse in two stages - this adds considerable complexity, >>>>>>>>>>>>>> run-time overhead, >>>>>>>>>>>>>> and duplicates everything in-memory while loading. I'm probably >>>>>>>>>>>>>> doing >>>>>>>>>>>>>> something wrong or missing something obvious? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "OrientDB" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/orient-database/9CKEun_Wrr >>>>>>>>>>>> A/unsubscribe. >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "OrientDB" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>> in the Google Groups "OrientDB" group. >>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>> https://groups.google.com/d/topic/orient-database/9CKEun_Wrr >>>>>>>>>>> A/unsubscribe. >>>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>>> to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "OrientDB" group. >>>>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>>>> topic/orient-database/9CKEun_WrrA/unsubscribe. >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "OrientDB" group. >>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>> topic/orient-database/9CKEun_WrrA/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "OrientDB" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/orient-database/9CKEun_WrrA/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
