Hi Kevin,
Thanks for the suggestion. I indeed considered this option of including all
the possible fields in one message. But since some of the financial
instruments, like convertible bonds/options, have 600-700 fields on the
server side, I felt like instantiating such a big object for every update
seems bit costly (though on the wire it is efficient ,as you mentioned, as
only the fields which are set will be present).* Using object pool we can
probably avoid the object instantiating cost though.* Also, once the object
is deserialized on the receiving side, I feel like it is going to be little
tricky to figure which of the fields are actually set on the sender side.
My understanding is that once the object is deserialized, all fields which
are not set on sender side will have default values and we need to iterate
over all the fields and compare them against default values to know which
are actually set by the sender in the message. So, overall i felt like for
classes having many fields this approach is bit inefficient. We can
probably group set of fields and use a separate message for each group
(like QuoteUpdate, PositionUpdate, etc) but that requires lot of changes in
my current code base and theoretically we might end up too many groups.
Also, fields like positions, market value, etc need to be published for
each portfolio where the portfolios can be dynamic. So, having support for
Map like object seems to solve both the problems. Having said this, I'm
open to the suggestions because the Map approach does have its limitation,
as the field names can be free form and can cause issues later.
Point duly noted about the timestamp_utc. I changed my schema accordingly.
Thanks
Ranadheer
On Saturday, April 2, 2016 at 10:47:47 PM UTC+5:30, Kevin Baker wrote:
>
> Hi Ranadheer,
>
> Just a piece of advice, but you may want to try to keep your types as
> strict as possible... i.e. instead of using a general map to store the
> parameters, use an additional type to keep everything easy to represent.
> Something like:
>
> message Tick {
> string subject = 1; // name of the financial instrument - something
> like MSFT, GOOG, etc
> uint64 timestamp = 2; //millis from epoch signifying the timestamp at
> which the object is constructed at the publisher side.
> TickData tick_data = 3;
> }
>
> message TickData {
> float ask_price = 1;
> float bid_price = 2;
> float trade_price = 3;
> uint32 trade_size = 4;
> }
>
> or even better, just add all the components into one message:
>
> message Tick {
> string subject = 1; // name of the financial instrument - something
> like MSFT, GOOG, etc
> uint64 timestamp = 2; //millis from epoch signifying the timestamp at
> which the object is constructed at the publisher side.
> float ask_price = 3;
> float bid_price = 4;
> float trade_price = 5;
> uint64 trade_size = 6;
> ...
> }
>
> ... adding any other possible fields you might have in your data. As well
> as being a lot more compact on-the-wire for bandwidth and CPU improvements,
> this forces you to think about your data and what might be in it, which
> will result in a lot less bugs down the road for you and your client
> consumers. You won't have to worry about typos like accidentally typing
> 'bid_prce' or 'bidPrice' or 'bidprice' in the Map.
>
> Protobuf will not send any fields that are different from their default
> values, so you don't pay any performance penalty for having a lot of
> optional data in the Tick message. You can also still add fields later to
> the message, while keeping backwards compatibility with old consumers.
>
> Also, another little pedantic thing, but if you are using the timestamp
> like Javascript's *Date.now()*, always name it *timestamp_utc *instead of
> just *timestamp*... eventually someone will stuff a local time in there
> and confuse everyone... better to be explicit.
>
> Kevin
>
> On Friday, April 1, 2016 at 7:50:09 PM UTC-5, Feng Xiao wrote:
>>
>>
>>
>> On Fri, Apr 1, 2016 at 6:06 AM, Ranadheer Pulluru <prana...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm planning to use protobuf for publishing tick data of the financial
>>> instruments. The consumers can be any of the java/python/node.js
>>> languages.The tick is expected to contain various fields like (symbol,
>>> ask_price, bid_price, trade_price, trade_time, trade_size, etc). Basically,
>>> it is sort of a map from field name to the value, where value type can be
>>> any of the primitive types. I thought I can define the schema of the Tick
>>> data structure, using map
>>> <https://developers.google.com/protocol-buffers/docs/proto3#maps>and Any
>>> <https://developers.google.com/protocol-buffers/docs/proto3#any>as
>>> follows
>>>
>>>
>>> syntax = "proto3";
>>>
>>> package tutorial;
>>>
>>> import "google/protobuf/any.proto";
>>>
>>> message Tick {
>>> string subject = 1; // name of the financial instrument - something
>>> like MSFT, GOOG, etc
>>> uint64 timestamp = 2; //millis from epoch signifying the timestamp
>>> at which the object is constructed at the publisher side.
>>> map<string, google.protobuf.Any> fvmap = 3; // the actual map
>>> having field name and values. Something like {ask_price: 10.5, bid_price:
>>> 9.5, trade_price: 10, trade_size=5}
>>> }
>>>
>>> Though I'm able to generate the code in different languages for this
>>> schema, I'm not sure how to populate the values in the *fvmap*.
>>>
>>> public class TickTest
>>> {
>>> public static void main(String[] args)
>>> {
>>> Tick.Builder tick = Tick.newBuilder();
>>> tick.setSubject("ucas");
>>> tick.setTimestamp(System.currentTimeMillis());
>>> Map<String, Any> fvMap = tick.getMutableFvmap();
>>> // fvMap.put("ask", value); // Not sure how to pass values like
>>> 10.5/9.5/10/5 to Any object here.
>>> }
>>> }
>>>
>>>
>>>
>>> Could you please let me know how to populate the fvMap with different
>>> fields and values here? Please feel tree to tell me if using map
>>> <https://developers.google.com/protocol-buffers/docs/proto3#maps>and
>>> Any <https://developers.google.com/protocol-buffers/docs/proto3#any>is
>>> not the right choice and if there are any better alternatives.
>>>
>> It seems to me a google.protobuf.Struct suites your purpose better:
>>
>> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L51
>>
>>
>>>
>>> Thanks
>>> Ranadheer
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
