[This message was posted by Rolf Andersson of Pantor Engineering <[email protected]> to the "FAST Protocol" discussion forum at http://fixprotocol.org/discuss/46. You can reply to it on-line at http://fixprotocol.org/discuss/read/a17129a9 - PLEASE DO NOT REPLY BY MAIL.]
Marc, thank you for your feedback. I have commented below on your comments about the map operator. At least one major feed, the ISE depth feed, includes a mapping of instrument identifiers. I think it is worth while to try to determine if this mapping should occur above the FAST layer or we should try to include a mapping feature into FAST. That said, I'm currently leaning towards sending the map operator proposal back to the drawing board. I believe we need to spend more time analyzing different alternatives for a mapping feature and to discuss more with the implementors. Your Best, Rolf > Hi, > > Here is our feedback on the fast 1.2 extension proposal. > Our comments are obviously biased by our ns latency concern. > > New type definition syntax: OK > Enumeration: OK > Set: OK > Bit Group: OK > PMAP field placement: OK > > For the Bit group and the PMAP field placement, special care > should be taken to properly define how they interact with the > PMAP and the operators. Default and copy are probably the only > safe operators usable with Bit groups, enum and set anyway. > > The PMAP fields should be restricted to a few bits only. > For an uInt7 there is nothing to gain by putting it into the > PMAP and for uInt6 you only gain 1 bit. This will be very good > though for bits or very small enum/sets though. > > Timestamp: OK. > > Map is really not OK! > > 1 - It breaks the FAST conceptual model where both the sender > and receiver share a pre-defined set of templates. With this, > you will also need a long term context and a large memory. > There are already a context used for the operators like > copy/inc/etc. but it's a short term one generally limited in > scope to one multicast frame for instance and also its memory > use is known and small. I don't see how the map operator breaks the conceptual model. The sender and the receiver will still share the same templates. The map operator may however make use of a large "context memory" to store previously defined mappings. This differs from previous FAST constructs and may present a challenge in resource constrained environments. The basic use of the map is obviously within the same context as all other operators. In the case of a multicast feed the lifetime of a mapping must either be one frame or mappings after a lost frame will be impossible to interpret correctly. It is entirely possible to recover from this situation, much as current feeds offer ways of recovering from lost frames when reconstructing a full orderbook from incremental updates. Both the ISE and the CME provide recovery mechanism that are designed to work above the FAST level. Furthermore, the ISE has a construct that is similar to the map construct, but the ISE construct is defined above the FAST level. The ISE template defines a "full refresh" message that contains a mapping between a complete but verbose instrument identifier and a smaller numeric identifier. The defined "incremental refresh" message contains the numeric identifier only. If a frame is lost you'll need to wait for a complete "full refresh" cycle in order to be sure that you have complete and correct information available. In the case of the ISE it seems to be implied that mappings don't change and identifiers don't get reused during a day and this provides an opportunity to make simplifying assumptions about the validity of instrument id mappings. > 2 - As it's a dynamic construct, it can't be optimized at > compile time and so will be slow. See above. I don't see that it is more dynamic than other operators, but it requires more memory and a longer access code path to look up values in the mapping context (possibly some sort of hash table) > 3- It will not work reliably. FAST is used a lot in multicast > and unicast market data diffusion. So what do you do when you > loose a packet? How do you know if you are in sync with the > sender? When you start to process a feed, how do you get the > current map? A giant snapshot? As consecutive IDs are not > mandatory, you can't use an array and so you will need to use > an hash table (or trees, skip-lists, whatever) and this will > use a lot of memory and will be SLOW. So when I read "only a > number of thousand values" I get really scared for the latency. > ;-) For example if you want short instrument identifiers instead of a more verbose symbol, you have to do the translation somewhere. Question is if we're ready to commit to placing this feature in the FAST layer or if it better to leave it to upper layer processing. I think this is analogous to maintaining a complete orderbook with a feed that provides incremental refresh messages without a guaranteed strict delivery order (as in the case with multicast feeds). (On a side-note: If you look at the execution path in the next processing step it may be better (performance wise) to have numeric instrument identifiers rather than more verbose string identifiers. So it could well be that you would want a FAST decoder to provide you with the numeric ids rather than the string variety. In the case that the feed doesn't present numeric ids you may even want the decoder to produce them ...) > So IMO this should not be added to FAST as it's really not fast > at all. > If really it is added to FAST, let's at least force it to use > consecutive IDs. > > Note that a pre-shared map is just a big enum and that is OK of > course. > > Best regards, > > Marc [You can unsubscribe from this discussion group by sending a message to mailto:[email protected]] --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Financial Information eXchange" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/FIX-Protocol?hl=en -~----------~----~----~----~------~----~------~--~---
