[This message was posted by Marc Battyani of HPC Platform <[email protected]> to the "FAST Protocol" discussion forum at http://fixprotocol.org/discuss/46. You can reply to it on-line at http://fixprotocol.org/discuss/read/b622a6e0 - PLEASE DO NOT REPLY BY MAIL.]
Rolf, I agree with you that very often the exchange use a mapping between the contracts names and integer contracts IDs but they do it out of band on a separate multicast channel or messages with specific templates and with an application level protocol. At least this is what EUREX EBS does. ISE uses the snapshot messages to give the IDs. These mappings are valid for the day and an important point is that the market data messages use these integers IDs but do not convert them to the long ascii names. So the whole chain can work with integers IDs and never have to go back to the text names. The proposed map element would make the FAST decoder replace the short integer IDs by the textual names and so the applications down the chain will have to rehash it again slowing down the order book fetch for instance. BTW this is also what you said in your reply below so I guess we agree on this. :) Best Regards, Marc > Marc, > > thank you for your feedback. > > I have commented below on your comments about the map operator. > > At least one major feed, the ISE depth feed, includes a mapping of > instrument identifiers. I think it is worth while to try to determine if > this mapping should occur above the FAST layer or we should try to > include a mapping feature into FAST. > > That said, I'm currently leaning towards sending the map operator > proposal back to the drawing board. I believe we need to spend more time > analyzing different alternatives for a mapping feature and to discuss > more with the implementors. > > Your > > Best, Rolf > > > Hi, > > > > Here is our feedback on the fast 1.2 extension proposal. Our comments > > are obviously biased by our ns latency concern. > > > > New type definition syntax: OK Enumeration: OK Set: OK Bit Group: OK > > PMAP field placement: OK > > > > For the Bit group and the PMAP field placement, special care should be > > taken to properly define how they interact with the PMAP and the > > operators. Default and copy are probably the only safe operators > > usable with Bit groups, enum and set anyway. > > > > The PMAP fields should be restricted to a few bits only. For an uInt7 > > there is nothing to gain by putting it into the PMAP and for uInt6 you > > only gain 1 bit. This will be very good though for bits or very small > > enum/sets though. > > > > Timestamp: OK. > > > > Map is really not OK! > > > > 1 - It breaks the FAST conceptual model where both the sender and > > receiver share a pre-defined set of templates. With this, you will > > also need a long term context and a large memory. There are already a > > context used for the operators like copy/inc/etc. but it's a short > > term one generally limited in scope to one multicast frame for > > instance and also its memory use is known and small. > > I don't see how the map operator breaks the conceptual model. The sender > and the receiver will still share the same templates. > > The map operator may however make use of a large "context memory" to > store previously defined mappings. This differs from previous FAST > constructs and may present a challenge in resource constrained > environments. > > The basic use of the map is obviously within the same context as all > other operators. In the case of a multicast feed the lifetime of a > mapping must either be one frame or mappings after a lost frame will be > impossible to interpret correctly. It is entirely possible to recover > from this situation, much as current feeds offer ways of recovering from > lost frames when reconstructing a full orderbook from incremental > updates. Both the ISE and the CME provide recovery mechanism that are > designed to work above the FAST level. > > Furthermore, the ISE has a construct that is similar to the map > construct, but the ISE construct is defined above the FAST level. The > ISE template defines a "full refresh" message that contains a mapping > between a complete but verbose instrument identifier and a smaller > numeric identifier. The defined "incremental refresh" message contains > the numeric identifier only. If a frame is lost you'll need to wait for > a complete "full refresh" cycle in order to be sure that you have > complete and correct information available. > > In the case of the ISE it seems to be implied that mappings don't change > and identifiers don't get reused during a day and this provides an > opportunity to make simplifying assumptions about the validity of > instrument id mappings. > > > 2 - As it's a dynamic construct, it can't be optimized at compile time > > and so will be slow. > > See above. I don't see that it is more dynamic than other operators, but > it requires more memory and a longer access code path to look up values > in the mapping context (possibly some sort of hash table) > > > 3- It will not work reliably. FAST is used a lot in multicast and > > unicast market data diffusion. So what do you do when you loose a > > packet? How do you know if you are in sync with the sender? When you > > start to process a feed, how do you get the current map? A giant > > snapshot? As consecutive IDs are not mandatory, you can't use an array > > and so you will need to use an hash table (or trees, skip-lists, > > whatever) and this will use a lot of memory and will be SLOW. So when > > I read "only a number of thousand values" I get really scared for the > > latency. ;-) > > For example if you want short instrument identifiers instead of a more > verbose symbol, you have to do the translation somewhere. Question is if > we're ready to commit to placing this feature in the FAST layer or if it > better to leave it to upper layer processing. > > I think this is analogous to maintaining a complete orderbook with a > feed that provides incremental refresh messages without a guaranteed > strict delivery order (as in the case with multicast feeds). > > (On a side-note: If you look at the execution path in the next > processing step it may be better (performance wise) to have numeric > instrument identifiers rather than more verbose string identifiers. So > it could well be that you would want a FAST decoder to provide you with > the numeric ids rather than the string variety. In the case that the > feed doesn't present numeric ids you may even want the decoder to > produce them ...) > > > So IMO this should not be added to FAST as it's really not fast at > > all. If really it is added to FAST, let's at least force it to use > > consecutive IDs. > > > > Note that a pre-shared map is just a big enum and that is OK of > > course. > > > > Best regards, > > > > Marc [You can unsubscribe from this discussion group by sending a message to mailto:[email protected]] --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Financial Information eXchange" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/FIX-Protocol?hl=en -~----------~----~----~----~------~----~------~--~---
