[This message was posted by Radu-Adrian Popescu of <[email protected]> to the "FAST Protocol" discussion forum at http://fixprotocol.org/discuss/46. You can reply to it on-line at http://fixprotocol.org/discuss/read/2647cd3d - PLEASE DO NOT REPLY BY MAIL.]
Hello gentlemen, I thought I'd chime in and support Marc's comments here, and also add that the application level protocol is not only a good idea, but rather mandatory. Here's a couple of examples that I think justify this attitude: - problems in the instruments' definitions and integer id mapping can invalidate the good-for-the-day mappings and require a refresh of the information; in practice this happens a couple of times a year at most; - exchange backend failure/failover can trigger a refresh of this referential information; - instrument lifecycle is a bit more complex, as instruments can be introduced and removed during the day; a good example are strategies (combos). In short, the point is that the issue of mapping is probably best left off to the specific application. All the best, kind regards, -- Radu > Rolf, > > I agree with you that very often the exchange use a mapping between the > contracts names and integer contracts IDs but they do it out of band on > a separate multicast channel or messages with specific templates and > with an application level protocol. At least this is what EUREX EBS > does. ISE uses the snapshot messages to give the IDs. These mappings are > valid for the day and an important point is that the market data > messages use these integers IDs but do not convert them to the long > ascii names. So the whole chain can work with integers IDs and never > have to go back to the text names. The proposed map element would make > the FAST decoder replace the short integer IDs by the textual names and > so the applications down the chain will have to rehash it again slowing > down the order book fetch for instance. > > BTW this is also what you said in your reply below so I guess we agree > on this. :) > > Best Regards, > > Marc > > > > Marc, > > > > thank you for your feedback. > > > > I have commented below on your comments about the map operator. > > > > At least one major feed, the ISE depth feed, includes a mapping of > > instrument identifiers. I think it is worth while to try to determine > > if this mapping should occur above the FAST layer or we should try to > > include a mapping feature into FAST. > > > > That said, I'm currently leaning towards sending the map operator > > proposal back to the drawing board. I believe we need to spend more > > time analyzing different alternatives for a mapping feature and to > > discuss more with the implementors. > > > > Your > > > > Best, Rolf > > > > > Hi, > > > > > > Here is our feedback on the fast 1.2 extension proposal. Our > > > comments are obviously biased by our ns latency concern. > > > > > > New type definition syntax: OK Enumeration: OK Set: OK Bit Group: OK > > > PMAP field placement: OK > > > > > > For the Bit group and the PMAP field placement, special care should > > > be taken to properly define how they interact with the PMAP and the > > > operators. Default and copy are probably the only safe operators > > > usable with Bit groups, enum and set anyway. > > > > > > The PMAP fields should be restricted to a few bits only. For an > > > uInt7 there is nothing to gain by putting it into the PMAP and for > > > uInt6 you only gain 1 bit. This will be very good though for bits or > > > very small enum/sets though. > > > > > > Timestamp: OK. > > > > > > Map is really not OK! > > > > > > 1 - It breaks the FAST conceptual model where both the sender and > > > receiver share a pre-defined set of templates. With this, you will > > > also need a long term context and a large memory. There are already > > > a context used for the operators like copy/inc/etc. but it's a short > > > term one generally limited in scope to one multicast frame for > > > instance and also its memory use is known and small. > > > > I don't see how the map operator breaks the conceptual model. The > > sender and the receiver will still share the same templates. > > > > The map operator may however make use of a large "context memory" to > > store previously defined mappings. This differs from previous FAST > > constructs and may present a challenge in resource constrained > > environments. > > > > The basic use of the map is obviously within the same context as all > > other operators. In the case of a multicast feed the lifetime of a > > mapping must either be one frame or mappings after a lost frame will > > be impossible to interpret correctly. It is entirely possible to > > recover from this situation, much as current feeds offer ways of > > recovering from lost frames when reconstructing a full orderbook from > > incremental updates. Both the ISE and the CME provide recovery > > mechanism that are designed to work above the FAST level. > > > > Furthermore, the ISE has a construct that is similar to the map > > construct, but the ISE construct is defined above the FAST level. The > > ISE template defines a "full refresh" message that contains a mapping > > between a complete but verbose instrument identifier and a smaller > > numeric identifier. The defined "incremental refresh" message contains > > the numeric identifier only. If a frame is lost you'll need to wait > > for a complete "full refresh" cycle in order to be sure that you have > > complete and correct information available. > > > > In the case of the ISE it seems to be implied that mappings don't > > change and identifiers don't get reused during a day and this provides > > an opportunity to make simplifying assumptions about the validity of > > instrument id mappings. > > > > > 2 - As it's a dynamic construct, it can't be optimized at compile > > > time and so will be slow. > > > > See above. I don't see that it is more dynamic than other operators, > > but it requires more memory and a longer access code path to look up > > values in the mapping context (possibly some sort of hash table) > > > > > 3- It will not work reliably. FAST is used a lot in multicast and > > > unicast market data diffusion. So what do you do when you loose a > > > packet? How do you know if you are in sync with the sender? When you > > > start to process a feed, how do you get the current map? A giant > > > snapshot? As consecutive IDs are not mandatory, you can't use an > > > array and so you will need to use an hash table (or trees, skip- > > > lists, whatever) and this will use a lot of memory and will be SLOW. > > > So when I read "only a number of thousand values" I get really > > > scared for the latency. ;-) > > > > For example if you want short instrument identifiers instead of a more > > verbose symbol, you have to do the translation somewhere. Question is > > if we're ready to commit to placing this feature in the FAST layer or > > if it better to leave it to upper layer processing. > > > > I think this is analogous to maintaining a complete orderbook with a > > feed that provides incremental refresh messages without a guaranteed > > strict delivery order (as in the case with multicast feeds). > > > > (On a side-note: If you look at the execution path in the next > > processing step it may be better (performance wise) to have numeric > > instrument identifiers rather than more verbose string identifiers. So > > it could well be that you would want a FAST decoder to provide you > > with the numeric ids rather than the string variety. In the case that > > the feed doesn't present numeric ids you may even want the decoder to > > produce them ...) > > > > > So IMO this should not be added to FAST as it's really not fast at > > > all. If really it is added to FAST, let's at least force it to use > > > consecutive IDs. > > > > > > Note that a pre-shared map is just a big enum and that is OK of > > > course. > > > > > > Best regards, > > > > > > Marc [You can unsubscribe from this discussion group by sending a message to mailto:[email protected]] --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Financial Information eXchange" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/FIX-Protocol?hl=en -~----------~----~----~----~------~----~------~--~---
