This is similar to the approach used with our (and many others) ITCH protocol, but with one significant exception. We don't sync using a full replay of historic data, instead from a snapshot that represents the current state of the market at a current point in time (using some form of transaction identifier). The customer attaches to the real-time feed and start buffering the data, then collects a snapshot from a separate server. This is generally much cheaper to ship to the customer than a full replay. The customer takes the snapshot and its associated transaction id, dumps any buffered real-time updates that have occurred prior to the snapshot and continues from their.
Historical data is then handled as a separate service and pulled on demand as most UIs only need the historic data to handle charting. For the historical data we have pre-processed CSV files that we ship from the file system to the customer using HTTP. On 16 April 2017 at 21:33, Vero K. <[email protected]> wrote: > We use Message Broker to communicate between different service components. > One of the requirement, after one of the clients logs in, firstly, we need > to send him a large amount of historical data (trade data for the last day) > to init UI, after that client receives subsequent trade data updates in > real-time. All data provided by our back-end servers which should push data > to our server which is responsible for client data, servers communicate > over MQ. Given that it might be not a good idea to send a lot of data when > a member logs in over MQ because of MQ message size limitation, we either > need to split a historical message into single messages - but this will be > lots of data in the MQ which can affect the performance of our MQ. One more > approach is to fetch historical data with REST and receive subsequent > updates with MQ, but this can cause unsync situation: while we wait for > REST results to travel to the server responsible for handling front-end > clients, more trades might happen and we will need to run some > reconciliation logic to filter out trades from MQ in our REST message > response or apply them - doesn't not look quite good to me. how would you > approach this problem given you have MQ Broker (RabbitMQ)? We use java for > our project. > > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
