Herman,
Yes, it is a well known fact that some trades are reported long after
execution. This is one of the sources of "bad ticks". Some trades are
negotiated in big blocks and are not part of the "normal" order flow. They get
reported when they get reported -- and that could be some time later. There
are reporting rules that give a max time to report the trade.
Cleaned up data feeds and corrected bad ticks as you know are not desirable to
backtest realtime systems. The real time system has to be able to adapt to the
bad data.
BR,
Dennis
On Jun 26, 2010, at 9:00 AM, Herman wrote:
>
>
> That's an idea... never looked at anything else as stocks,might do some one
> of these days.
>
> btw, It is not an Internet delay, it must be that the market is releasing
> data out of sequence. This may be worse for tickers that are traded on
> different markets.
>
> herman
>
>
>
>
>
>
>
> Herman,
>
> I also use DTN IQFeed, however, I am only watching @ES# on a constant basis.
> I gave up on stocks, because of the poor realtime data quality. I did not
> know how to automate the process with all that random noise. I went to the
> highest volume futures because the feed was clean and fast. When I test a
> strategy, it trades the same way. However, I still manually input my trades
> and watch the order book and charts. I have seen nothing fishy.
>
> A 5 minute delay is not tradable. The best you can do is detect the
> anomalies and abort the strategy during bad delays.
>
> BR,
> Dennis
>
> On Jun 25, 2010, at 9:41 PM, Herman wrote:
>
>
>
> Thank you Dennis!
>
> Yes,i realize that there is an order queue you can't jump.
>
> In addition to your observation there is the fact that your orders may
> actually may change the market, draw the price one way or the other. That
> effect is impossible to Backtest. The only real test is to trade real money.
>
> I can live with all that, but FIVE MINUTES delay from market to my computer?
> I expected 15 seconds under heavy load... Five minutes seems really strange
> and changes everything.
>
> Can you think about an explanation for that?
>
> herman
>
>
>
>
>
>
> Herman,
>
> I assume that you are aware that you will not get a limit fill just because
> it touches your price. You have to stand in line behind all the other limit
> orders and will also be superseded by any market orders before you turn comes.
>
> It is only if the backtest shows exceeding the limit order by a tick that you
> know for sure that you would have been filled.
>
> The other issue, is that there is about a 2 second delay from the time your
> program sees the data until an order could be placed in reality. A really
> good high speed trading backtest needs to be done at the tick level, and the
> backtest has to assume a 2 second delay in placing the order at the exchange.
> Add any AB/AFL processing delay to that number also.
>
> Timestamps on data from the exchanges are not when the trade occurred, but
> when it is transmitted from the buffer to the internet. When things get
> really busy, they can delay sending out the data, but you can't know what the
> real delays were.
>
> Most high speed trading computer systems sit right on the exchange floor to
> eliminate these communication delays. A fast trading algorithm located
> across the internet must take into consideration the timestamps of the data
> relative to current time. Add to that the uncertainty that the timestamps do
> not even reflect the execution time, and you can see how this limits the
> practical trading speed of the system.
>
> In my trading, I use limit orders, but realize that I have to place them in
> advance to get a fill. I look at the order book when I place the order. If
> there are a lot of orders ahead of me at a certain price, I may change my
> price by a tick to increase my odds of a fill. When I see the volume of
> contracts filled start to approach the number when I placed my order, I know
> I am about to get filled.
>
> Not an easy thing to backtest.
>
> BR,
> Dennis
>
> On Jun 25, 2010, at 4:22 PM, Herman wrote:
>
>
>
> If anyone is trading real-time it might help you to know your data latency;
> it might explain why you don't get your LMT fills :-)
>
> I invite you to post possible explanations for the delays I experience, or
> the code I wrote. Remember to adjust the code if you live in a different
> timezone (subtract hours).
>
> I am using IQFeed RT data. My time-zone is EST (UTC 5:00), my computer clock
> is synchronized, I run AB at a 32% load factor, I ping DTN at about 134
> milliseconds.
> My Internet tested slow today, at about 1Mb/sec, but can that explain a 5
> minute delay at peak hours? I tested the system with 100 and 1800 tickers, no
> difference.
>
> The data delay ranges from < 1 sec to several minutes. The worst time
> appears to be around 3:45PM, i.e., 15 minutes before the market closes. See
> captures of my real-time scan at that time below. The results at about 3:45
> PM market time are scary! At better times it is usually 1-3 seconds.
>
> I'd like to know what your data latency is. Below is some test code that you
> can rework into a loop for your own system.
>
> Data latency at 3:45 PM:
> <int_1.jpg> <int_2.jpg>
>
> function TimeNumToSeconds( TN )
> {
> Seconds = int( TN % 100 );
> Minutes = int( TN / 100 % 100 );
> Hours = int( TN / 10000 % 100 );
> SecondNum = int( Hours * 3600 + Minutes * 60 + Seconds );
> return SecondNum;
> }
>
> RequestTimedRefresh( 1 );
> Now4 = Now( 4 );
> UpdateTime = StrToNum( "" + GetRTData( "UpdateTime" ) );
> UpdateSeconds = TimeNumtoseconds( UpdateTime );
> Now4Seconds = TimeNumToSeconds( Now4 );
>
> if ( UpdateSeconds >= 0 ) Latency = Now4Seconds - UpdateSeconds;
> else Latency = Null;
>
> if ( Latency >= 0 ) LatencyStr = NumToStr( Latency, 1.0, False );
> else LatencyStr = "-";
>
> Title = "Data latency: " + LatencyStr + " Sec.";
>
> <int_1.jpg><int_2.jpg>
>
>
>
>
>
>
>
>
>
