Hi Patrick Thanks for taking the time to reply.
I took the "option of last resort" and nuked my computer and reinstalled everything from scratch over the weekend. It was painful but hopefully (fingers crossed) worth it. I haven't re-imported all my symbols/charts and stuff etc yet but so far running an explore on >1000 seems to be working so I guess it was "something" with my computer in general... weird. To answer your question though, by "see" in QT, I meant visually. I didn't actually look at the data in the data window (I assumed the data in the chart in QT = the data in the data window, though I suppose that may not necessarily be the case). If this problem happens again, I suppose another option I could try (and probably should've tried) is to run only AB and see if the spikes come up when QT isn't running and vice-versa. Or to run another program which can tap into the IQfeed and see if there are any abnormalities in the data in that app as well. Thanks again for your feedback. Cheers! --- In [email protected], "NW Trader" <pk47har...@...> wrote: > > Hi Kurasake, > > When you say you don't "see" the spikes in QT, do you mean that a chart in QT > doesn't show them, or that when you open a data window (again to do so click > on the chart of the symbol and press <CTRL> D ) and scroll to the time that > AB says there is a spike there is no corresponding spike in the QT data? > > I ask because I often do not "see" a spike on a QT chart, however the data > when examined in a data window has the spike. This may be a function of the > spike filter setting in QT, however I've never asked Jerry Medved about it. > The bad data must be removed from QT before editing the AB database or it > will be reimported to AB on the next exploration. Once the data in AB is > older than the coverage in QT (in my case, 20 days), the AB data can be > edited safely as there is no chance for reimporting the bad data via an > exploration. > > I'm genuinely curious about what you are "seeing" as I have always found the > bad spikes in QT when I look at the data. > > Another possible solution would be to delete the data in QT for a particular > symbol (or that period which is affected) and then backfill the symbol, > however as I collect data at an interval of less than a minute and the > backfill is limited to a minute period, this is not an optimal solution for > my situation. > > Hope this helps with your mystery. > > Peace and Justice --- Patrick > ----- Original Message ----- > From: kurasake > To: [email protected] > Sent: Friday, April 30, 2010 11:23 AM > Subject: [amibroker] Re: Easy data refresh? > > > Hi Patrick. > > Thanks for taking the time to write so much. > > I've cross checked a few symbols in AB vs QT and I'm definitely only seeing > the spikes in AB. For example for WDC I'll see the spike in the daily chart > in AB, I'll switch to the 1-minute chart in AB and see the spike(s), note the > date/time and look at the symbol in QT and scan through the entire range of > date/time in QT for WDC and everything is normal. Another way of looking at > it for me is that while the bad data spikes don't happen all the time or on > every symbol in AB, I've never seen bad data spikes in QT (minute view, > historical or raw data window) so I'm fairly certain the issue has to be in > in AB or something related only to AB. > > I would also tend to believe that if the bad data were consistent across QT > and AB that it would most likely be a problem in the IQFeed data stream or > the IQ agent in which case I would think it would be affecting a large number > of their customers. > > Curiously, no matter what symbol, the spikes always seem to pop up 3-5 days > prior and after I perform an explore in AA. So I could have looked at WDC > earlier, and everything could've been fine. Then I'll run an Explore and > then look at WDC again and the spikes would appear. I also don't think I've > ever seen the spikes occur for the current day or for any day >5 days ago > which is strange (though I'm not 100% positive about this part). This > further increases my suspicion that it is something in AB. > > The spikes also only seem to appear (or at least I notice it more) when > I've already looked at a large number of symbols, so I'm thinking it might be > an issue with the DB/memory cache but it's still hard for me to rationalize > how/why it happens. I've tried playing around with the database and memory > cache settings before but I guess I'll try other combinations. > > As for the large number of symbols, I wish I could settle on 200 or so but > my "system" (for lack of a better word) relies on a combination of technicals > and candles and I don't always get the right signals triggered if the number > of symbols I'm scanning across is too small. I don't always have that many > symbols though and there are days when I don't need to perform an Explore and > the spikes aren't so numerous as to be overwhelming (though some days are > worse then others). > > I'll figure this out eventually. > > Cheers! > > > --- In [email protected], "NW Trader" <pk47hargus@> wrote: > > > > Hi Kurasake, > > > > I know exactly what you are experiencing. However, the bad data is in > QT, but then once you have imported it to AB, it will persist. It will most > frequently occur when you do a backfill of some or all stocks in QT, but > depending on your data source and the spike settings you have defined in QT, > it can happen anytime -- especially on volatile opening, closing, or FOMC > days. And I see spikes virtually every day on symbols like SPY just after > the close. Be aware that the spikes do not always appear on a QT chart, > especially the instances of a 0 value for a close on an intraday bar (like a > one minute bar). But they are there and you can find them by opening a QT > data window (<CTRL>D) and scrolling to the time of the bad data. > > > > Here is my solution. First I've created an exploration which I run > (daily setting in AA) on all symbols, at least once a day. The output of the > exploration is the current Close, the HHV of high for the past 21 days, the > LLV of low for the past 21 days, and a 5 period ATR divided by the close. Why > 21? Well I have a version of QT which allows 20 days of historic data before > it starts to drop off, so this will cover all QT data available on a given > day. Why the ATR/C ? This is a quick volatility tool which can help spot > when a stock may have a split, an outlier data point, or some other radical > move. > > > > So you run the scan and identify the stocks that have data that needs > correcting. Now open the Quote Editor in AB and sort the columns (e.g., if > you have a LLV(L,21) of 0, sort the lows and find the bars where 0 exists. > You could delete or edit those bars, and exit the Quote Editor, however this > won't fix things so long as the bad data is in QT. So once you've identified > the time of the bad data in AB, you have to find and delete the bad data in > QT. This can be accomplished either with the QT data window (go to the bad > data and delete), or zoom the QT chart in on the time period with the bad > quote (even tho it may not be visible on a QT chart) and press <SHIFT> <CTRL> > L to remove a low point (may have to do this more than once if multiple > minutes are bad) or <SHIFT> <CTRL> H to remove a high quote. > > > > Splits can also be adjusted in QT, as well as in AB. See the help files > for each program, tho if you fix data in QT it should be ok in AB. Again, > always fix the QT data first, then AB. > > > > Finally a gratuitous observation & suggestion. IMHO, trying to track > 1500 stocks intra day is neither worth it (given all the data maintenance you > will be doing) nor going to leave you much time to trade. Consider narrowing > the stocks you are following intraday to those you realistically would trade > based on price, volume, and your other trading metrics. I now track > approximately 200 symbols intraday, importing them to AB, but I trade only > about - 2 to 3 dozen regularly (based on my Price/Volume and other metrics). > There are an additional 4 dozen or so that are possible trade candidates that > are not regularly traded but may move into the regular camp as others drop > out. The remaining symbols are indexes, sector ETFs, and longer term holds > and swings that I want to generally keep an eye on. For me this works well > as I can thoroughly know the stocks I trade regularly; I study their intraday > support and resistance levels, daily patterns, know when they report earnings > or have other news releases, and such. I didn't always have 3 dozen traders > -- years ago I tried to follow hundreds, but realized that the concept of > span of control was causing me to miss many trades and make some bad > decisions on trades I took. So I focused on about a dozen stocks and over > the years I've expanded the number of my traders based on experience, > knowledge of those stocks, and developing a trading rhythm for how and when I > trade. > > > > Of course if you want to track 1500 stocks for end of day study purposes, > consider either a separate data base set to daily data, or a different data > provider. > > > > This works for me, but your mileage may vary. > > > > Peace and Justice --- Patrick > > ----- Original Message ----- > > From: kurasake > > To: [email protected] > > Sent: Thursday, April 29, 2010 12:19 PM > > Subject: [amibroker] Easy data refresh? > > > > > > > > I've been having a problem with bad data appearing in AB (abnormally > high volume or wrong OHLC) in AB when I perform an "Explore" from the > "Automatic Analysis" on a large number of symbols (>1500) when I have "wait > for backfill" checked even for something as simple as filter=C>O. > > > > Fortunately the bad data points are very easy to identify and seem to > occur on the same days for the affected symbols and since I haven't been able > to resolve it, I've sort of learned to live with it (Note, I don't get any > bad data in QuoteTracker running on the same machine using the same RT data > feed so I'm pretty sure it's in AB). > > > > Anyway, when I see the bad data, I right click on the "Plug-in Status" > area of the status bar and do a "Force Backfill" and all the data is > correctly filled in. My question is, is there an easier way, either from the > menu (so I can setup a macro) or better yet, via some code/formula that will > > 1. Keep all the symbols but delete all the EOD and RT data > > 2. Automatically "Force Backfill" from my RT feed? > > > > If someone has an idea on what's causing the problem in the first > place, that would be helpful too but I've posted this regarding this problem > before and haven't seen a solution so I'm guessing I'm a unique case. > > > > I've also tried flushing the DB cache and playing around with different > database parameters, in-memory cache size, setting up a new database etc. > > > > Thanks. > > > > Roy > > > > System: > > Windows XP Pro/Sp3 > > Intel Core i7, 4GB Ram, many gigs of HD free > > AB v5.3 (tried uninstall/reinstalling) > > RT Data from IQFeed using the latest client > > > > > > > > > > ------------------------------------ > > > > **** IMPORTANT PLEASE READ **** > > This group is for the discussion between users only. > > This is *NOT* technical support channel. > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > SUPPORT {at} amibroker.com > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > http://www.amibroker.com/feedback/ > > (submissions sent via other channels won't be considered) > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > http://www.amibroker.com/devlog/ > > > > Yahoo! Groups Links > > > > > > > ------------------------------------ > > **** IMPORTANT PLEASE READ **** > This group is for the discussion between users only. > This is *NOT* technical support channel. > > TO GET TECHNICAL SUPPORT send an e-mail directly to > SUPPORT {at} amibroker.com > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > http://www.amibroker.com/feedback/ > (submissions sent via other channels won't be considered) > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > http://www.amibroker.com/devlog/ > > Yahoo! Groups Links >
