Thanks Sid. I'll give this a shot tomorrow. Cheers!
--- In [email protected], Sidney Kaiser <skbi...@...> wrote: > > For troubleshooting purposes go into QT and turn their spike off or set it to > a very big number. That way you can see the spikes on the QT chart and not > have AB running. > > Sid > > --- On Wed, 5/5/10, kurasake <kuras...@...> wrote: > > > From: kurasake <kuras...@...> > Subject: [amibroker] Re: Easy data refresh? > To: [email protected] > Date: Wednesday, May 5, 2010, 7:48 AM > > > Hi Patrick, > > In case you're still following this thread I just wanted to let you know that > I'm still having problems, despite the rebuild. > > And the problem seems to come up regardless of whether or not I have QT > running so I think the problem (at least in my case) is isolated to AB and/or > IQFeed. As far as seeing if it happens if QT is the only thing running, it's > not going to be realistic for me to confirm if the problem happens in QT if > AB isn't running since I never get a visual cue in QT that the data is > corrupt, which would mean I'd have to constantly monitor the QT data window > to numerically check for spikes. And since it doesn't happen frequently > enough or in every symbol, it becomes impractical. > > Anyway, no need to reply. Just wanted to give you an update. If I can't > narrow it down with tweaks in the memory/db cache then I'll consider opening > a trouble ticket with AB. > > Thanks again. > > --- In [email protected], "kurasake" <kurasake@> wrote: > > > > > > Hi Patrick > > > > Thanks for taking the time to reply. > > > > I took the "option of last resort" and nuked my computer and reinstalled > > everything from scratch over the weekend. It was painful but hopefully > > (fingers crossed) worth it. I haven't re-imported all my symbols/charts > > and stuff etc yet but so far running an explore on >1000 seems to be > > working so I guess it was "something" with my computer in general... weird. > > > > To answer your question though, by "see" in QT, I meant visually. I didn't > > actually look at the data in the data window (I assumed the data in the > > chart in QT = the data in the data window, though I suppose that may not > > necessarily be the case). > > > > If this problem happens again, I suppose another option I could try (and > > probably should've tried) is to run only AB and see if the spikes come up > > when QT isn't running and vice-versa. Or to run another program which can > > tap into the IQfeed and see if there are any abnormalities in the data in > > that app as well. > > > > Thanks again for your feedback. > > > > Cheers! > > > > > > > > --- In [email protected], "NW Trader" <pk47hargus@> wrote: > > > > > > Hi Kurasake, > > > > > > When you say you don't "see" the spikes in QT, do you mean that a chart > > > in QT doesn't show them, or that when you open a data window (again to do > > > so click on the chart of the symbol and press <CTRL> D ) and scroll to > > > the time that AB says there is a spike there is no corresponding spike in > > > the QT data? > > > > > > I ask because I often do not "see" a spike on a QT chart, however the > > > data when examined in a data window has the spike. This may be a function > > > of the spike filter setting in QT, however I've never asked Jerry Medved > > > about it. The bad data must be removed from QT before editing the AB > > > database or it will be reimported to AB on the next exploration. Once > > > the data in AB is older than the coverage in QT (in my case, 20 days), > > > the AB data can be edited safely as there is no chance for reimporting > > > the bad data via an exploration. > > > > > > I'm genuinely curious about what you are "seeing" as I have always found > > > the bad spikes in QT when I look at the data. > > > > > > Another possible solution would be to delete the data in QT for a > > > particular symbol (or that period which is affected) and then backfill > > > the symbol, however as I collect data at an interval of less than a > > > minute and the backfill is limited to a minute period, this is not an > > > optimal solution for my situation. > > > > > > Hope this helps with your mystery. > > > > > > Peace and Justice --- Patrick > > > ----- Original Message ----- > > > From: kurasake > > > To: [email protected] > > > Sent: Friday, April 30, 2010 11:23 AM > > > Subject: [amibroker] Re: Easy data refresh? > > > > > > > > > Hi Patrick. > > > > > > Thanks for taking the time to write so much. > > > > > > I've cross checked a few symbols in AB vs QT and I'm definitely only > > >seeing the spikes in AB. For example for WDC I'll see the spike in the > > >daily chart in AB, I'll switch to the 1-minute chart in AB and see the > > >spike(s), note the date/time and look at the symbol in QT and scan through > > >the entire range of date/time in QT for WDC and everything is normal. > > >Another way of looking at it for me is that while the bad data spikes > > >don't happen all the time or on every symbol in AB, I've never seen bad > > >data spikes in QT (minute view, historical or raw data window) so I'm > > >fairly certain the issue has to be in in AB or something related only to > > >AB. > > > > > > I would also tend to believe that if the bad data were consistent > > >across QT and AB that it would most likely be a problem in the IQFeed data > > >stream or the IQ agent in which case I would think it would be affecting a > > >large number of their customers. > > > > > > Curiously, no matter what symbol, the spikes always seem to pop up 3-5 > > >days prior and after I perform an explore in AA. So I could have looked > > >at WDC earlier, and everything could've been fine. Then I'll run an > > >Explore and then look at WDC again and the spikes would appear. I also > > >don't think I've ever seen the spikes occur for the current day or for any > > >day >5 days ago which is strange (though I'm not 100% positive about this > > >part). This further increases my suspicion that it is something in AB. > > > > > > The spikes also only seem to appear (or at least I notice it more) when > > >I've already looked at a large number of symbols, so I'm thinking it might > > >be an issue with the DB/memory cache but it's still hard for me to > > >rationalize how/why it happens. I've tried playing around with the > > >database and memory cache settings before but I guess I'll try other > > >combinations. > > > > > > As for the large number of symbols, I wish I could settle on 200 or so > > >but my "system" (for lack of a better word) relies on a combination of > > >technicals and candles and I don't always get the right signals triggered > > >if the number of symbols I'm scanning across is too small. I don't always > > >have that many symbols though and there are days when I don't need to > > >perform an Explore and the spikes aren't so numerous as to be overwhelming > > >(though some days are worse then others). > > > > > > I'll figure this out eventually. > > > > > > Cheers! > > > > > > > > > --- In [email protected], "NW Trader" <pk47hargus@> wrote: > > > > > > > > Hi Kurasake, > > > > > > > > I know exactly what you are experiencing. However, the bad data is > > >in QT, but then once you have imported it to AB, it will persist. It will > > >most frequently occur when you do a backfill of some or all stocks in QT, > > >but depending on your data source and the spike settings you have defined > > >in QT, it can happen anytime -- especially on volatile opening, closing, > > >or FOMC days. And I see spikes virtually every day on symbols like SPY > > >just after the close. Be aware that the spikes do not always appear on a > > >QT chart, especially the instances of a 0 value for a close on an intraday > > >bar (like a one minute bar). But they are there and you can find them by > > >opening a QT data window (<CTRL>D) and scrolling to the time of the bad > > >data. > > > > > > > > Here is my solution. First I've created an exploration which I run > > >(daily setting in AA) on all symbols, at least once a day. The output of > > >the exploration is the current Close, the HHV of high for the past 21 > > >days, the LLV of low for the past 21 days, and a 5 period ATR divided by > > >the close. Why 21? Well I have a version of QT which allows 20 days of > > >historic data before it starts to drop off, so this will cover all QT data > > >available on a given day. Why the ATR/C ? This is a quick volatility > > >tool which can help spot when a stock may have a split, an outlier data > > >point, or some other radical move. > > > > > > > > So you run the scan and identify the stocks that have data that needs > > >correcting. Now open the Quote Editor in AB and sort the columns (e.g., > > >if you have a LLV(L,21) of 0, sort the lows and find the bars where 0 > > >exists. You could delete or edit those bars, and exit the Quote Editor, > > >however this won't fix things so long as the bad data is in QT. So once > > >you've identified the time of the bad data in AB, you have to find and > > >delete the bad data in QT. This can be accomplished either with the QT > > >data window (go to the bad data and delete), or zoom the QT chart in on > > >the time period with the bad quote (even tho it may not be visible on a QT > > >chart) and press <SHIFT> <CTRL> L to remove a low point (may have to do > > >this more than once if multiple minutes are bad) or <SHIFT> <CTRL> H to > > >remove a high quote. > > > > > > > > Splits can also be adjusted in QT, as well as in AB. See the help > > >files for each program, tho if you fix data in QT it should be ok in AB. > > >Again, always fix the QT data first, then AB. > > > > > > > > Finally a gratuitous observation & suggestion. IMHO, trying to track > > >1500 stocks intra day is neither worth it (given all the data maintenance > > >you will be doing) nor going to leave you much time to trade. Consider > > >narrowing the stocks you are following intraday to those you realistically > > >would trade based on price, volume, and your other trading metrics. I now > > >track approximately 200 symbols intraday, importing them to AB, but I > > >trade only about - 2 to 3 dozen regularly (based on my Price/Volume and > > >other metrics). There are an additional 4 dozen or so that are possible > > >trade candidates that are not regularly traded but may move into the > > >regular camp as others drop out. The remaining symbols are indexes, > > >sector ETFs, and longer term holds and swings that I want to generally > > >keep an eye on. For me this works well as I can thoroughly know the > > >stocks I trade regularly; I study their intraday support and resistance > > >levels, daily > patterns, know when they report earnings or have other news releases, and > such. I didn't always have 3 dozen traders -- years ago I tried to follow > hundreds, but realized that the concept of span of control was causing me to > miss many trades and make some bad decisions on trades I took. So I focused > on about a dozen stocks and over the years I've expanded the number of my > traders based on experience, knowledge of those stocks, and developing a > trading rhythm for how and when I trade. > > > > > > > > Of course if you want to track 1500 stocks for end of day study > > >purposes, consider either a separate data base set to daily data, or a > > >different data provider. > > > > > > > > This works for me, but your mileage may vary. > > > > > > > > Peace and Justice --- Patrick > > > > ----- Original Message ----- > > > > From: kurasake > > > > To: [email protected] > > > > Sent: Thursday, April 29, 2010 12:19 PM > > > > Subject: [amibroker] Easy data refresh? > > > > > > > > > > > > > > > > I've been having a problem with bad data appearing in AB > > >(abnormally high volume or wrong OHLC) in AB when I perform an "Explore" > > >from the "Automatic Analysis" on a large number of symbols (>1500) when I > > >have "wait for backfill" checked even for something as simple as > > >filter=C>O. > > > > > > > > Fortunately the bad data points are very easy to identify and seem > > >to occur on the same days for the affected symbols and since I haven't > > >been able to resolve it, I've sort of learned to live with it (Note, I > > >don't get any bad data in QuoteTracker running on the same machine using > > >the same RT data feed so I'm pretty sure it's in AB). > > > > > > > > Anyway, when I see the bad data, I right click on the "Plug-in > > >Status" area of the status bar and do a "Force Backfill" and all the data > > >is correctly filled in. My question is, is there an easier way, either > > >from the menu (so I can setup a macro) or better yet, via some > > >code/formula that will > > > > 1. Keep all the symbols but delete all the EOD and RT data > > > > 2. Automatically "Force Backfill" from my RT feed? > > > > > > > > If someone has an idea on what's causing the problem in the first > > >place, that would be helpful too but I've posted this regarding this > > >problem before and haven't seen a solution so I'm guessing I'm a unique > > >case. > > > > > > > > I've also tried flushing the DB cache and playing around with > > >different database parameters, in-memory cache size, setting up a new > > >database etc. > > > > > > > > Thanks. > > > > > > > > Roy > > > > > > > > System: > > > > Windows XP Pro/Sp3 > > > > Intel Core i7, 4GB Ram, many gigs of HD free > > > > AB v5.3 (tried uninstall/reinstalling) > > > > RT Data from IQFeed using the latest client > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > **** IMPORTANT PLEASE READ **** > > > > This group is for the discussion between users only. > > > > This is *NOT* technical support channel. > > > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > SUPPORT {at} amibroker.com > > > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > http://www.amibroker.com/feedback/ > > > > (submissions sent via other channels won't be considered) > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > http://www.amibroker.com/devlog/ > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > **** IMPORTANT PLEASE READ **** > > > This group is for the discussion between users only. > > > This is *NOT* technical support channel. > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > SUPPORT {at} amibroker.com > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > http://www.amibroker.com/feedback/ > > > (submissions sent via other channels won't be considered) > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > http://www.amibroker.com/devlog/ > > > > > > Yahoo! Groups Links > > > > > > > > > > ------------------------------------ > > **** IMPORTANT PLEASE READ **** > This group is for the discussion between users only. > This is *NOT* technical support channel. > > TO GET TECHNICAL SUPPORT send an e-mail directly to > SUPPORT {at} amibroker.com > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > http://www.amibroker.com/feedback/ > (submissions sent via other channels won't be considered) > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > http://www.amibroker.com/devlog/ > > Yahoo! Groups Links >
