Hi Keith I missed this thread so I apologize for the late reply.
I did a memtest86 and HD scan etc when I originally built the machine a little over a year or so ago. While I haven't checked the memory since, I regularly run various HD intensive stuff (defraggler, looking for dupe files across all drives, large backups/copies etc) and I do a pretty regular diskchk so that should be ok. I think the problem (which has since reappeared since I've rebuilt everything) is more a problem in code "somewhere" (in AB or otherwise) rather than a random bit in memory getting flipped (which I think would result in the OS crashing). While I haven't done any extensive tests, speaking anecdotally, I think the problem is more likely to come up if/when the quotation data cache size is at or exceeds the max value. Unfortunately, it takes a while to run an explore using backfills across so many symbols, so I don't/can't necessarily just hunker down to try and troubleshoot the problem. I hope to figure it out one day though. Thanks very much for your input. --- In [email protected], Keith McCombs <kmcco...@...> wrote: > > Kurasake -- > Could it be that you might have a hardware problem? > To test you RAM, run MemTest86+ at least overnight. Free from: > http://www.memtest.org/ > > and your hard drive(s), run SpinRite. $89US from: > http://www.grc.com/cs/prepurch.htm > This is an excellent product, though a bit expensive and geeky. Manual > is available at: > http://www.grc.com/files/sr5_manual.pdf > That is the latest manual, even though the latest version of SpinRite is > 6.0. > And for everything you don't need to know about SpinRite: > http://www.grc.com/files/technote.pdf > > You might read the following for more info on other HDD testing and > repairing ideas: > http://www.techsupportforum.com/hardware-support/hard-drive-support/461234-questions-about-spin-rite.html > > -- Keith > > On 4/30/2010 14:23, kurasake wrote: > > > > Hi Patrick. > > > > Thanks for taking the time to write so much. > > > > I've cross checked a few symbols in AB vs QT and I'm definitely only > > seeing the spikes in AB. For example for WDC I'll see the spike in the > > daily chart in AB, I'll switch to the 1-minute chart in AB and see the > > spike(s), note the date/time and look at the symbol in QT and scan > > through the entire range of date/time in QT for WDC and everything is > > normal. Another way of looking at it for me is that while the bad data > > spikes don't happen all the time or on every symbol in AB, I've never > > seen bad data spikes in QT (minute view, historical or raw data > > window) so I'm fairly certain the issue has to be in in AB or > > something related only to AB. > > > > I would also tend to believe that if the bad data were consistent > > across QT and AB that it would most likely be a problem in the IQFeed > > data stream or the IQ agent in which case I would think it would be > > affecting a large number of their customers. > > > > Curiously, no matter what symbol, the spikes always seem to pop up 3-5 > > days prior and after I perform an explore in AA. So I could have > > looked at WDC earlier, and everything could've been fine. Then I'll > > run an Explore and then look at WDC again and the spikes would appear. > > I also don't think I've ever seen the spikes occur for the current day > > or for any day >5 days ago which is strange (though I'm not 100% > > positive about this part). This further increases my suspicion that it > > is something in AB. > > > > The spikes also only seem to appear (or at least I notice it more) > > when I've already looked at a large number of symbols, so I'm thinking > > it might be an issue with the DB/memory cache but it's still hard for > > me to rationalize how/why it happens. I've tried playing around with > > the database and memory cache settings before but I guess I'll try > > other combinations. > > > > As for the large number of symbols, I wish I could settle on 200 or so > > but my "system" (for lack of a better word) relies on a combination of > > technicals and candles and I don't always get the right signals > > triggered if the number of symbols I'm scanning across is too small. I > > don't always have that many symbols though and there are days when I > > don't need to perform an Explore and the spikes aren't so numerous as > > to be overwhelming (though some days are worse then others). > > > > I'll figure this out eventually. > > > > Cheers! > > > > --- In [email protected] <mailto:amibroker%40yahoogroups.com>, > > "NW Trader" <pk47hargus@> wrote: > > > > > > Hi Kurasake, > > > > > > I know exactly what you are experiencing. However, the bad data is > > in QT, but then once you have imported it to AB, it will persist. It > > will most frequently occur when you do a backfill of some or all > > stocks in QT, but depending on your data source and the spike settings > > you have defined in QT, it can happen anytime -- especially on > > volatile opening, closing, or FOMC days. And I see spikes virtually > > every day on symbols like SPY just after the close. Be aware that the > > spikes do not always appear on a QT chart, especially the instances of > > a 0 value for a close on an intraday bar (like a one minute bar). But > > they are there and you can find them by opening a QT data window > > (<CTRL>D) and scrolling to the time of the bad data. > > > > > > Here is my solution. First I've created an exploration which I run > > (daily setting in AA) on all symbols, at least once a day. The output > > of the exploration is the current Close, the HHV of high for the past > > 21 days, the LLV of low for the past 21 days, and a 5 period ATR > > divided by the close. Why 21? Well I have a version of QT which allows > > 20 days of historic data before it starts to drop off, so this will > > cover all QT data available on a given day. Why the ATR/C ? This is a > > quick volatility tool which can help spot when a stock may have a > > split, an outlier data point, or some other radical move. > > > > > > So you run the scan and identify the stocks that have data that > > needs correcting. Now open the Quote Editor in AB and sort the columns > > (e.g., if you have a LLV(L,21) of 0, sort the lows and find the bars > > where 0 exists. You could delete or edit those bars, and exit the > > Quote Editor, however this won't fix things so long as the bad data is > > in QT. So once you've identified the time of the bad data in AB, you > > have to find and delete the bad data in QT. This can be accomplished > > either with the QT data window (go to the bad data and delete), or > > zoom the QT chart in on the time period with the bad quote (even tho > > it may not be visible on a QT chart) and press <SHIFT> <CTRL> L to > > remove a low point (may have to do this more than once if multiple > > minutes are bad) or <SHIFT> <CTRL> H to remove a high quote. > > > > > > Splits can also be adjusted in QT, as well as in AB. See the help > > files for each program, tho if you fix data in QT it should be ok in > > AB. Again, always fix the QT data first, then AB. > > > > > > Finally a gratuitous observation & suggestion. IMHO, trying to track > > 1500 stocks intra day is neither worth it (given all the data > > maintenance you will be doing) nor going to leave you much time to > > trade. Consider narrowing the stocks you are following intraday to > > those you realistically would trade based on price, volume, and your > > other trading metrics. I now track approximately 200 symbols intraday, > > importing them to AB, but I trade only about - 2 to 3 dozen regularly > > (based on my Price/Volume and other metrics). There are an additional > > 4 dozen or so that are possible trade candidates that are not > > regularly traded but may move into the regular camp as others drop > > out. The remaining symbols are indexes, sector ETFs, and longer term > > holds and swings that I want to generally keep an eye on. For me this > > works well as I can thoroughly know the stocks I trade regularly; I > > study their intraday support and resistance levels, daily patterns, > > know when they report earn ings or have other news releases, and such. > > I didn't always have 3 dozen traders -- years ago I tried to follow > > hundreds, but realized that the concept of span of control was causing > > me to miss many trades and make some bad decisions on trades I took. > > So I focused on about a dozen stocks and over the years I've expanded > > the number of my traders based on experience, knowledge of those > > stocks, and developing a trading rhythm for how and when I trade. > > > > > > Of course if you want to track 1500 stocks for end of day study > > purposes, consider either a separate data base set to daily data, or a > > different data provider. > > > > > > This works for me, but your mileage may vary. > > > > > > Peace and Justice --- Patrick > > > ----- Original Message ----- > > > From: kurasake > > > To: [email protected] <mailto:amibroker%40yahoogroups.com> > > > Sent: Thursday, April 29, 2010 12:19 PM > > > Subject: [amibroker] Easy data refresh? > > > > > > > > > > > > I've been having a problem with bad data appearing in AB (abnormally > > high volume or wrong OHLC) in AB when I perform an "Explore" from the > > "Automatic Analysis" on a large number of symbols (>1500) when I have > > "wait for backfill" checked even for something as simple as filter=C>O. > > > > > > Fortunately the bad data points are very easy to identify and seem > > to occur on the same days for the affected symbols and since I haven't > > been able to resolve it, I've sort of learned to live with it (Note, I > > don't get any bad data in QuoteTracker running on the same machine > > using the same RT data feed so I'm pretty sure it's in AB). > > > > > > Anyway, when I see the bad data, I right click on the "Plug-in > > Status" area of the status bar and do a "Force Backfill" and all the > > data is correctly filled in. My question is, is there an easier way, > > either from the menu (so I can setup a macro) or better yet, via some > > code/formula that will > > > 1. Keep all the symbols but delete all the EOD and RT data > > > 2. Automatically "Force Backfill" from my RT feed? > > > > > > If someone has an idea on what's causing the problem in the first > > place, that would be helpful too but I've posted this regarding this > > problem before and haven't seen a solution so I'm guessing I'm a > > unique case. > > > > > > I've also tried flushing the DB cache and playing around with > > different database parameters, in-memory cache size, setting up a new > > database etc. > > > > > > Thanks. > > > > > > Roy > > > > > > System: > > > Windows XP Pro/Sp3 > > > Intel Core i7, 4GB Ram, many gigs of HD free > > > AB v5.3 (tried uninstall/reinstalling) > > > RT Data from IQFeed using the latest client > > > > > > > > > > > > > > > ------------------------------------ > > > > > > **** IMPORTANT PLEASE READ **** > > > This group is for the discussion between users only. > > > This is *NOT* technical support channel. > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > SUPPORT {at} amibroker.com > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > http://www.amibroker.com/feedback/ <http://www.amibroker.com/feedback/> > > > (submissions sent via other channels won't be considered) > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > http://www.amibroker.com/devlog/ <http://www.amibroker.com/devlog/> > > > > > > Yahoo! Groups Links > > > > > > > >
