Kurasake --
Could it be that you might have a hardware problem?
To test you RAM, run MemTest86+ at least overnight. Free from:
http://www.memtest.org/
and your hard drive(s), run SpinRite. $89US from:
http://www.grc.com/cs/prepurch.htm
This is an excellent product, though a bit expensive and geeky. Manual
is available at:
http://www.grc.com/files/sr5_manual.pdf
That is the latest manual, even though the latest version of SpinRite is
6.0.
And for everything you don't need to know about SpinRite:
http://www.grc.com/files/technote.pdf
You might read the following for more info on other HDD testing and
repairing ideas:
http://www.techsupportforum.com/hardware-support/hard-drive-support/461234-questions-about-spin-rite.html
-- Keith
On 4/30/2010 14:23, kurasake wrote:
Hi Patrick.
Thanks for taking the time to write so much.
I've cross checked a few symbols in AB vs QT and I'm definitely only
seeing the spikes in AB. For example for WDC I'll see the spike in the
daily chart in AB, I'll switch to the 1-minute chart in AB and see the
spike(s), note the date/time and look at the symbol in QT and scan
through the entire range of date/time in QT for WDC and everything is
normal. Another way of looking at it for me is that while the bad data
spikes don't happen all the time or on every symbol in AB, I've never
seen bad data spikes in QT (minute view, historical or raw data
window) so I'm fairly certain the issue has to be in in AB or
something related only to AB.
I would also tend to believe that if the bad data were consistent
across QT and AB that it would most likely be a problem in the IQFeed
data stream or the IQ agent in which case I would think it would be
affecting a large number of their customers.
Curiously, no matter what symbol, the spikes always seem to pop up 3-5
days prior and after I perform an explore in AA. So I could have
looked at WDC earlier, and everything could've been fine. Then I'll
run an Explore and then look at WDC again and the spikes would appear.
I also don't think I've ever seen the spikes occur for the current day
or for any day >5 days ago which is strange (though I'm not 100%
positive about this part). This further increases my suspicion that it
is something in AB.
The spikes also only seem to appear (or at least I notice it more)
when I've already looked at a large number of symbols, so I'm thinking
it might be an issue with the DB/memory cache but it's still hard for
me to rationalize how/why it happens. I've tried playing around with
the database and memory cache settings before but I guess I'll try
other combinations.
As for the large number of symbols, I wish I could settle on 200 or so
but my "system" (for lack of a better word) relies on a combination of
technicals and candles and I don't always get the right signals
triggered if the number of symbols I'm scanning across is too small. I
don't always have that many symbols though and there are days when I
don't need to perform an Explore and the spikes aren't so numerous as
to be overwhelming (though some days are worse then others).
I'll figure this out eventually.
Cheers!
--- In [email protected] <mailto:amibroker%40yahoogroups.com>,
"NW Trader" <pk47har...@...> wrote:
>
> Hi Kurasake,
>
> I know exactly what you are experiencing. However, the bad data is
in QT, but then once you have imported it to AB, it will persist. It
will most frequently occur when you do a backfill of some or all
stocks in QT, but depending on your data source and the spike settings
you have defined in QT, it can happen anytime -- especially on
volatile opening, closing, or FOMC days. And I see spikes virtually
every day on symbols like SPY just after the close. Be aware that the
spikes do not always appear on a QT chart, especially the instances of
a 0 value for a close on an intraday bar (like a one minute bar). But
they are there and you can find them by opening a QT data window
(<CTRL>D) and scrolling to the time of the bad data.
>
> Here is my solution. First I've created an exploration which I run
(daily setting in AA) on all symbols, at least once a day. The output
of the exploration is the current Close, the HHV of high for the past
21 days, the LLV of low for the past 21 days, and a 5 period ATR
divided by the close. Why 21? Well I have a version of QT which allows
20 days of historic data before it starts to drop off, so this will
cover all QT data available on a given day. Why the ATR/C ? This is a
quick volatility tool which can help spot when a stock may have a
split, an outlier data point, or some other radical move.
>
> So you run the scan and identify the stocks that have data that
needs correcting. Now open the Quote Editor in AB and sort the columns
(e.g., if you have a LLV(L,21) of 0, sort the lows and find the bars
where 0 exists. You could delete or edit those bars, and exit the
Quote Editor, however this won't fix things so long as the bad data is
in QT. So once you've identified the time of the bad data in AB, you
have to find and delete the bad data in QT. This can be accomplished
either with the QT data window (go to the bad data and delete), or
zoom the QT chart in on the time period with the bad quote (even tho
it may not be visible on a QT chart) and press <SHIFT> <CTRL> L to
remove a low point (may have to do this more than once if multiple
minutes are bad) or <SHIFT> <CTRL> H to remove a high quote.
>
> Splits can also be adjusted in QT, as well as in AB. See the help
files for each program, tho if you fix data in QT it should be ok in
AB. Again, always fix the QT data first, then AB.
>
> Finally a gratuitous observation & suggestion. IMHO, trying to track
1500 stocks intra day is neither worth it (given all the data
maintenance you will be doing) nor going to leave you much time to
trade. Consider narrowing the stocks you are following intraday to
those you realistically would trade based on price, volume, and your
other trading metrics. I now track approximately 200 symbols intraday,
importing them to AB, but I trade only about - 2 to 3 dozen regularly
(based on my Price/Volume and other metrics). There are an additional
4 dozen or so that are possible trade candidates that are not
regularly traded but may move into the regular camp as others drop
out. The remaining symbols are indexes, sector ETFs, and longer term
holds and swings that I want to generally keep an eye on. For me this
works well as I can thoroughly know the stocks I trade regularly; I
study their intraday support and resistance levels, daily patterns,
know when they report earn ings or have other news releases, and such.
I didn't always have 3 dozen traders -- years ago I tried to follow
hundreds, but realized that the concept of span of control was causing
me to miss many trades and make some bad decisions on trades I took.
So I focused on about a dozen stocks and over the years I've expanded
the number of my traders based on experience, knowledge of those
stocks, and developing a trading rhythm for how and when I trade.
>
> Of course if you want to track 1500 stocks for end of day study
purposes, consider either a separate data base set to daily data, or a
different data provider.
>
> This works for me, but your mileage may vary.
>
> Peace and Justice --- Patrick
> ----- Original Message -----
> From: kurasake
> To: [email protected] <mailto:amibroker%40yahoogroups.com>
> Sent: Thursday, April 29, 2010 12:19 PM
> Subject: [amibroker] Easy data refresh?
>
>
>
> I've been having a problem with bad data appearing in AB (abnormally
high volume or wrong OHLC) in AB when I perform an "Explore" from the
"Automatic Analysis" on a large number of symbols (>1500) when I have
"wait for backfill" checked even for something as simple as filter=C>O.
>
> Fortunately the bad data points are very easy to identify and seem
to occur on the same days for the affected symbols and since I haven't
been able to resolve it, I've sort of learned to live with it (Note, I
don't get any bad data in QuoteTracker running on the same machine
using the same RT data feed so I'm pretty sure it's in AB).
>
> Anyway, when I see the bad data, I right click on the "Plug-in
Status" area of the status bar and do a "Force Backfill" and all the
data is correctly filled in. My question is, is there an easier way,
either from the menu (so I can setup a macro) or better yet, via some
code/formula that will
> 1. Keep all the symbols but delete all the EOD and RT data
> 2. Automatically "Force Backfill" from my RT feed?
>
> If someone has an idea on what's causing the problem in the first
place, that would be helpful too but I've posted this regarding this
problem before and haven't seen a solution so I'm guessing I'm a
unique case.
>
> I've also tried flushing the DB cache and playing around with
different database parameters, in-memory cache size, setting up a new
database etc.
>
> Thanks.
>
> Roy
>
> System:
> Windows XP Pro/Sp3
> Intel Core i7, 4GB Ram, many gigs of HD free
> AB v5.3 (tried uninstall/reinstalling)
> RT Data from IQFeed using the latest client
>
>
>
>
> ------------------------------------
>
> **** IMPORTANT PLEASE READ ****
> This group is for the discussion between users only.
> This is *NOT* technical support channel.
>
> TO GET TECHNICAL SUPPORT send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> http://www.amibroker.com/feedback/ <http://www.amibroker.com/feedback/>
> (submissions sent via other channels won't be considered)
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/ <http://www.amibroker.com/devlog/>
>
> Yahoo! Groups Links
>
