Very interesting. So what's taking the time here...?? Streaming the data onto your HD...? Hence you remove that bottleneck by moving your HD into RAM...?
Why would you want to a 64 bit compatible RAM Drive utility...? You are running 32 bit AB right...? How did you create your current 4GB RAM HDD...? Thanks --- In [email protected], "etoketrader" <et...@...> wrote: > > All, > > I was able to drastically improve the performance of the exploration during > market hours from 6 minutes to just over a minute!! That is an incredible 6X > improvement! > > Outside of market hours I was able to get over 2X improvement. > > I have 8GB of RAM on my 64bit machine yet I have to run 32bit AB due to the > fact that IQFeed do not support a 64bit plugin - hence half my RAM is > practically unusable. > > My AB database is currently approx 3.75GB. This is what I did: I created a > 4GB RAMDRIVE (this will take 4GB of RAM and create a virtual HDD), then > copied the entire AB folder into it, and ran the program from there. The > result was a fascinating improvement: everything runs just so much faster. > > Since I am running a large 1 minute DB I keep my cache symbols low to 11. > This I suspect causes a lot of HDD interaction under normal circumstances, > but since the HDD is now in RAM, this isn't an issue any more. > > Also, the physical HDD is now silent and hardly active any longer. Over all I > think this saves some life out of it as a tick-by-tick update every single > market day cannot do it much good. > > There are various free/cheap RAM Drive utilities out there. Most importantly > one would need to pick up a 64bit compatible one. I have not done any > research whatsoever on these programs... I just picked up the first one I > found. > > There are downsides of course: most notably is that you would lose all the > data if the power goes down, however since I am planning to use it this way > throughout market hours, it is not an issue for me. I can take the latest > backup and update it with a backfill run. > > I am curious to see whether anybody has experimented with this before and can > compare results. > > Also, anybody aware of other potential downsides when using this method? > > cheers, > eToke > > --- In [email protected], "Paul Ho" <paul.tsho@> wrote: > > > > The following test would help you determine which "elements" are critical > > to improving exporation speed > > 1. > > Temporary change your IQFeed database from IQFeed to local, make sure local > > storage is enabled. run your exploration, That is the speed your hardware > > can do with your afl and with no limitation on connection bandwidth. > > 2. I suspect that IQFeed connection at application level, not at adsl > > level, is your bottle neck. To improve it you have to consider: > > 3. Backfilling is far more time consuming than using RTGetData > > 4. The more array data your afl uses, the slower it is. > > 5. There is only certain amount improvement you can make with new computer > > hardware. most of the improvements can be gained by reorganising your afl > > so things that doesnt change change intraday are scanned overnight, and use > > RTGETDATA as a replacement to array data as much as possible. > > > > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > > > Hi Tomasz, > > > > > > Agreed on load imposed by the rapid update of 500 symbols. No question > > > about it. > > > > > > What I would like to understand is what elements (hardware, or system > > > optimisation) would help reduce the duration of a scan. > > > > > > Unfortunately running the 64bit version is out of the question because of > > > the realtime plugin issue. This rules out the possibility of adding a > > > large amount of RAM although I don't know whether that alone would help. > > > > > > Are there any other ways I might be able to increase the speed? Would > > > copying the entire db on several GB of a RAM disk before a market session > > > be worth trying? > > > > > > Best regards, > > > eToke > > > > > > > > > > > > > > > > > > --- In [email protected], "Tomasz Janeczko" <groups@> wrote: > > > > > > > > Hello, > > > > > > > > You need to understand that there can be even 800 ticks PER SECOND on > > > > one symbol > > > > If you are for example following 500 symbols, it presents the stream of > > > > 400000 updates per second. > > > > Even when processing took as little as 2 microseconds per update (1 > > > > millionth part of the second) > > > > this would mean 80% CPU utilisation. So yes, if there are lots of > > > > quotes streaming > > > > on lots of symbols you will see higher CPU usage and possible also > > > > slower exploration. > > > > > > > > Best regards, > > > > Tomasz Janeczko > > > > amibroker.com > > > > ----- Original Message ----- > > > > From: "etoketrader" <etoke@> > > > > To: <[email protected]> > > > > Sent: Tuesday, August 04, 2009 6:09 PM > > > > Subject: [amibroker] Re: Speeding exporation intraday > > > > > > > > > > > > > hi Tomasz, > > > > > > > > > > that makes sense. > > > > > > > > > > However in my case all my symbols are already active both on and off > > > > > market hours. > > > > > > > > > > When quotes are being updated constantly during market hours, a test > > > > > exploration code which displays the closing price takes > > > > > approx. 6 - 10 times the duration of when the quotes are also > > > > > streaming but outside of market hours (i.e. hardly any updates). > > > > > > > > > > My question is if there is anything I can do to speed this up. > > > > > > > > > > BR > > > > > > > > > > eToke > > > > > > > > > > > > > > > > > > > > --- In [email protected], "Tomasz Janeczko" <groups@> wrote: > > > > >> > > > > >> Hello, > > > > >> > > > > >> That depends if given symbol is in the "active" (i.e. streaming) > > > > >> list or not. > > > > >> If it is - then turning on/off "wait for backfill" does not make > > > > >> difference. > > > > >> > > > > >> If it is not, then with "wait for backfill" turned on it will of > > > > >> course wait for "missing" bars > > > > >> before proceeding to next symbol. > > > > >> > > > > >> Charts on the other hand, always refresh immediatelly with data that > > > > >> are present right now. > > > > >> If symbol is not present in the active list it is automatically > > > > >> added and backfill request > > > > >> may be sent but this does not block chart from refreshing - it will > > > > >> refresh immediatelly, > > > > >> and once backfill data eventually arrives it will refresh once more. > > > > >> In other words - charts are asynchronous with regards to backfill. > > > > >> AA can be synchronous > > > > >> ("wait for backfill" turned ON) or asynchronous ("wait for backfill" > > > > >> turned OFF). > > > > >> > > > > >> That's the only difference. AFL execution IS IDENTICAL regardless > > > > >> where it occurs. > > > > >> Of course with different zoom levels / different AA range you may be > > > > >> using different number of bars, > > > > >> but that should be obvious. > > > > >> > > > > >> Best regards, > > > > >> Tomasz Janeczko > > > > >> amibroker.com > > > > >> ----- Original Message ----- > > > > >> From: "sidhartha70" <sidhartha70@> > > > > >> To: <[email protected]> > > > > >> Sent: Tuesday, August 04, 2009 5:14 PM > > > > >> Subject: [amibroker] Re: Speeding exporation intraday > > > > >> > > > > >> > > > > >> > I'd be interested to hear TJ's answer on this eToke... > > > > >> > > > > > >> > Because I was asking from very similar questions of support & TJ > > > > >> > recently... namely that equivalent AFL on charts seems to > > > > >> > execute > > > > >> > much quicker than a scan or exploration. Which obviously leads one > > > > >> > to think that other operations that occur in a scan or > > > > >> > exploration, outside of the AFL execution, are what take the > > > > >> > time... > > > > >> > > > > > >> > Why this time is greater with a streaming feed I don't know... > > > > >> > > > > > >> > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > >> >> > > > > >> >> Hi, for testing I am currently running: > > > > >> >> > > > > >> >> n Last quotations = 1 > > > > >> >> A very basic AFL code which lists closing price. > > > > >> >> QuickAFL turned on > > > > >> >> > > > > >> >> I am concerned about the comparison between how it runs outside > > > > >> >> of market hours vs how it runs during. > > > > >> >> > > > > >> >> BR, > > > > >> >> eToke > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> --- In [email protected], "sidhartha70" <sidhartha70@> > > > > >> >> wrote: > > > > >> >> > > > > > >> >> > How many quotations are you running the explorations over...? > > > > >> >> > > > > > >> >> > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > >> >> > > > > > > >> >> > > Tried that. I always run the first backfill before market > > > > >> >> > > hours. After that, backfill setting does not seem to make any > > > > >> >> > > difference as long as everything has already been backfilled > > > > >> >> > > and all symbols are streaming. > > > > >> >> > > > > > > >> >> > > BR > > > > >> >> > > eToke > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > --- In [email protected], "murthysuresh" > > > > >> >> > > <murthysuresh@> wrote: > > > > >> >> > > > > > > > >> >> > > > backfill before the start of the market hours. turn off > > > > >> >> > > > backfill as you only need to do it at the beginning of the > > > > >> >> > > > session. > > > > >> >> > > > > > > > >> >> > > > --- In [email protected], "etoketrader" <etoke@> > > > > >> >> > > > wrote: > > > > >> >> > > > > > > > > >> >> > > > > Hi, > > > > >> >> > > > > > > > > >> >> > > > > I currently run IQFeed on 500 symbols. Exploration > > > > >> >> > > > > outside of market hours run under 1 minute but slow down > > > > >> >> > > > > to 6 > > > > >> >> > > > > minutes > > > > >> >> > > > > during market hours. > > > > >> >> > > > > > > > > >> >> > > > > Can anybody recommend some ideal settings to speed up > > > > >> >> > > > > intraday explorations while streaming? > > > > >> >> > > > > > > > > >> >> > > > > thanks, > > > > >> >> > > > > eToke > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > ------------------------------------ > > > > >> > > > > > >> > **** IMPORTANT PLEASE READ **** > > > > >> > This group is for the discussion between users only. > > > > >> > This is *NOT* technical support channel. > > > > >> > > > > > >> > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > >> > SUPPORT {at} amibroker.com > > > > >> > > > > > >> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > >> > http://www.amibroker.com/feedback/ > > > > >> > (submissions sent via other channels won't be considered) > > > > >> > > > > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > >> > http://www.amibroker.com/devlog/ > > > > >> > > > > > >> > Yahoo! Groups Links > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > **** IMPORTANT PLEASE READ **** > > > > > This group is for the discussion between users only. > > > > > This is *NOT* technical support channel. > > > > > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > > SUPPORT {at} amibroker.com > > > > > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > > http://www.amibroker.com/feedback/ > > > > > (submissions sent via other channels won't be considered) > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > > http://www.amibroker.com/devlog/ > > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > >
