I am not sure about the technical details, but I can say that until now on today's current market volume, what was taking 6-7 minutes yesterday is taking around 1m 20sec... that's an incredible increase in performance.
Everything else seems stable and responsive. I am not sure about the 64bit RAM drive, but I am running 64bit windows Xp and 32b ones I downloaded didn't work. I am not sure whether a 32bit system is capable of creating a RAM drive beyond the 4GB limit... cannot test that as I haven't got a 32bit OS here. cheers, eToke --- In [email protected], "sidhartha70" <sidharth...@...> wrote: > > Very interesting. So what's taking the time here...?? Streaming the data onto > your HD...? Hence you remove that bottleneck by moving your HD into RAM...? > > Why would you want to a 64 bit compatible RAM Drive utility...? You are > running 32 bit AB right...? How did you create your current 4GB RAM HDD...? > > Thanks > > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > All, > > > > I was able to drastically improve the performance of the exploration during > > market hours from 6 minutes to just over a minute!! That is an incredible > > 6X improvement! > > > > Outside of market hours I was able to get over 2X improvement. > > > > I have 8GB of RAM on my 64bit machine yet I have to run 32bit AB due to the > > fact that IQFeed do not support a 64bit plugin - hence half my RAM is > > practically unusable. > > > > My AB database is currently approx 3.75GB. This is what I did: I created a > > 4GB RAMDRIVE (this will take 4GB of RAM and create a virtual HDD), then > > copied the entire AB folder into it, and ran the program from there. The > > result was a fascinating improvement: everything runs just so much faster. > > > > Since I am running a large 1 minute DB I keep my cache symbols low to 11. > > This I suspect causes a lot of HDD interaction under normal circumstances, > > but since the HDD is now in RAM, this isn't an issue any more. > > > > Also, the physical HDD is now silent and hardly active any longer. Over all > > I think this saves some life out of it as a tick-by-tick update every > > single market day cannot do it much good. > > > > There are various free/cheap RAM Drive utilities out there. Most > > importantly one would need to pick up a 64bit compatible one. I have not > > done any research whatsoever on these programs... I just picked up the > > first one I found. > > > > There are downsides of course: most notably is that you would lose all the > > data if the power goes down, however since I am planning to use it this way > > throughout market hours, it is not an issue for me. I can take the latest > > backup and update it with a backfill run. > > > > I am curious to see whether anybody has experimented with this before and > > can compare results. > > > > Also, anybody aware of other potential downsides when using this method? > > > > cheers, > > eToke > > > > --- In [email protected], "Paul Ho" <paul.tsho@> wrote: > > > > > > The following test would help you determine which "elements" are critical > > > to improving exporation speed > > > 1. > > > Temporary change your IQFeed database from IQFeed to local, make sure > > > local storage is enabled. run your exploration, That is the speed your > > > hardware can do with your afl and with no limitation on connection > > > bandwidth. > > > 2. I suspect that IQFeed connection at application level, not at adsl > > > level, is your bottle neck. To improve it you have to consider: > > > 3. Backfilling is far more time consuming than using RTGetData > > > 4. The more array data your afl uses, the slower it is. > > > 5. There is only certain amount improvement you can make with new > > > computer hardware. most of the improvements can be gained by reorganising > > > your afl so things that doesnt change change intraday are scanned > > > overnight, and use RTGETDATA as a replacement to array data as much as > > > possible. > > > > > > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > > > > > Hi Tomasz, > > > > > > > > Agreed on load imposed by the rapid update of 500 symbols. No question > > > > about it. > > > > > > > > What I would like to understand is what elements (hardware, or system > > > > optimisation) would help reduce the duration of a scan. > > > > > > > > Unfortunately running the 64bit version is out of the question because > > > > of the realtime plugin issue. This rules out the possibility of adding > > > > a large amount of RAM although I don't know whether that alone would > > > > help. > > > > > > > > Are there any other ways I might be able to increase the speed? Would > > > > copying the entire db on several GB of a RAM disk before a market > > > > session be worth trying? > > > > > > > > Best regards, > > > > eToke > > > > > > > > > > > > > > > > > > > > > > > > --- In [email protected], "Tomasz Janeczko" <groups@> wrote: > > > > > > > > > > Hello, > > > > > > > > > > You need to understand that there can be even 800 ticks PER SECOND on > > > > > one symbol > > > > > If you are for example following 500 symbols, it presents the stream > > > > > of 400000 updates per second. > > > > > Even when processing took as little as 2 microseconds per update (1 > > > > > millionth part of the second) > > > > > this would mean 80% CPU utilisation. So yes, if there are lots of > > > > > quotes streaming > > > > > on lots of symbols you will see higher CPU usage and possible also > > > > > slower exploration. > > > > > > > > > > Best regards, > > > > > Tomasz Janeczko > > > > > amibroker.com > > > > > ----- Original Message ----- > > > > > From: "etoketrader" <etoke@> > > > > > To: <[email protected]> > > > > > Sent: Tuesday, August 04, 2009 6:09 PM > > > > > Subject: [amibroker] Re: Speeding exporation intraday > > > > > > > > > > > > > > > > hi Tomasz, > > > > > > > > > > > > that makes sense. > > > > > > > > > > > > However in my case all my symbols are already active both on and > > > > > > off market hours. > > > > > > > > > > > > When quotes are being updated constantly during market hours, a > > > > > > test exploration code which displays the closing price takes > > > > > > approx. 6 - 10 times the duration of when the quotes are also > > > > > > streaming but outside of market hours (i.e. hardly any updates). > > > > > > > > > > > > My question is if there is anything I can do to speed this up. > > > > > > > > > > > > BR > > > > > > > > > > > > eToke > > > > > > > > > > > > > > > > > > > > > > > > --- In [email protected], "Tomasz Janeczko" <groups@> wrote: > > > > > >> > > > > > >> Hello, > > > > > >> > > > > > >> That depends if given symbol is in the "active" (i.e. streaming) > > > > > >> list or not. > > > > > >> If it is - then turning on/off "wait for backfill" does not make > > > > > >> difference. > > > > > >> > > > > > >> If it is not, then with "wait for backfill" turned on it will of > > > > > >> course wait for "missing" bars > > > > > >> before proceeding to next symbol. > > > > > >> > > > > > >> Charts on the other hand, always refresh immediatelly with data > > > > > >> that are present right now. > > > > > >> If symbol is not present in the active list it is automatically > > > > > >> added and backfill request > > > > > >> may be sent but this does not block chart from refreshing - it > > > > > >> will refresh immediatelly, > > > > > >> and once backfill data eventually arrives it will refresh once > > > > > >> more. > > > > > >> In other words - charts are asynchronous with regards to backfill. > > > > > >> AA can be synchronous > > > > > >> ("wait for backfill" turned ON) or asynchronous ("wait for > > > > > >> backfill" turned OFF). > > > > > >> > > > > > >> That's the only difference. AFL execution IS IDENTICAL regardless > > > > > >> where it occurs. > > > > > >> Of course with different zoom levels / different AA range you may > > > > > >> be using different number of bars, > > > > > >> but that should be obvious. > > > > > >> > > > > > >> Best regards, > > > > > >> Tomasz Janeczko > > > > > >> amibroker.com > > > > > >> ----- Original Message ----- > > > > > >> From: "sidhartha70" <sidhartha70@> > > > > > >> To: <[email protected]> > > > > > >> Sent: Tuesday, August 04, 2009 5:14 PM > > > > > >> Subject: [amibroker] Re: Speeding exporation intraday > > > > > >> > > > > > >> > > > > > >> > I'd be interested to hear TJ's answer on this eToke... > > > > > >> > > > > > > >> > Because I was asking from very similar questions of support & TJ > > > > > >> > recently... namely that equivalent AFL on charts seems to > > > > > >> > execute > > > > > >> > much quicker than a scan or exploration. Which obviously leads > > > > > >> > one to think that other operations that occur in a scan or > > > > > >> > exploration, outside of the AFL execution, are what take the > > > > > >> > time... > > > > > >> > > > > > > >> > Why this time is greater with a streaming feed I don't know... > > > > > >> > > > > > > >> > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > > >> >> > > > > > >> >> Hi, for testing I am currently running: > > > > > >> >> > > > > > >> >> n Last quotations = 1 > > > > > >> >> A very basic AFL code which lists closing price. > > > > > >> >> QuickAFL turned on > > > > > >> >> > > > > > >> >> I am concerned about the comparison between how it runs outside > > > > > >> >> of market hours vs how it runs during. > > > > > >> >> > > > > > >> >> BR, > > > > > >> >> eToke > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> --- In [email protected], "sidhartha70" <sidhartha70@> > > > > > >> >> wrote: > > > > > >> >> > > > > > > >> >> > How many quotations are you running the explorations over...? > > > > > >> >> > > > > > > >> >> > --- In [email protected], "etoketrader" <etoke@> > > > > > >> >> > wrote: > > > > > >> >> > > > > > > > >> >> > > Tried that. I always run the first backfill before market > > > > > >> >> > > hours. After that, backfill setting does not seem to make > > > > > >> >> > > any > > > > > >> >> > > difference as long as everything has already been > > > > > >> >> > > backfilled and all symbols are streaming. > > > > > >> >> > > > > > > > >> >> > > BR > > > > > >> >> > > eToke > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > --- In [email protected], "murthysuresh" > > > > > >> >> > > <murthysuresh@> wrote: > > > > > >> >> > > > > > > > > >> >> > > > backfill before the start of the market hours. turn off > > > > > >> >> > > > backfill as you only need to do it at the beginning of > > > > > >> >> > > > the > > > > > >> >> > > > session. > > > > > >> >> > > > > > > > > >> >> > > > --- In [email protected], "etoketrader" <etoke@> > > > > > >> >> > > > wrote: > > > > > >> >> > > > > > > > > > >> >> > > > > Hi, > > > > > >> >> > > > > > > > > > >> >> > > > > I currently run IQFeed on 500 symbols. Exploration > > > > > >> >> > > > > outside of market hours run under 1 minute but slow > > > > > >> >> > > > > down to 6 > > > > > >> >> > > > > minutes > > > > > >> >> > > > > during market hours. > > > > > >> >> > > > > > > > > > >> >> > > > > Can anybody recommend some ideal settings to speed up > > > > > >> >> > > > > intraday explorations while streaming? > > > > > >> >> > > > > > > > > > >> >> > > > > thanks, > > > > > >> >> > > > > eToke > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > ------------------------------------ > > > > > >> > > > > > > >> > **** IMPORTANT PLEASE READ **** > > > > > >> > This group is for the discussion between users only. > > > > > >> > This is *NOT* technical support channel. > > > > > >> > > > > > > >> > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > > >> > SUPPORT {at} amibroker.com > > > > > >> > > > > > > >> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > > >> > http://www.amibroker.com/feedback/ > > > > > >> > (submissions sent via other channels won't be considered) > > > > > >> > > > > > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > > >> > http://www.amibroker.com/devlog/ > > > > > >> > > > > > > >> > Yahoo! Groups Links > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > > > **** IMPORTANT PLEASE READ **** > > > > > > This group is for the discussion between users only. > > > > > > This is *NOT* technical support channel. > > > > > > > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > > > SUPPORT {at} amibroker.com > > > > > > > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > > > http://www.amibroker.com/feedback/ > > > > > > (submissions sent via other channels won't be considered) > > > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > > > http://www.amibroker.com/devlog/ > > > > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
