Thanks. But what did you use to create the RAM drive HDD on your machine currently...?
TJ, anything to be aware of using this approach...? TIA --- In [email protected], "etoketrader" <et...@...> wrote: > > I am not sure about the technical details, but I can say that until now on > today's current market volume, what was taking 6-7 minutes yesterday is > taking around 1m 20sec... that's an incredible increase in performance. > > Everything else seems stable and responsive. > > I am not sure about the 64bit RAM drive, but I am running 64bit windows Xp > and 32b ones I downloaded didn't work. I am not sure whether a 32bit system > is capable of creating a RAM drive beyond the 4GB limit... cannot test that > as I haven't got a 32bit OS here. > > cheers, > eToke > > --- In [email protected], "sidhartha70" <sidhartha70@> wrote: > > > > Very interesting. So what's taking the time here...?? Streaming the data > > onto your HD...? Hence you remove that bottleneck by moving your HD into > > RAM...? > > > > Why would you want to a 64 bit compatible RAM Drive utility...? You are > > running 32 bit AB right...? How did you create your current 4GB RAM HDD...? > > > > Thanks > > > > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > > > All, > > > > > > I was able to drastically improve the performance of the exploration > > > during market hours from 6 minutes to just over a minute!! That is an > > > incredible 6X improvement! > > > > > > Outside of market hours I was able to get over 2X improvement. > > > > > > I have 8GB of RAM on my 64bit machine yet I have to run 32bit AB due to > > > the fact that IQFeed do not support a 64bit plugin - hence half my RAM is > > > practically unusable. > > > > > > My AB database is currently approx 3.75GB. This is what I did: I created > > > a 4GB RAMDRIVE (this will take 4GB of RAM and create a virtual HDD), then > > > copied the entire AB folder into it, and ran the program from there. The > > > result was a fascinating improvement: everything runs just so much faster. > > > > > > Since I am running a large 1 minute DB I keep my cache symbols low to 11. > > > This I suspect causes a lot of HDD interaction under normal > > > circumstances, but since the HDD is now in RAM, this isn't an issue any > > > more. > > > > > > Also, the physical HDD is now silent and hardly active any longer. Over > > > all I think this saves some life out of it as a tick-by-tick update every > > > single market day cannot do it much good. > > > > > > There are various free/cheap RAM Drive utilities out there. Most > > > importantly one would need to pick up a 64bit compatible one. I have not > > > done any research whatsoever on these programs... I just picked up the > > > first one I found. > > > > > > There are downsides of course: most notably is that you would lose all > > > the data if the power goes down, however since I am planning to use it > > > this way throughout market hours, it is not an issue for me. I can take > > > the latest backup and update it with a backfill run. > > > > > > I am curious to see whether anybody has experimented with this before and > > > can compare results. > > > > > > Also, anybody aware of other potential downsides when using this method? > > > > > > cheers, > > > eToke > > > > > > --- In [email protected], "Paul Ho" <paul.tsho@> wrote: > > > > > > > > The following test would help you determine which "elements" are > > > > critical to improving exporation speed > > > > 1. > > > > Temporary change your IQFeed database from IQFeed to local, make sure > > > > local storage is enabled. run your exploration, That is the speed your > > > > hardware can do with your afl and with no limitation on connection > > > > bandwidth. > > > > 2. I suspect that IQFeed connection at application level, not at adsl > > > > level, is your bottle neck. To improve it you have to consider: > > > > 3. Backfilling is far more time consuming than using RTGetData > > > > 4. The more array data your afl uses, the slower it is. > > > > 5. There is only certain amount improvement you can make with new > > > > computer hardware. most of the improvements can be gained by > > > > reorganising your afl so things that doesnt change change intraday are > > > > scanned overnight, and use RTGETDATA as a replacement to array data as > > > > much as possible. > > > > > > > > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > > > > > > > Hi Tomasz, > > > > > > > > > > Agreed on load imposed by the rapid update of 500 symbols. No > > > > > question about it. > > > > > > > > > > What I would like to understand is what elements (hardware, or system > > > > > optimisation) would help reduce the duration of a scan. > > > > > > > > > > Unfortunately running the 64bit version is out of the question > > > > > because of the realtime plugin issue. This rules out the possibility > > > > > of adding a large amount of RAM although I don't know whether that > > > > > alone would help. > > > > > > > > > > Are there any other ways I might be able to increase the speed? Would > > > > > copying the entire db on several GB of a RAM disk before a market > > > > > session be worth trying? > > > > > > > > > > Best regards, > > > > > eToke > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --- In [email protected], "Tomasz Janeczko" <groups@> wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > You need to understand that there can be even 800 ticks PER SECOND > > > > > > on one symbol > > > > > > If you are for example following 500 symbols, it presents the > > > > > > stream of 400000 updates per second. > > > > > > Even when processing took as little as 2 microseconds per update (1 > > > > > > millionth part of the second) > > > > > > this would mean 80% CPU utilisation. So yes, if there are lots of > > > > > > quotes streaming > > > > > > on lots of symbols you will see higher CPU usage and possible also > > > > > > slower exploration. > > > > > > > > > > > > Best regards, > > > > > > Tomasz Janeczko > > > > > > amibroker.com > > > > > > ----- Original Message ----- > > > > > > From: "etoketrader" <etoke@> > > > > > > To: <[email protected]> > > > > > > Sent: Tuesday, August 04, 2009 6:09 PM > > > > > > Subject: [amibroker] Re: Speeding exporation intraday > > > > > > > > > > > > > > > > > > > hi Tomasz, > > > > > > > > > > > > > > that makes sense. > > > > > > > > > > > > > > However in my case all my symbols are already active both on and > > > > > > > off market hours. > > > > > > > > > > > > > > When quotes are being updated constantly during market hours, a > > > > > > > test exploration code which displays the closing price takes > > > > > > > approx. 6 - 10 times the duration of when the quotes are also > > > > > > > streaming but outside of market hours (i.e. hardly any updates). > > > > > > > > > > > > > > My question is if there is anything I can do to speed this up. > > > > > > > > > > > > > > BR > > > > > > > > > > > > > > eToke > > > > > > > > > > > > > > > > > > > > > > > > > > > > --- In [email protected], "Tomasz Janeczko" <groups@> > > > > > > > wrote: > > > > > > >> > > > > > > >> Hello, > > > > > > >> > > > > > > >> That depends if given symbol is in the "active" (i.e. streaming) > > > > > > >> list or not. > > > > > > >> If it is - then turning on/off "wait for backfill" does not make > > > > > > >> difference. > > > > > > >> > > > > > > >> If it is not, then with "wait for backfill" turned on it will of > > > > > > >> course wait for "missing" bars > > > > > > >> before proceeding to next symbol. > > > > > > >> > > > > > > >> Charts on the other hand, always refresh immediatelly with data > > > > > > >> that are present right now. > > > > > > >> If symbol is not present in the active list it is automatically > > > > > > >> added and backfill request > > > > > > >> may be sent but this does not block chart from refreshing - it > > > > > > >> will refresh immediatelly, > > > > > > >> and once backfill data eventually arrives it will refresh once > > > > > > >> more. > > > > > > >> In other words - charts are asynchronous with regards to > > > > > > >> backfill. AA can be synchronous > > > > > > >> ("wait for backfill" turned ON) or asynchronous ("wait for > > > > > > >> backfill" turned OFF). > > > > > > >> > > > > > > >> That's the only difference. AFL execution IS IDENTICAL > > > > > > >> regardless where it occurs. > > > > > > >> Of course with different zoom levels / different AA range you > > > > > > >> may be using different number of bars, > > > > > > >> but that should be obvious. > > > > > > >> > > > > > > >> Best regards, > > > > > > >> Tomasz Janeczko > > > > > > >> amibroker.com > > > > > > >> ----- Original Message ----- > > > > > > >> From: "sidhartha70" <sidhartha70@> > > > > > > >> To: <[email protected]> > > > > > > >> Sent: Tuesday, August 04, 2009 5:14 PM > > > > > > >> Subject: [amibroker] Re: Speeding exporation intraday > > > > > > >> > > > > > > >> > > > > > > >> > I'd be interested to hear TJ's answer on this eToke... > > > > > > >> > > > > > > > >> > Because I was asking from very similar questions of support & > > > > > > >> > TJ recently... namely that equivalent AFL on charts seems to > > > > > > >> > execute > > > > > > >> > much quicker than a scan or exploration. Which obviously leads > > > > > > >> > one to think that other operations that occur in a scan or > > > > > > >> > exploration, outside of the AFL execution, are what take the > > > > > > >> > time... > > > > > > >> > > > > > > > >> > Why this time is greater with a streaming feed I don't know... > > > > > > >> > > > > > > > >> > --- In [email protected], "etoketrader" <etoke@> wrote: > > > > > > >> >> > > > > > > >> >> Hi, for testing I am currently running: > > > > > > >> >> > > > > > > >> >> n Last quotations = 1 > > > > > > >> >> A very basic AFL code which lists closing price. > > > > > > >> >> QuickAFL turned on > > > > > > >> >> > > > > > > >> >> I am concerned about the comparison between how it runs > > > > > > >> >> outside of market hours vs how it runs during. > > > > > > >> >> > > > > > > >> >> BR, > > > > > > >> >> eToke > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> --- In [email protected], "sidhartha70" > > > > > > >> >> <sidhartha70@> wrote: > > > > > > >> >> > > > > > > > >> >> > How many quotations are you running the explorations > > > > > > >> >> > over...? > > > > > > >> >> > > > > > > > >> >> > --- In [email protected], "etoketrader" <etoke@> > > > > > > >> >> > wrote: > > > > > > >> >> > > > > > > > > >> >> > > Tried that. I always run the first backfill before market > > > > > > >> >> > > hours. After that, backfill setting does not seem to make > > > > > > >> >> > > any > > > > > > >> >> > > difference as long as everything has already been > > > > > > >> >> > > backfilled and all symbols are streaming. > > > > > > >> >> > > > > > > > > >> >> > > BR > > > > > > >> >> > > eToke > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > --- In [email protected], "murthysuresh" > > > > > > >> >> > > <murthysuresh@> wrote: > > > > > > >> >> > > > > > > > > > >> >> > > > backfill before the start of the market hours. turn off > > > > > > >> >> > > > backfill as you only need to do it at the beginning of > > > > > > >> >> > > > the > > > > > > >> >> > > > session. > > > > > > >> >> > > > > > > > > > >> >> > > > --- In [email protected], "etoketrader" > > > > > > >> >> > > > <etoke@> wrote: > > > > > > >> >> > > > > > > > > > > >> >> > > > > Hi, > > > > > > >> >> > > > > > > > > > > >> >> > > > > I currently run IQFeed on 500 symbols. Exploration > > > > > > >> >> > > > > outside of market hours run under 1 minute but slow > > > > > > >> >> > > > > down to 6 > > > > > > >> >> > > > > minutes > > > > > > >> >> > > > > during market hours. > > > > > > >> >> > > > > > > > > > > >> >> > > > > Can anybody recommend some ideal settings to speed up > > > > > > >> >> > > > > intraday explorations while streaming? > > > > > > >> >> > > > > > > > > > > >> >> > > > > thanks, > > > > > > >> >> > > > > eToke > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > ------------------------------------ > > > > > > >> > > > > > > > >> > **** IMPORTANT PLEASE READ **** > > > > > > >> > This group is for the discussion between users only. > > > > > > >> > This is *NOT* technical support channel. > > > > > > >> > > > > > > > >> > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > > > >> > SUPPORT {at} amibroker.com > > > > > > >> > > > > > > > >> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > > > >> > http://www.amibroker.com/feedback/ > > > > > > >> > (submissions sent via other channels won't be considered) > > > > > > >> > > > > > > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check > > > > > > >> > DEVLOG: > > > > > > >> > http://www.amibroker.com/devlog/ > > > > > > >> > > > > > > > >> > Yahoo! Groups Links > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > > > > > **** IMPORTANT PLEASE READ **** > > > > > > > This group is for the discussion between users only. > > > > > > > This is *NOT* technical support channel. > > > > > > > > > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > > > > > SUPPORT {at} amibroker.com > > > > > > > > > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > > > > > http://www.amibroker.com/feedback/ > > > > > > > (submissions sent via other channels won't be considered) > > > > > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > > > > http://www.amibroker.com/devlog/ > > > > > > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
