TJ, what's the back of the envelope calculation we should do to work out if our DB data set fits into on board CPU cache...?? For example, if I have an 8MB L2 Cache... how many bars can I fit into that comfortably...?
Many Thanks --- In [email protected], Tomasz Janeczko <gro...@...> wrote: > > Hello, > > There is no such thing as "no limit". Your HARDWARE IS LIMITED, 3rd > party data vendor HAS LIMITS on amounts of data sent. > The limits are there for a reason. To allow smooth REAL TIME OPERATION. > If you override limits you are shooting yourself in a foot and will face > problems with real-time operation. > *DO NOT override limits* > Real-time database must be able to keep up with data stream counting > even thousands of ticks per second PER SYMBOL. > Everyone who thinks that his/her hardware has no limits is simply very, > very, very wrong. > The person who thinks so is deeply wrong because he/she does not know > how the amount of memory affects the performance. > You guys should run memtest86 program to see what is the difference > between processing speed of ON-CHIP CPU CACHE > and normal memory. The difference is TEN FOLD (or larger). There are in > fact differences as large as 2x or 3x in speed between level-1 > and level-2 caches. If you allow that your data are larger than your CPU > cache is, you will automatically > get 10x performance hit because of limited memory thoughput. CPU will > WAIT for memory. Memory speeds are way behind processor speeds. > > If you need want "endless" database use LOCAL DATABASE only without > external source. Such database working OFF-LINE > is not subject to real-time constraints. > > Best regards, > Tomasz Janeczko > amibroker.com > > On 2010-03-11 11:19, Magesh wrote: > > > > > > Some how my mails aren't getting through into the group. Hoping to get > > luckier this time. > > > > Thanks to supp...@..., for I got clarity on this mail that # > > of Bar limitations is meant only for plugin-driven databases. > > Though that solved my worries, as I use a local database for caching > > RT data, I still am curious to understand how such scenarios are > > handled in real-time while using Plugin driven databases > > > > In other words, how to preserve my data from getting truncated once > > the size limit exceeds in a Plugin-Driven database. > > Any thoughts or suggestions are much appreciated. > > > > thanks and regards > > Magesh > > > > On 10/03/10 11:01 AM, Magesh wrote: > >> > >> Does this mean a limitation of AB for saving data to infinite without > >> truncating? > >> > >> I am using a real-time stream, Base Time Interval is "Tick" and No. > >> of Bars=500,000. > >> Total no of symbols = 10 (all with real-time Tick). > >> I have a fixed layout, and no frequent change in symbols. > >> > >> And with my rough math of a tick for every 2 seconds, > >> # of bars per trading day = 11700 bars > >> Max # of days of history (@ 500,000 bars)= 43 days > >> Max # of days of history (@ 1,500,000 bars) = 128 days > >> > >> My questions are: > >> 1) Is my understandings on the bar count, size per symbol and limit > >> is correct? > >> > >> 2) Is there any ways of overcoming this limit, my reason being, I > >> would like to hold the historical tick size data for more than a year > >> (at-least to start with)? > >> > >> 3) I think its scalable by altering "Base Time Interval" from "Tick" > >> to "5/15 seconds" and will allow me to store up to 18 & 60 months > >> approx. of data respectively. Is this right? > >> > >> 4) What's the max integer limit (size of variable) for MaximumNoOfBars? > >> > >> 5) Can alternate dBs (MySQL)be used in synch with AB's database, so > >> that periodic push/flushing of AB's data into an external dBase? (as > >> I won't be looking or using historical intraday's in realtime or > >> atleast not so frequently) > >> > >> Btw, this is a total scaled down version of my requirement, as my > >> database is just about a month old and currently I have about 30 > >> symbols and just begun learning AFL and back-testing/explorations. > >> And with the current bar count setting, I think my database will > >> start getting truncated anytime sooner (for few of the symbols > >> updated via RT). > >> > >> So I would like to keep my options very open for any kind/type of > >> suggestions as my current attempt is to have AB retain data and stop > >> draining LIFO. > >> > >> Thanks in Advance, > >> Magesh > >> > >> > >> P.S.:I couldn't find any updates beyond this chain on this email > >> thread subject from gmane, so any pointers or suggestions in this > >> direction will be of great help for me. > >> > >> > >> > >> On 02/08/08 3:59 PM, Herman wrote: > >>> > >>> Hi Tomasz (or anyone who knows), > >>> > >>> > >>> Could you define "advanced users and powerful computers"? > >>> > >>> > >>> Who would use this feature and what what exactly does it do? > >>> > >>> > >>> Thanks, > >>> > >>> herman > >>> > >>> > >>> > >>> Friday, August 1, 2008, 7:21:11 AM, you wrote: > >>> > >>> > >>> > No, it shouldn't. This key is a "hidden extra feature" for advanced > >>> > >>> > users having computers powerful enough to handle extra load. > >>> > >>> > By default the key does not exist and then the default max is 500000. > >>> > >>> > >>> > Best regards, > >>> > >>> > Tomasz Janeczko > >>> > >>> > amibroker.com > >>> > >>> > ----- Original Message ----- > >>> > >>> > From: "progster01" > >>> <progster-sfbTBCYm2/nd5v35psfx2nbpr1lh4...@... > >>> <mailto:progster-sfbTBCYm2/nd5v35psfx2nbpr1lh4...@...>> > >>> > >>> > To: <amibroker-hhksg33tihhbjbujkae...@... > >>> <mailto:amibroker-hhksg33tihhbjbujkae...@...>> > >>> > >>> > Sent: Friday, August 01, 2008 12:55 PM > >>> > >>> > Subject: [amibroker] Re: Database settings and the amount of data saved > >>> > >>> > >>> > >>> > >>> >>> Make sure you check the path exactly! The key name is > >>> > >>> >>> HKEY_CURRENT_USER\Software\TJP\Broker\Settings\MaximumNumber > >>> > >>> >>> OfBars > >>> > >>> >>> (watch out for line truncations) > >>> > >>> > >>> >>> The key is a DWORD. > >>> > >>> > >>> >> This key does not exist on the machine I'm looking at (which has AB > >>> > >>> >> 5.15 installed). > >>> > >>> > >>> >> Obviously I could add it myself, but "should" it already be there? > >>> > >>> > >>> >> IOW, what is the design scenario for the creation of this key? > >>> > >>> > >>> > >>> > >>> > >>> > >>> >> ------------------------------------ > >>> > >>> > >>> >> Please note that this group is for discussion between users only. > >>> > >>> > >>> >> To get support from AmiBroker please send an e-mail directly to > >>> > >>> >> SUPPORT {at} amibroker.com > >>> > >>> > >>> >> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > >>> > >>> >> http://www.amibroker.com/devlog/ > >>> > >>> > >>> >> For other support material please check also: > >>> > >>> >> http://www.amibroker.com/support.html > >>> > >>> >> Yahoo! Groups Links > >>> > >>> > >>> > >>> > >>> > >>> > ------------------------------------ > >>> > >>> > >>> > Please note that this group is for discussion between users only. > >>> > >>> > >>> > To get support from AmiBroker please send an e-mail directly to > >>> > >>> > SUPPORT {at} amibroker.com > >>> > >>> > >>> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > >>> > >>> > http://www.amibroker.com/devlog/ > >>> > >>> > >>> > For other support material please check also: > >>> > >>> > http://www.amibroker.com/support.html > >>> > >>> > Yahoo! Groups Links > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > (Yahoo! ID required) > >>> > >>> > >>> > >>> > mailto:amibroker-fullfeatured-hhksg33tihhbjbujkae...@... > >>> > >>> > >>> <mailto:amibroker-unsubscribe-hhksg33tihhbjbujkae...@...> > >>> > >>> > >>> > >>> > >>> > > > > > > >
