Thanks guys. I was actually able to optimize SQL to get a decent performance that fits my needs. Again, my strategies are not high frequency.
I basically just created a SQL table with the AB required fields only. I also created an index specifically for AB's query. The historical data is inserted before the open and the RT data is inserted as trades occur. Using the native ODBC plug-in, I am running an exploration for 200 tickers and 1000 daily bars in under 2 seconds. The same exploration using the local database takes about 1 second, therefore the performance is entirely satisfactory. I can see how this is not the optimal solution for high frequency strategies that use intraday intervals or a watchlist with a great number of tickers. But it does work fine for more simple strategies. --- In [email protected], "Yofa" <jtoth...@...> wrote: > > Hi, > > depending on what you want to do... > You can use SQL but you must have in-depth knowlegde of the specific > database sw. > Logical and mostly physical database design is crutial to success. Both > designs must be very efficent. > Generally SQL database engine is used as a storage system. So it is bit > overkill. > > Using SQL is handy if you want to store data and from this store you want to > prepare/transform/export data to other processes. In such case it is very > flexible. > You CAN play back IB tick data at real time speed (~300ms snap shots) for a > few tickers directly from SQL db. > That should not be a problem with a good db design and on a good hw. > > Going beyond this limit is not for SQL! > > You must also consider SQL database resource needs. Disk and memory are > needed to achive good performance. > A binary file based storage system is very efficent but no really flexible. > SQL is flexible but not efficient... > You have to choose what you need.. > > Regards, > > Y > > -------------------------------------------------- > From: "Tomasz Janeczko" <gro...@...> > Sent: Monday, May 18, 2009 9:37 PM > To: <[email protected]> > Subject: Re: [amibroker] ODBC + DDE > > > SQL database & ODBC is wrong path for high-perf RT. > > > > ODBC is rather slow as it was never designed by Microsoft to be used as a > > real-time streaming tool. > > Also all standard SQL databases are way too slow for real time > > applications. > > > > Only proprietary binary low-level dbs like AmiBroker own, that are very > > low level and not SQL based > > are able to offer right speed. > > > > As for interfaces - only direct APIs (like AmiBroker Dev Kit) that operate > > directly > > on binary data via direct pointers and no need for copying data and > > without SQL overhead are fast enough. > > > > For commercial databases that offer better performance than SQL see > > Berkeley DB > > http://www.oracle.com/database/berkeley-db/db/index.html > > > > (Read their white papers that explain why all relational, sql-based > > databases are simply too slow) > > > > Best regards, > > Tomasz Janeczko > > amibroker.com > > ----- Original Message ----- > > From: "lucianomt" <lucian...@...> > > To: <[email protected]> > > Sent: Monday, May 18, 2009 6:30 PM > > Subject: [amibroker] ODBC + DDE > > > > > >>I am trying to set up real-time feeds on AB. I have a SQL database that > >>contains all the necessary data. The only problem is the > >>very high delay I get using the ODBC plug-in. To pull up about 1000 bars > >>for 150 tickers takes approx. 10 minutes, which makes it > >>impractical for real-time. (The same process using local database takes 1 > >>second.) > >> > >> So my first question is: am I doing something wrong? Is there any way to > >> improve this performance significantly or is the ODBC > >> solution inherently slow? > >> > >> If nothing can be done to improve ODBC, I thought I could upload the > >> historical data into the local database and them use a > >> different source, say DDE, to update the real-time quotes for the current > >> day. Would that solution work? > >> > >> Please note that the historical data needs to be updated daily since it > >> is a total-return series. > >> > >> > >> > >> ------------------------------------ > >> > >> **** IMPORTANT PLEASE READ **** > >> This group is for the discussion between users only. > >> This is *NOT* technical support channel. > >> > >> TO GET TECHNICAL SUPPORT send an e-mail directly to > >> SUPPORT {at} amibroker.com > >> > >> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > >> http://www.amibroker.com/feedback/ > >> (submissions sent via other channels won't be considered) > >> > >> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > >> http://www.amibroker.com/devlog/ > >> > >> Yahoo! Groups Links > >> > >> > >> > > > > > > > > ------------------------------------ > > > > **** IMPORTANT PLEASE READ **** > > This group is for the discussion between users only. > > This is *NOT* technical support channel. > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > SUPPORT {at} amibroker.com > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > http://www.amibroker.com/feedback/ > > (submissions sent via other channels won't be considered) > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > http://www.amibroker.com/devlog/ > > > > Yahoo! Groups Links > > > > > > > > >
