New topic: Memory Problems
<http://forums.realsoftware.com/viewtopic.php?t=47634> Page 1 of 1 [ 9 posts ] Previous topic | Next topic Author Message DanSandbergUCONN Post subject: Memory ProblemsPosted: Thu Apr 18, 2013 2:32 pm Joined: Fri Dec 21, 2007 9:50 am Posts: 237 Hi All - I have a program that reads in data from a database and stores it in an array. The program runs fine on MOST of my database files but crashes unexpectedly for one really large database. I've narrowed down the crash to this line of code:redim myarr(ntimesteps,UBound(mytickers),7+nfullyears)which makes me think that I am running into memory problems. The array dimensions I'm asking for are 3553 x 4906 x 22 and I have 16GB of memory. Can anyone confirm the required memory for that array size (variable type is string)? Thanks so much. Top Jason_Adams Post subject: Re: Memory ProblemsPosted: Thu Apr 18, 2013 2:41 pm Joined: Fri Nov 10, 2006 4:10 pm Posts: 1782 Location: Michigan, USA Am I understanding you correctly that what you're storing is 22 bytes each? In any case, that's a lot of memory to suddenly demand congruently. It's entirely possible that would crash and, at the very least, take a long time. I have to ask, are you sure this is the best way to do this? I do a lot of algorithms from a database and find there's always a way to parse the data in chunks. _________________ Windows 7 Ultimate x64 Windows XP Pro SP3 Ubuntu 11.04 via Virtual Box RS Enterprise 2012r1.1 Programming Tutorials & Free Projects: http://www.JasonTheAdams.com "Christianity has not been tried and found wanting; it has been found difficult and not tried." - G.K. Chesterton Top DanSandbergUCONN Post subject: Re: Memory ProblemsPosted: Thu Apr 18, 2013 5:21 pm Joined: Fri Dec 21, 2007 9:50 am Posts: 237 Jason_Adams wrote:Am I understanding you correctly that what you're storing is 22 bytes each? In any case, that's a lot of memory to suddenly demand congruently. It's entirely possible that would crash and, at the very least, take a long time. I have to ask, are you sure this is the best way to do this? I do a lot of algorithms from a database and find there's always a way to parse the data in chunks. It may very well not be the best way to do this. What I am doing is storing 3553 days of stock prices (about 15 years) for 4906 ticker symbols and for each symbol I am storing 22 pieces of information (day's high price, low price, market cap, etc.). I have this data currently organized in an excel spreadsheet but I want to read it into my program to do a trend analysis. I may need to find a work-around. Top timhare Post subject: Re: Memory ProblemsPosted: Thu Apr 18, 2013 6:02 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 12237 Location: Portland, OR USA The maximum amount of memory your app can use is something less than 4GB. Empty, that array uses more than 1.5GB. So, yeah, it's probably a memory issue. I'd recommend using a database instead of an array. Top charonn0 Post subject: Re: Memory ProblemsPosted: Thu Apr 18, 2013 7:22 pm Joined: Mon Apr 02, 2007 2:08 am Posts: 1191 Location: San Francisco, CA, USA timhare wrote:The maximum amount of memory your app can use is something less than 4GB. Empty, that array uses more than 1.5GB. So, yeah, it's probably a memory issue. I'd recommend using a database instead of an array. I've come up against a soft limit of about 1.1GB in my RB apps, after which they tend to lock up/crash. Though it may be due to something I'm doing rather than to the framework. _________________ Boredom Software Top Jason_Adams Post subject: Re: Memory ProblemsPosted: Fri Apr 19, 2013 7:06 am Joined: Fri Nov 10, 2006 4:10 pm Posts: 1782 Location: Michigan, USA DanSandbergUCONN wrote:It may very well not be the best way to do this. What I am doing is storing 3553 days of stock prices (about 15 years) for 4906 ticker symbols and for each symbol I am storing 22 pieces of information (day's high price, low price, market cap, etc.). I have this data currently organized in an excel spreadsheet but I want to read it into my program to do a trend analysis. I may need to find a work-around. Without understanding more about your project, it's hard to give specifically helpful advice, but you can certainly find the highest price, lowest price, etc., by doing a query for it. Databases do much more than just "hold" data; they parse through it MUCH faster and can perform multiple functions. They also know how to sift through terabytes of data without killing themselves. Hope this helps! _________________ Windows 7 Ultimate x64 Windows XP Pro SP3 Ubuntu 11.04 via Virtual Box RS Enterprise 2012r1.1 Programming Tutorials & Free Projects: http://www.JasonTheAdams.com "Christianity has not been tried and found wanting; it has been found difficult and not tried." - G.K. Chesterton Top DanSandbergUCONN Post subject: Re: Memory ProblemsPosted: Fri Apr 19, 2013 10:00 am Joined: Fri Dec 21, 2007 9:50 am Posts: 237 A database definitely seems to be the right way to go. So, I'm sure this information is available in other places but would someone mind giving me a quick guide on how to convert my comma-delimited data file into a database? Presently, what I do is this: the excel file contains one stock ticker symbol per column and each row contains the closing day price. Each row makes up the first dimension of the array, each column makes up the second dimension, and then the third dimension contains information about the ticker based on the historical price data up to that row. So, I may compute several time-dependent quantities for each ticker and then store them in array(i,j,k) where k represents the index of the time-dependent value at time step i for ticker symbol j. Any suggestions on doing this with a database is much appreciated. Thanks, Dan Top Jason_Adams Post subject: Re: Memory ProblemsPosted: Fri Apr 19, 2013 10:16 am Joined: Fri Nov 10, 2006 4:10 pm Posts: 1782 Location: Michigan, USA I'm sorry, I'm having a difficult time visualizing what you're saying. I'm not sure what a "sock ticker symbol" is. Could you give an example of what a few rows might look like in the comma-delimited form? How do you store the day? As a date? What's "information" exactly, concerning the historical price? To help we really need to know what exactly is what. _________________ Windows 7 Ultimate x64 Windows XP Pro SP3 Ubuntu 11.04 via Virtual Box RS Enterprise 2012r1.1 Programming Tutorials & Free Projects: http://www.JasonTheAdams.com "Christianity has not been tried and found wanting; it has been found difficult and not tried." - G.K. Chesterton Top DanSandbergUCONN Post subject: Re: Memory ProblemsPosted: Fri Apr 19, 2013 11:45 am Joined: Fri Dec 21, 2007 9:50 am Posts: 237 Jason_Adams wrote:I'm sorry, I'm having a difficult time visualizing what you're saying. I'm not sure what a "sock ticker symbol" is. Could you give an example of what a few rows might look like in the comma-delimited form? How do you store the day? As a date? What's "information" exactly, concerning the historical price? To help we really need to know what exactly is what. Sorry that was supported to be STOCK not sock. So for example the first row of a file would be Date,ABC,DEF,XYZ,..... (so first line is a comma-delimited string of ticker symbols that define stocks) next line 04/19/2013, 20.35, 10.17, 123.50, ... (so there are ~6000 lines after the first line, each starting with a date (yes I store it as a date) and then the price for each ticker's price) Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 9 posts ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
