New topic: Handling lots of files
<http://forums.realsoftware.com/viewtopic.php?t=40254> Page 1 of 1 [ 13 posts ] Previous topic | Next topic Author Message Farflame Post subject: Handling lots of filesPosted: Sat Aug 13, 2011 8:45 am Joined: Tue Oct 05, 2010 6:25 am Posts: 85 I'm writing a program which has lots of small files (under 1k) and I'm noticing what a bad choice this is, because having the program open and close lots of files over and over causes it to really slow down. *Edit* I think binary files will probably do the trick, but then I have another question. If I made a single large binary file which I read and write to regularly, is it a bad idea to keep the file open for long periods of time (e.g the whole time the program is running)? A solution would be to have my file open all the time and read and write to the exact location I need whenever necessary, without the constant opening and closing, but is it a bad idea to keep a file open like this for long periods? Last edited by Farflame on Sat Aug 13, 2011 8:50 am, edited 1 time in total. Top HardyMachia Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 8:50 am Joined: Mon Dec 11, 2006 12:25 pm Posts: 28 It depends on what the files are being used for. Do you need to create the files to send to other people or are you just using the files to save data for your app. If they are just used to save your app data, then you want to be using database. _________________ http://www.catamount.com PocketMoney for iOS, OSX, Linux, Windows, and Android Top Farflame Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 8:51 am Joined: Tue Oct 05, 2010 6:25 am Posts: 85 It's just for saving data for my app. Top timhare Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 12:49 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 10058 Location: Portland, OR USA Do you really need to access the data off the disk while your program is running? Why not read it all in at program startup and write it out when you quit. During execution, access it all from memory. You can use individual variables or a dictionary to store the info during program execution. Top Farflame Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 2:11 pm Joined: Tue Oct 05, 2010 6:25 am Posts: 85 Well there's a lot, probably too much to keep in memory at once. I'm re-writing my Soccer management game and it can have thousands of players (the players on the football pitch, not customers (I wish ). Even in a small test I had 13,000 players, all with about 1k of information. So that'd be 13mb of memory if they all just sat in memory... which probably isn't too much but it could get bigger. On the other hand, I don't like constantly reading and writing from disk, but most of the players aren't needed most of the time. When there's a match starting, it just needs to load in the 40 or so players required, process the match, then save them back. I'm wondering if it's possible to just keep the player file open all of the time, and load and save as I go along, using 'flush' to write stuff to disk. But then, I don't like leaving the file open like that, seems like a bad idea to me. Top Bob Keeney Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 2:24 pm Joined: Fri Sep 30, 2005 11:48 am Posts: 2602 Location: Lenexa, KS After reading about your app I think a database would be a better solution. This is what databases are designed to handle quickly and efficiently. One database file can have millions of records. Queries are lightning fast. And once you add data you can easily have it flush out to disk with just a single line of code. You can load only the data you want, etc., etc. _________________ Bob K. 30+ hours of Real Studio Video Training (including over 6 hours of Web Edition) at http://www.bkeeney.com/ Real Studio Consulting http://www.bkeeney.com/consulting/real-studio-consulting BKeeney Briefs Blog For Real Studio Developers http://www.bkeeneybriefs.com/ Top Farflame Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 2:41 pm Joined: Tue Oct 05, 2010 6:25 am Posts: 85 Thanks, I was gonna ask if anyone knows of a good tutorial for RB and databases, but I believe you have some in your videos, so I'll go and take a look at those. Thanks again. Top charonn0 Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 3:17 pm Joined: Mon Apr 02, 2007 2:08 am Posts: 501 Location: San Francisco, CA, USA http://sonicgenie.com/rbdev/db/ is a great series of video tutorials for RB databases. _________________ Boredom Software Top timhare Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 4:27 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 10058 Location: Portland, OR USA Yeah, you have definitely just described a database application. No way should you do it with individual files. Top Farflame Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 7:34 pm Joined: Tue Oct 05, 2010 6:25 am Posts: 85 Ok cool, I've just ran through that tutorial and it all went good, and I can see a database is the way forward. Something odd just happened though, I went back to my main program to start changing over my old file methods to a database, but the program is now stopping at a totally unrelated point. This piece of code.... Code: dim g as folderitem g=GetFolderItem("").Child("files").Child("teamfirstnames.txt") If g.Exists then .. which is in a totally seperate part of the program, is now stopping, with g having no value. Now that file does exist, and if I remove the new database code, which is totally seperate to this code in a different method, it runs fine again. I still need this piece of code to work as it gets some values when my program first opens. It's a bit odd because this code and the database code (just a bit I've ripped from the docs at the moment) are seperate and don't share any variables or anything. The code from the docs is just.... Code: myDB=New REALSQLDatabase myDB.databaseFile = GetFolderItem("").Child("gamedata").Child("players.rsd") I've removed the rest of the code because it's this line that seems to be causing the problem. Is there some problem if I search in a folder for a database item, and then search for a text item deeper in that folder, later? Top charonn0 Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 7:49 pm Joined: Mon Apr 02, 2007 2:08 am Posts: 501 Location: San Francisco, CA, USA Using a database will mean that your app now has a "Lib" folder created next to it. If your app didn't previously have a Libs folder, then GetFolderItem("") no longer refers to the folder you might think it does. When an app with Libs is run in the debugger, a new folder is created to contain both the main exe and the Libs folder: Without Libs: Main Directory <--------- GetFolderItem("") ---DebugMyApp.exe With Libs: Main Directory ---DebugMyApp <--------- GetFolderItem("") ------DebugMyApp Libs ---------REALSQL.dll ------DebugMyApp.exe Try this: Code: dim g as folderitem #If DebugBuild Then g=App.ExecutableFile.Parent.Parent.Child("files").Child("teamfirstnames.txt") #Else g=App.ExecutableFile.Parent.Child("files").Child("teamfirstnames.txt") #endif If g.Exists then _________________ Boredom Software Top Farflame Post subject: Re: Handling lots of filesPosted: Sat Aug 13, 2011 9:26 pm Joined: Tue Oct 05, 2010 6:25 am Posts: 85 Thank you. I've never really used databases before but I can see the beauty of them already. Just got to get this re-written and learn the basics of databases, but it doesn't look too hard. A few days and I reckon my app will be much improved Top Specialised Post subject: Re: Handling lots of filesPosted: Sun Aug 14, 2011 12:59 am Joined: Thu Aug 26, 2010 6:57 am Posts: 347 Location: Sydney Australia I started a project once that used a database with very little data and some colleagues of my client said it was overkill. My client ended up with an aggressive business partner that made his business explode. While I had to rejig his app the data structure was in place and could easily be scaled to suit his massive growth. So the moral of the story is stick to files for config settings and use databases as much as possible. _________________ Regards Chris Musty Specialised Technologies Pty Ltd Software | Electronics | IT | Multimedia http://www.specialised.net.au ----------------------------------------------- RB 2011r2 Enterprise on Win7 Ultimate, Ubuntu 10.10 on Virtualbox Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 13 posts ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
