[android-developers] Re: Unified Database Populating Solution
Hi I have an offer for you keep your dictionary as text file zip it and store it in the asset or raw resource direcory ZIP can provide really good results on text use Java ZipInputStream to expand it to the SDCard in you app , do background SQLlite buildup, simple do the population on a different thread. users will agree that upon running (not installing) your app they will be slower or wait a bit maybe consider some progress bar... --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
On Mar 14, 2:17 am, Dianne Hackborn hack...@android.com wrote: On Tue, Mar 10, 2009 at 6:36 PM, Justin Allen Jaynes jus...@ragblue.comwrote: 1. Why is there a file size limit on resources in the raw or assets folders (such a small limit, that is)? Assets are normally stored compressed, so currently opening it requires uncompressing into an in-memory buffer. From that perspective, 1MB is pretty big -- it's over 1% of the total memory available to the kernel and all apps on the G1! To get around this, you can store your file uncompressed -- the new aapt has an option to turn off compression for custom fie extensions -- so you can read it directly from the .apk as an AssetFileDescriptor. It would also be nice to support streaming access that uncompresses while you read... patches welcome. :) 2. Does this limit apply to all raw and asset files, or only to files that you get an input stream for? It is compressed files. PNGs, MP3s, etc are not compressed in the .apk since their data is already compressed. 2. Can I change the file size limit, or is it a fixed feature? Fixed. 3. Is it appropriate to split a raw file into two or three files and read them in as three consecutive separate streams, all output to one file? Or is this a bad practice since they must have implemented a file size limit for storage conservation reasons? Sure, that is fine. 4. Can I delete the resources from raw or assets at runtime (or is this totally invalid since the compiler sets up R.raw to have methods representing the files) to save space? Not at all, sorry. You .apk is completely read-only. 5. How big is too big an application? (mine is a large word dictionary and the whole app after first run takes 4 meg) Given the storage space on the G1, that is pretty big -- 5-10% of the space available to all applications, their private data (Gmail messages and such), and the browser cache. That size of an .apk will probably reduce the number of users you have, depending on how valuable people find the app to be. -- Dianne Hackborn Android framework engineer hack...@android.com Note: please don't send private questions to me, as I don't have time to provide private support. All such questions should be posted on public forums, where I and others can see and answer them. How do we specify to not compress some resource data in order to break the 1MB limitation using Eclipse ? Carlo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
If you ever find yourself in need of a Trie, I've implemented one for lexic and released it under the Apache-2 license: http://code.google.com/p/lexic/source/browse/#svn/trunk/src/net/healeys/trie It's capable of supporting multiple dialects and languages. I've used it to cram about 80,000 US UK words into less than 800K. Reading the entire trie takes about 30 seconds, but it has the ability to prune the trie based on 2-letter markov chains to drastically increase the speed. It's got a bit of lexic-specific code in it right now, but I'd be interested in working with anyone who wants to adapt it to their own project. -Johnny On Mar 11, 5:52 am, Al Sutton a...@funkyandroid.com wrote: http://en.wikipedia.org/wiki/Trie In my view the best way to do a static dictionary. Then populate the SQL database with and additions the user makes. Thing to remember; SQLites' design goals are read/write capabilities. That doesn't mean it's using the best way to handle read-only data. Al. Stoyan Damov wrote: On Wed, Mar 11, 2009 at 3:52 AM, Mark Murphy mmur...@commonsware.com wrote: Justin Allen Jaynes wrote: I couldn't deal with a 5 minute startup time on an app, so I decided to include the actual database file in my resource raw folder (others suggested this to me). One potential flaw with put-the-database-file-in-the-APK approach is SQLite file format. Suppose Android 1.5 includes an update to SQLite that causes the SQLite file format to change, such that SQLite cannot read your binary database file you packaged up? Now your app is broken for newer Android devices. It may be that SQLite will be backwards compatible, or it may be that this is just unlikely to happen. However, it is a risk developers need to keep in mind. This is not going to happen. If Google rolls out SQLite X.YY and it's incompatible with the database file format used by currently installed apps (some probably paid), would you think that the OTA upgrade will kind of search for all SQLite databases on the device (some might be on the SD card) and offer the user to upgrade them? The same general argument holds true for packaging the data in the form of SQL statements, though given that SQL is more or less a standard, I would expect fewer problems this way...performance notwithstanding. Obviously, given the performance characteristics in your case, if SQLite has to be the answer (see below), then you have no choice but to package the binary database and pray. If I was building a dictionary app and it had to use database and not some other binary format I'd compress a database with non-indexed tables, put it as a resource, then on 1st run inflate it to SD card, and create an index (if that's supported, I'm assuming running an SQLite db off the SD card is). This would decrease my storage requirements to the bare minimum. (mine is a large word dictionary and the whole app after first run takes 4 meg) Another thing to consider with large yet simple database structures is whether SQLite is the right answer at all. This is particularly true if the data is read-only or read-mostly. It may be you are better served finding a Java library implementing a lower-level data structure (red-black tree, AVL tree, etc.) that is better tuned for your scenario. You might even find one that can work efficiently with an InputStream, thereby allowing you to keep the data in the APK without a second copy. For example, when I see large word dictionary, I think of spell-checkers, and implementing a spell-checker with a SQLite backend would seem likely to have poor performance. Of course, then again, it's been about two decades since I wrote my last spell-checker... -- Mark Murphy (a Commons Guy) http://commonsware.com Android Training in Sweden --http://www.sotrium.com/training.php -- * Written an Android App? - List it athttp://andappstore.com/* == Funky Android Limited is registered in England Wales with the company number 6741909. The registered head office is Kemp House, 152-160 City Road, London, EC1V 2NX, UK. The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
Do reply on the list, I'm sure other people are interested as well (I am for example). On Fri, Mar 13, 2009 at 1:15 AM, Mark Murphy mmur...@commonsware.com wrote: Justin Allen Jaynes wrote: How in-depth do you go in your books, especially in demonstrating best-practices of common tasks such as managing large amounts of data on such a small platform? Also, how are the two books related? Does the second book supersede the first, or are they companions? I'd like to buy both but I'm on a tight budget. Thanks for the interest! I'll reply to you off-list shortly, so this doesn't turn into a commercial... -- Mark Murphy (a Commons Guy) http://commonsware.com _The Busy Coder's Guide to Android Development_ Version 2.0 Available! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
On Tue, Mar 10, 2009 at 6:36 PM, Justin Allen Jaynes jus...@ragblue.comwrote: 1. Why is there a file size limit on resources in the raw or assets folders (such a small limit, that is)? Assets are normally stored compressed, so currently opening it requires uncompressing into an in-memory buffer. From that perspective, 1MB is pretty big -- it's over 1% of the total memory available to the kernel and all apps on the G1! To get around this, you can store your file uncompressed -- the new aapt has an option to turn off compression for custom fie extensions -- so you can read it directly from the .apk as an AssetFileDescriptor. It would also be nice to support streaming access that uncompresses while you read... patches welcome. :) 2. Does this limit apply to all raw and asset files, or only to files that you get an input stream for? It is compressed files. PNGs, MP3s, etc are not compressed in the .apk since their data is already compressed. 2. Can I change the file size limit, or is it a fixed feature? Fixed. 3. Is it appropriate to split a raw file into two or three files and read them in as three consecutive separate streams, all output to one file? Or is this a bad practice since they must have implemented a file size limit for storage conservation reasons? Sure, that is fine. 4. Can I delete the resources from raw or assets at runtime (or is this totally invalid since the compiler sets up R.raw to have methods representing the files) to save space? Not at all, sorry. You .apk is completely read-only. 5. How big is too big an application? (mine is a large word dictionary and the whole app after first run takes 4 meg) Given the storage space on the G1, that is pretty big -- 5-10% of the space available to all applications, their private data (Gmail messages and such), and the browser cache. That size of an .apk will probably reduce the number of users you have, depending on how valuable people find the app to be. -- Dianne Hackborn Android framework engineer hack...@android.com Note: please don't send private questions to me, as I don't have time to provide private support. All such questions should be posted on public forums, where I and others can see and answer them. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
Stoyan Damov wrote: Do reply on the list, I'm sure other people are interested as well (I am for example). Well, I'll meet you halfway: I updated the CommonsWare Web site (wow, was *that* ever dusty!) and put the answers to at least some of the questions right on the home page: http://commonsware.com There, you can learn more about the three (yes, three!) Android books CommonsWare has available. Other questions about the books are best sent straight to me, or to the [cw-android] Google Group. Thanks for your interest! -- Mark Murphy (a Commons Guy) http://commonsware.com _The Busy Coder's Guide to Android Development_ Version 2.0 Published! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
Mark Murphy wrote: Another thing to consider with large yet simple database structures is whether SQLite is the right answer at all. This is particularly true if the data is read-only or read-mostly. It may be you are better served finding a Java library implementing a lower-level data structure (red-black tree, AVL tree, etc.) that is better tuned for your scenario. You might even find one that can work efficiently with an InputStream, thereby allowing you to keep the data in the APK without a second copy. For example, when I see large word dictionary, I think of spell-checkers, and implementing a spell-checker with a SQLite backend would seem likely to have poor performance. Of course, then again, it's been about two decades since I wrote my last spell-checker... Mark, I observed that you are the author of two books on android (one yet to be released). I currently own a book on android by another author and I'm somewhat disappointed. It's rather superficial and lacks depth of discussion on issues like this, and actually skips a lot of simple features as well. How in-depth do you go in your books, especially in demonstrating best-practices of common tasks such as managing large amounts of data on such a small platform? Also, how are the two books related? Does the second book supersede the first, or are they companions? I'd like to buy both but I'm on a tight budget. Justin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
Justin Allen Jaynes wrote: How in-depth do you go in your books, especially in demonstrating best-practices of common tasks such as managing large amounts of data on such a small platform? Also, how are the two books related? Does the second book supersede the first, or are they companions? I'd like to buy both but I'm on a tight budget. Thanks for the interest! I'll reply to you off-list shortly, so this doesn't turn into a commercial... -- Mark Murphy (a Commons Guy) http://commonsware.com _The Busy Coder's Guide to Android Development_ Version 2.0 Available! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
On Wed, Mar 11, 2009 at 3:52 AM, Mark Murphy mmur...@commonsware.com wrote: Justin Allen Jaynes wrote: I couldn't deal with a 5 minute startup time on an app, so I decided to include the actual database file in my resource raw folder (others suggested this to me). One potential flaw with put-the-database-file-in-the-APK approach is SQLite file format. Suppose Android 1.5 includes an update to SQLite that causes the SQLite file format to change, such that SQLite cannot read your binary database file you packaged up? Now your app is broken for newer Android devices. It may be that SQLite will be backwards compatible, or it may be that this is just unlikely to happen. However, it is a risk developers need to keep in mind. This is not going to happen. If Google rolls out SQLite X.YY and it's incompatible with the database file format used by currently installed apps (some probably paid), would you think that the OTA upgrade will kind of search for all SQLite databases on the device (some might be on the SD card) and offer the user to upgrade them? The same general argument holds true for packaging the data in the form of SQL statements, though given that SQL is more or less a standard, I would expect fewer problems this way...performance notwithstanding. Obviously, given the performance characteristics in your case, if SQLite has to be the answer (see below), then you have no choice but to package the binary database and pray. If I was building a dictionary app and it had to use database and not some other binary format I'd compress a database with non-indexed tables, put it as a resource, then on 1st run inflate it to SD card, and create an index (if that's supported, I'm assuming running an SQLite db off the SD card is). This would decrease my storage requirements to the bare minimum. (mine is a large word dictionary and the whole app after first run takes 4 meg) Another thing to consider with large yet simple database structures is whether SQLite is the right answer at all. This is particularly true if the data is read-only or read-mostly. It may be you are better served finding a Java library implementing a lower-level data structure (red-black tree, AVL tree, etc.) that is better tuned for your scenario. You might even find one that can work efficiently with an InputStream, thereby allowing you to keep the data in the APK without a second copy. For example, when I see large word dictionary, I think of spell-checkers, and implementing a spell-checker with a SQLite backend would seem likely to have poor performance. Of course, then again, it's been about two decades since I wrote my last spell-checker... -- Mark Murphy (a Commons Guy) http://commonsware.com Android Training in Sweden -- http://www.sotrium.com/training.php --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
http://en.wikipedia.org/wiki/Trie In my view the best way to do a static dictionary. Then populate the SQL database with and additions the user makes. Thing to remember; SQLites' design goals are read/write capabilities. That doesn't mean it's using the best way to handle read-only data. Al. Stoyan Damov wrote: On Wed, Mar 11, 2009 at 3:52 AM, Mark Murphy mmur...@commonsware.com wrote: Justin Allen Jaynes wrote: I couldn't deal with a 5 minute startup time on an app, so I decided to include the actual database file in my resource raw folder (others suggested this to me). One potential flaw with put-the-database-file-in-the-APK approach is SQLite file format. Suppose Android 1.5 includes an update to SQLite that causes the SQLite file format to change, such that SQLite cannot read your binary database file you packaged up? Now your app is broken for newer Android devices. It may be that SQLite will be backwards compatible, or it may be that this is just unlikely to happen. However, it is a risk developers need to keep in mind. This is not going to happen. If Google rolls out SQLite X.YY and it's incompatible with the database file format used by currently installed apps (some probably paid), would you think that the OTA upgrade will kind of search for all SQLite databases on the device (some might be on the SD card) and offer the user to upgrade them? The same general argument holds true for packaging the data in the form of SQL statements, though given that SQL is more or less a standard, I would expect fewer problems this way...performance notwithstanding. Obviously, given the performance characteristics in your case, if SQLite has to be the answer (see below), then you have no choice but to package the binary database and pray. If I was building a dictionary app and it had to use database and not some other binary format I'd compress a database with non-indexed tables, put it as a resource, then on 1st run inflate it to SD card, and create an index (if that's supported, I'm assuming running an SQLite db off the SD card is). This would decrease my storage requirements to the bare minimum. (mine is a large word dictionary and the whole app after first run takes 4 meg) Another thing to consider with large yet simple database structures is whether SQLite is the right answer at all. This is particularly true if the data is read-only or read-mostly. It may be you are better served finding a Java library implementing a lower-level data structure (red-black tree, AVL tree, etc.) that is better tuned for your scenario. You might even find one that can work efficiently with an InputStream, thereby allowing you to keep the data in the APK without a second copy. For example, when I see large word dictionary, I think of spell-checkers, and implementing a spell-checker with a SQLite backend would seem likely to have poor performance. Of course, then again, it's been about two decades since I wrote my last spell-checker... -- Mark Murphy (a Commons Guy) http://commonsware.com Android Training in Sweden -- http://www.sotrium.com/training.php -- * Written an Android App? - List it at http://andappstore.com/ * == Funky Android Limited is registered in England Wales with the company number 6741909. The registered head office is Kemp House, 152-160 City Road, London, EC1V 2NX, UK. The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
Stoyan Damov wrote: This is not going to happen. If Google rolls out SQLite X.YY and it's incompatible with the database file format used by currently installed apps (some probably paid), would you think that the OTA upgrade will kind of search for all SQLite databases on the device (some might be on the SD card) and offer the user to upgrade them? It's possible. It might also do something at database-open time (conversion on the first open after the OTA upgrade). I just want developers to be aware that there *might* be issues here. I'm not saying you should avoid Mr. Jaynes' technique, just that you should use it with full knowledge that the SQLite file format might change and it might cause problems. -- Mark Murphy (a Commons Guy) http://commonsware.com Android Training in Sweden -- http://www.sotrium.com/training.php --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Unified Database Populating Solution
Justin Allen Jaynes wrote: I couldn't deal with a 5 minute startup time on an app, so I decided to include the actual database file in my resource raw folder (others suggested this to me). One potential flaw with put-the-database-file-in-the-APK approach is SQLite file format. Suppose Android 1.5 includes an update to SQLite that causes the SQLite file format to change, such that SQLite cannot read your binary database file you packaged up? Now your app is broken for newer Android devices. It may be that SQLite will be backwards compatible, or it may be that this is just unlikely to happen. However, it is a risk developers need to keep in mind. The same general argument holds true for packaging the data in the form of SQL statements, though given that SQL is more or less a standard, I would expect fewer problems this way...performance notwithstanding. Obviously, given the performance characteristics in your case, if SQLite has to be the answer (see below), then you have no choice but to package the binary database and pray. (mine is a large word dictionary and the whole app after first run takes 4 meg) Another thing to consider with large yet simple database structures is whether SQLite is the right answer at all. This is particularly true if the data is read-only or read-mostly. It may be you are better served finding a Java library implementing a lower-level data structure (red-black tree, AVL tree, etc.) that is better tuned for your scenario. You might even find one that can work efficiently with an InputStream, thereby allowing you to keep the data in the APK without a second copy. For example, when I see large word dictionary, I think of spell-checkers, and implementing a spell-checker with a SQLite backend would seem likely to have poor performance. Of course, then again, it's been about two decades since I wrote my last spell-checker... -- Mark Murphy (a Commons Guy) http://commonsware.com Android Training in Sweden -- http://www.sotrium.com/training.php --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Android Developers group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
