Philippe RIO <51...@protonmail.ch> wrote: > Hello, > I have an application composed of 256 databases. Each database occupied 42Mb > for more than 950 000 records into. > > [samedi 7 septembre 2019 13:11:45] : Number of passwords stored : 244 152 > 645 > [samedi 7 septembre 2019 13:19:28] : Closing Log file > [samedi 7 septembre 2019 13:19:28] : *** Log Stoped *** > > These 42Mb are measured after a Vacuum. > When I lauched my application it only opens the databases (all) and the > memory is filled by 57mb just for SQLITE ! > When I launch a treatment to generate random passwords the memory is filled > at 87% (>2Gb). > I would like to reduce this used memory but I don't know how. > > I have the following pragmas : > > main.page_size=65536 > wal_autocheckpoint=0 > encoding "UTF_8" > temp_store=FILE > case_sensitive_like=TRUE > foreign_keys=OFF > legacy_file_format=OFF > cache_size=-200 > main.cache_size=-200 > main.auto_vacuum=NONE > main.journal_mode=OFF > main.secure_delete=OFF > main.synchronous=OFF > main.locking_mode=EXCLUSIVE > main.user_version=230714the data > main.application_id=241259 > shrink_memory > > Some pragmas have a sense when creating the database and the other when > creating the tables. > > I have recompiled SQLITE with the following options : > > SQLITE_ENABLE_COLUMN_METADATA = 1 > SQLITE_THREAD_SAFE = 2 > SQLITE_DEFAULT_MEMSTATUS = 0 > SQLITE_DEFAULT_FILE_FORMAT = 4 > SQLITE_DEFAULT_PCACHE_INITSZ = 0 > SQLITE_WIN32_MALLOC = 1 > SQLITE_TEMP_STORE = 0 > SQLITE_CASE_SENSITIVE_LIKE = 1 > SQLITE_ENABLE_API_ARMOR 1 > SQLLITE_ENABLE_FTS3 = 1 > SQLLITE_ENABLE_FTS3_PARENTHESIS = 1 > SQLLITE_ENABLE_FTS3_TOKENIZER = 1 > SQLLITE_ENABLE_FTS4 = 1 > SQLLITE_ENABLE_FTS5 = 1 > SQLLITE_ENABLE_GEOPOLY = 1 > SQLLITE_ENABLE_DESERIALIZE = 1 > SQLLITE_ENABLE_JSON1 = 1 > SQLLITE_ENABLE_MEMORY_MANAGEMENT = 1 > SQLLITE_ENABLE_RTREE = 1 > SQLLITE_ENABLE_ENABLE_SESSION = 1 > SQLLITE_ENABLE_SOUNDEX = 1 > > The software is running under W7 Pro > I have no problem with it, I find it very fast. > See my blog at https://md5finder.blogspot.com/2019/08/md5finder.html > But this memory used/consummed is really a problem. > I have a small machine with only 3Gb of RAM and only one program needs 87% ! > How to reduce that. > > I would appreciate some help. > Thank You
How about running a memory profiler to find out when memory is used? On Linux I'd use massif, but I don't know what's available on Windows. If you open many DB connections, then: 1) each connection uses memory for the schema 2) each connection uses memory for the prepared statement. 3) each connection uses paged cache You can use sqlite3_soft_heap_limit64() to limit the total memory use in SQLite. It's a soft limit so SQLite may allocate more if it has no other choice. Open DBs in read-only mode if possible, it saves memory as constraints are then not stored in memory. Regards Dominique _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
