> So I am guessing that this is probably expected behavior for .NET. You may be > able to tune some runtime settings, to tell it to limit heap growth. Or it > may be possible via the .NET SQLite API to read the blobs into existing byte > array objects, which would mostly eliminate the garbage production. I am very familiar with how the .Net Framework Garbage Collection works, and this isn't really that type of problem. The memory usage of the application/process shows completely normal and the massively Large Active Mapped File remains as long as the connection remains, even if the program doing nothing for hours, days, or weeks. The improper state of the Large Active Mapped File isn't caused by the .Net Framework Garbage Collector.
My guess is there's something in the System.Data.SQLite library for reading byte data (or BLOBs) that is unintentionally opening the database file in the way where it'll remain as an Active Mapped File until the connection closes. Maybe it's a typo; maybe the wrong file flags are used, maybe there is some improperly used unsafe code? I don't really know. The other alternative there's still something wrong in my test application. I appreciated your insights on it though, and I'm still open to any suggestions or comments, or anything I can change in my test application that may resolve the issue (http://i.imgur.com/mod5ISX.png). Thank you everyone! On Saturday, May 27, 2017 3:22 PM, Jens Alfke <j...@mooseyard.com> wrote: > On May 26, 2017, at 5:46 PM, Jamie <eqrecov...@yahoo.com> wrote: > > Without going into too much detail of explaining C#, everything is discarded > appropriately. All disposable objects are wrapped by using statements. And > byte arrays vanish once they're out of scope. They vanish from _your_ perspective, but they do remain in the heap until the garbage collector finds and removes them. So from the garbage collector’s POV, your program is generating garbage byte arrays as fast as they can be read from SQLite. All garbage collectors have a dilemma when garbage generation outpaces collection, and the current heap fills up. They can either block the allocator until they’ve cleared enough space to satisfy the request, or they can grow the heap by requesting more space from the OS. Usually they have heuristics for doing one or the other in different circumstances. If they go too far one way, the process thrashes in the garbage collector, but too far the other way and its memory usage blows up (and eventually the entire OS goes into page-swap thrash.) I don’t use .NET so I don’t know its garbage collector's behavior. But my experience with other systems (most recently Go) is that they tend to lean more toward allocating more memory, so it’s more common to see a runaway garbage generator blow up its address space. So I am guessing that this is probably expected behavior for .NET. You may be able to tune some runtime settings, to tell it to limit heap growth. Or it may be possible via the .NET SQLite API to read the blobs into existing byte array objects, which would mostly eliminate the garbage production. —Jens _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
