On Sat, Jan 27, 2018 at 8:20 AM, Martin Vahi <[email protected]> wrote:
> About 5GiB of it is the Fossil repository file and > about 17GiB of it is a tar-file with about 140k files that > the test tries to insert to the fossil repository. > i'm gonna go ahead and say it because every else is thinking it: Short answer: Nope. The longer answer: As Warren said before, Fossil is going to need some multiplier of that size in memory. Simply reading that file for insertion into the db requires 1x its size. The sqlite3 bind process is, i see now, already optimized as far as it can be to eliminate yet another in-memory copy of that blob: http://fossil-scm.org/fossil/artifact/6d07632054b709a5?ln=350-351 Your system is very likely failing on that first allocation of 17GiB. If it's not, then it's going to fail further down the line when... a) you use the 'zip' or 'tar' commands, which build their archives in memory. If it doesn't fail here then it will fail when... b) you try to commit a change to that file. In that case, fossil needs 2-3x that amount of memory (in separate allocations) in order to be able to create and apply the delta: 37.x-51GiB of RAM _just for that one file_. That's excluding any other memory costs it has. Fossil is intended for managing source code, not... whatever it is that you believe a 17GiB blob needs to be doing in a source control system (in a piece of hardware containing only a small fraction that amount of memory, no less). Barring major architectural upheaval (one step of which would be reimplementing the delta generator and applicator to stream their i/o, rather than working in-memory), your use case simply is not realistic in fossil. -- ----- stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________ fossil-users mailing list [email protected] http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
