On 16 February 2011 01:12, Rob Pettefar <[email protected]> wrote: > Hi guys > I am using Windows 7 Ultimate and I managed to get round the 4GB database > limit using this installer: > https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe > With this I have a 5.8GB database that is happily running, even if I restart > the service. > > However when this was applied to a Windows Server 2003 Standard Edition > machine the 4GB limit still seems to exist.
Hi Rob If this is still causing errors then it's pretty serious - over-written (lost) data at the head of the couch db file is the worst case scenario. The 1.0.2 binary was checked for this issue on both 32 and 64bit platforms & came up clean. Firstly more detail please - William of Ockham[3] is betting on the first two - was this a clean install of CouchDB into a new folder, or overwriting an existing install (not recommended)? - are you running on an NTFS volume for /var/lib/couchdb/*? - what caused you to see a 4GiB limit? errors, logs, unexpected behaviour? History - The 4GiB error is caused by inability in erlang 13B04 and earlier versions to append data to a file that is already > 4 GiB on windows only. This means that: 1) A running CouchDB instance can't grow a db file from < 4 GiB to over 4 GiB without crashing. 2) On restart, the first time that > 4GiB db file has a reason to write - e.g. you add a doc - it starts over-writing not from the end (append) but from the beginning of the DB = lost header = possible lost data (I think depending if compaction is needed or not). This was first resolved in 14B01 only with Juhani's patch. 3 checks you can do on a *clean* folder install of CouchDB 1) confirm what erl/werl version you are using 2) confirm there are no issues at erlang level using werl.exe shell - create a large file (I use one around 128MiB) & use Juhani's quick checker [1] below from the erlang shell to drive it past 4GiB. This should be successful. Note the size & the content of the header of the file. - re-run the quick checker again - each time a segment is written you would normally (e.g. in 1 above) see the file size grow. It does not, and examining the header shows you have overwritten it. 3) less direct but more couchy: - make a large DB just under 4GiB = 4096 * 1024 * 1024 -1 max size. I upload many docs with large attachments for this. - stop couchdb & record header and filesize of db. - upload more docs & increase the db size past 4GiB -- if you can.... - watch couch/erlang die if error is present http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230 - stop couchdb & record header and filesize of db. - start couchdb & relaunch the script again. - confirm header changed, and filesize has not, even though docs were written. > Is there something funky with Server 2003 that would cause this? Not to my knowledge. It would be an OS bug in this case if the API is inconsistent between XP/2003/2008 32 and 64 bit. > Is there a binary or installer available that would work instead? At the moment I suspect > Also do database files that grow to the magical 4GB limit become corrupt or > are they usable on other setups without this issue? All fine so long as you don't restart. Best is to stop couch, move the db files safely, and upgrade. > (I have to ask as the 2003 Server machine is over in china and will take an > age to transfer it for examination) > > Any help on this matter would be greatly appreciated. > Rob > Look for me on irc d_ch & we can go through the gory bits. I'm in UTC + 12 but very keen to help here. A+ Dave [3] http://www.phys.ncku.edu.tw/mirrors/physicsfaq/General/occam.html [2] http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230 [1] filetest.erl -module(filetest). -export([main/0]). main() -> {ok, Binary}=file:read_file("128MiB"), {ok, WriteDescr} = file:open("grow_past_4_GiB.file", [raw, append]), loop(1000, WriteDescr,Binary), file:close(WriteDescr). loop(0,_NotNeeded,_NotNeeded) -> ok; loop(N,WriteDescr,Binary) -> file:write(WriteDescr,Binary), io:format("wrote ~w\n", [N]), loop(N-1,WriteDescr,Binary).
