Dave,
I hope you are well
I too began to look at this a couple of nights ago.
The process that I was going to follow;
Establish a base case
Write some file IO tests
Apply these tests to the Couchdb environment and see what
differences popped out.
At the very least I thought that I could produce some tests to ascertain
why 1.01 is misbehaving on windows, if Erlang is problematic and to use
as future tests for new releases of Couchdb on windows.
I downloaded the latest full Erlang windows binary as my base case
(currently R14B) and wrote the following very simple bit of code as my
first test
<code>
-module(filetest).
-export([main/0]).
main() ->
{ok, Binary}=file:read_file("blanks"),
{ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
loop(1000, WriteDescr,Binary),
file:close(WriteDescr).
loop(0,_NotNeeded,_NotNeeded) -> ok;
loop(N,WriteDescr,Binary) ->
file:write(WriteDescr,Binary),
loop(N-1,WriteDescr,Binary).
</code>
Note blanks is a zero filled file of about 2.2MB that i originally used
to confirm that i had the same issues as Peter first reported.
Run with a loop of 10 and you get a twenty MB file ........... nice
Run again with a loop of 10 and you get a Forty MB file ......... nice
delete filetest.txt
Run again with a loop of 2000 and you get a file in excess of 4GB
.......... nice .......... but slow
delete filetest.txt
Run again with a loop of 4000 and you get a file in excess of 8GB
...........nice............but very slow
Now everything seems fine and dandy but .......
Delete filetest.txt
Run with a loop of 10 and you get a twenty MB file ........... nice
Run again with a loop of 10 and you get a Forty MB file ......... nice
Run again with a loop of 2000 and you get a file in excess of 4GB
.......... nice .......... but slow
Run again with a loop of 2000 and the program begins to "append" from
the beginning of the file so the file does not grow but instead gets
overwritten. This behaviour seems to occur at the 4GB file size on
change of file descriptor ............. not nice ....... not good.
I've had a look at the way Erlang is compiled for windows and there are
a couple of flags pertaining to large file support. Over the next few
days I will attempt my own compilation of Erlang with these flags set
and report back. (Need to get hold of VC++ first)
regards
Cliff
On 29/09/10 01:02, Dave Cottlehuber wrote:
On 26 September 2010 10:11, Peter Somers<petsom...@gmail.com> wrote:
Hi Dave,
No luck. Couchdb crashed again :-(
I really hope someone can help me on this issue...
Best regards,
Peter
Hi Peter
There may be 2 things going on here.
- your current crash appears to be due to file corruption;
- {{badmatch,{error,file_corruption}},
this is after couch reads an md5 hash in read_raw_iolist_int - maybe
due to going over length, I can't tell.
- your first crash is the one I can replicate - the whole VM dies with
a memory allocation issue. I can replicate this on my win7 enterprise
couch running 1.01, but not 0.11.0.
I can load the DB up to this size; 0xffd9a0f6d, couchdb dies:
{{badmatch,{error,enomem}}, [{couch_file,read_raw_iolist_int,3},
{couch_file,handle_call,3},
Restarting windows/couchdb& then curl http://localhost:5984/fatso yields
{"error":"kill","reason":"{gen_server,call,
[couch_server,
{open,<<\"fatso\">>,
[{user_ctx,
{user_ctx,null,
[<<\"_admin\">>],
<<\"{couch_httpd_auth,
default_authentication_handler}\">>}}]},
infinity]}"}
Anyway some more work ahead to identify exactly where this comes from
- thanks for raising the jira.
cheers
Dave
