Hello all,
I wrote earlier with questions about mogstored dying, and someone
suggested that I should replace my old RedHat 8 machines with a newer
build.
I've rebuilt the cluster (16 nodes) using CentOS5.1 and the results
were a bit better at first. But on the first real large-file test, I
still get problems with some of the chunks not being saved properly,
and mogtool is not able to reliably store large files for me.
I would love it if someone could tell me if I'm just doing something
wrong, if I have some sort of hidden problem, or if mogtool is just
not complete, obsolete or what. Some of the files I need to store are
30G (definitely larger than the RAM on the client or storage nodes.
If mogtool is known to have problems, what other options are there for
dealing with very very large files? I'd really like mogile for this
but I'd rather not write my own scheme for handling large files (these
would be like a 1.6 TB directory as a tar stream, or a number of 30G
files)... Is mogile just not a good option here? should I start
evalulating other solutions like hapdoop?
Here's the error generated by mogtool.
WARNING: Unable to save file 'collect-20080513-vol9,2201': Unable to
create new file at /usr/bin/mogtool line 812, <Sock_minime336:7001>
line 11907.
MogileFS backend error message: unknown_key unknown_key
System error message: Unable to create new file at /usr/bin/mogtool
line 812, <Sock_minime336:7001> line 11907.
This was try #9707 and it's been 10578.37 seconds since we first
tried. Retrying...
WARNING: Unable to save file 'collect-20080513-vol9,2203': Unable to
create new file at /usr/bin/mogtool line 812, <Sock_minime336:7001>
line 11907.
MogileFS backend error message: unknown_key unknown_key
System error message: Unable to create new file at /usr/bin/mogtool
line 812, <Sock_minime336:7001> line 11907.
This was try #9705 and it's been 10572.94 seconds since we first
tried. Retrying..
multi/IO/Handle.pm line 399
MogileFS backend error message: unknown_key unknown_key
System error message: MogileFS::NewHTTPFile: unable to write to any
allocated storage node at /usr/lib64/perl5/5.8.5/x86_64-linux-thread-
multi/IO/Handle.pm line 399
This was try #1 and it's been 10.79 seconds since we first tried.
Retrying...
w
