Alright guys, I'm at the end of my rope here. For those that haven't seen
my previous emails here's the (not so) quick breakdown:
Overview:
FreeBSD ?? - 7.4 never crash
FreeBSD 8.0 - 8.2 crashes
FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our
production at this time, and we were hoping we could base some stuff on
8.3 for long term stability...)
ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others.
History:
Over the course of the last 2 years we've been banging our heads on the
wall. VMWare is done debugging this. They claim it's not a VMWare issue.
They can't identify what the heck happens. We had a glimmer of hope with
ESXi 5.0 fixing it because we never saw any crashes in the handful of
deployments, but our dreams were crushed today -- two days before an
outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server
and FreeBSD 8.2 guest crashed.
Crash Details:
The keyboard/mouse usually stops responding for input on the console;
normally we can't type in a username or password. However, we can switch
VTs.
If there's a shell on the console and we can type, we can only run things
in memory. Any time we try to access the disk it will hang indefinitely.
The server still has network access. We can ping it without issue. SSH of
course kicks you out because it can't do any I/O.
If we were to serve a lightweight http server off a memory backed
filesystem I'm confident it would run just fine as long as it wasn't
logging or anything.
On ESXi you see that there is a CPU spike of 100% that goes on
indefinitely. No idea what the FreeBSD OS itself thinks it is doing
because we can't run top during the crash.
This crash can affect a server and happen multiple times a week. It can
also not show up for 180 days or more. But it does happen. The server can
be 100% idle and crash. We have servers that do more I/O than the ones
that crash could ever attempt to do and these don't crash at all.
Completely inexplicable.
Things we've looked into:
Nothing about the installed software matters. We've tried cross
referencing the crashed servers by the programs they run but the base OS
is the only common denominator due to the wide variety of servers it has
affected.
Storage doesn't matter. We've tried different iSCSI SANs, we've tried
different switches, we've tried local datastores on the ESXi servers
themselves.
HP servers, Dell servers -- doesn't seem to matter either. (All with
latest firmwares, BIOSes, etc)
VMWare gave us a ton of debugging tasks, and we've given them gigabytes of
debugging info and data; they can't find anything.
VMWare tools -- with, without, using open-vm-tools makes no difference. I
think we've done a fair job ruling out VMWare.
I think we've finally found enough data that this is definitely something
in the FreeBSD world. I'm going to begin prepping some of the known crashy
servers with more debugging. Any suggestions on what I should build the
kernel with? They never do a proper panic, but I definitely want to at
least *try* to get into the debugger the next time it crashes. And when it
crashes, what the heck should I be running? I've never played with the KDB
before...
Thank you for any suggestions and help you can give me....
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
