I'm thinking that the issue is simply with zfs destroy, not with dedup or
compression.
Yesterday I decided to do some iscsi testing, I created a new dataset in my
pool, 1TB. I did not use compression or dedup.
After copying about 700GB of data from my windows box (NTFS on top of the iscsi
In this last iteration, I switched to a completely different box with twice the
resources. Somehow, from the symptoms, I don't think trying it on one of the
48 or 128GB servers at work is going to change the outcome. The hang happens
too fast. it seems like something in the destroy is causing
If pool isnt rpool you might to want to boot into singleuser mode (-s after
kernel parameters on boot) remove /etc/zfs/zpool.cache and then reboot.
after that you can merely ssh into box and watch iostat while import.
Yours
Markus Kovero
___
On Sat, Jan 2, 2010 at 13:10, Markus Kovero markus.kov...@nebula.fi wrote:
If pool isnt rpool you might to want to boot into singleuser mode (-s after
kernel parameters on boot) remove /etc/zfs/zpool.cache and then reboot.
after that you can merely ssh into box and watch iostat while import.
If pool isnt rpool you might to want to boot into
singleuser mode (-s after kernel parameters on boot)
remove /etc/zfs/zpool.cache and then reboot.
after that you can merely ssh into box and watch
iostat while import.
Yours
Markus Kovero
___
That's the thing, the drive lights aren't blinking,
but I was thinking maybe the writes are going so slow
that it's possible they aren't registering. And since
I can't keep a running iostat, Ican't tell if
anything is going on. I can however get into the
KMDB. is there something in there
Hey Markus,
Thanks for the suggestion, but as stated in the thread, I am booting using
-s -kv -m
verbose and deleting the cache file was one of the first troubleshooting
steps we and
the others affected did.The other problem is that we were all starting an
iostat at
the
Yeah, still no joy. I moved the disks to another machine altogether with 8gb
and a quad core intel versus the dual core amd I was using and it still just
hangs the box on import. this time I did a nohup zpool import -fFX vault after
booting off the b130 live dvd on this machine into single
On Jan 1, 2010, at 2:23 PM, tom wagner wrote:
Yeah, still no joy. I moved the disks to another machine altogether
with 8gb and a quad core intel versus the dual core amd I was using
and it still just hangs the box on import. this time I did a nohup
zpool import -fFX vault after booting
That's the thing, the drive lights aren't blinking, but I was thinking maybe
the writes are going so slow that it's possible they aren't registering. And
since I can't keep a running iostat, Ican't tell if anything is going on. I
can however get into the KMDB. is there something in there that
Yeah, still no joy on getting my pool back. I think
I might have to try grabbing another server with a
lot more memory and slapping the HBA and the drives
in that. Can ZFS deal with a controller change?
Just some more info that 'may' help.
After I upgraded to 8GB of RAM, I did not limit the
Yeah, still no joy on getting my pool back. I think I might have to try
grabbing another server with a lot more memory and slapping the HBA and the
drives in that. Can ZFS deal with a controller change?
--
This message posted from opensolaris.org
On Dec 30, 2009, at 10:26 AM, tom wagner wrote:
Yeah, still no joy on getting my pool back. I think I might have to
try grabbing another server with a lot more memory and slapping the
HBA and the drives in that. Can ZFS deal with a controller change?
Yes.
-- richard
I booted the snv_130 live cd and ran zpool import -fFX and it took a day, but
it imported my pool and rolled it back to a previous version. I haven't looked
to see what was missing, but I didn't need any of the changes over the last few
weeks.
Scott
--
This message posted from
I booted the snv_130 live cd and ran zpool import
-fFX and it took a day, but it imported my pool and
rolled it back to a previous version. I haven't
looked to see what was missing, but I didn't need any
of the changes over the last few weeks.
Scott
I'll give it a shot. Hope this works,
I got my pool back
Did a rig upgrade (new motherboard, processor, and 8 GB of RAM), re-installed
opensolaris 2009.06, did an upgrade to snv_130, and did the import!
The import only took about 4 hours!
I have a hunch that I was running into some sort of issue with not having
enough RAM
I should note that my import command was:
zpool import -f vault
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
4 Gigabytes. The hang on my system happens much faster. I can watch the drives
light up and run iostat but 3 minutes in like clockwork everything gets hung
and I'm left with a blinking cursor at the console that newlines but doesn't do
anything. Although if I run kmdb and hit f1-a I can get
It sounds like you have less data on yours, perhaps that is why yours freezes
faster.
Whatever mine is doing during the import, it reads my disks now for nearly
24-hours, and then starts writing to the disks.
The reads start out fast, then they just sit, going at something like 20k /
second
Here is iostat output of my disks being read:
r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
45.30.0 27.60.0 0.0 0.60.0 13.3 0 60 c3d0
44.30.0 27.00.0 0.0 0.30.07.7 0 34 c3d1
43.50.0 27.40.0 0.0 0.50.0
I have a pool in the same state. I deleted a file set that was compressed and
deduped and had a bunch of zero blocks in it. The delete ran for a while and
then it hung. Trying to import with any combination of -f or -fF or -fFX gives
the same results you guys get. zdb -eud shows all my file
One thing that bugged me is that I can not ssh as myself to my box when a zpool
import is running. It just hangs after accepting my password.
I had to convert root from a role to a user and ssh as root to my box.
I now know why this is, when I log in, /usr/sbin/quota gets called. This must
do
I am having the exact same problem after destroying a dataset with a few
gigabytes of data and dedup. I type zfs destroy vault/virtualmachines which
was a zvol with dedup turned on and the server hung, couldn't ping, couldn't
get on the console. Next bootup same thing just hangs when
I still haven't given up :)
I moved my Virtual Machines to my main rig (which gets rebooted often, so this
is 'not optimal' to say the least) :)
I have since upgraded to 129. I noticed that even if timeslider/autosnaps are
disabled, a zpool command still gets generated every 15 minutes. Since
Just wondering,
How much RAM is in your system?
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
I don't mean to sound ungrateful (because I really do appreciate all the help I
have received here), but I am really missing the use of my server.
Over Christmas, I want to be able to use my laptop (right now, it's acting as a
server for some of the things my OpenSolaris server did). This means
Ok, dump uploaded!
Thanks for your upload
Your file has been stored as /cores/redshirt-vmdump.0 on the Supportfiles
service.
Size of the file (in bytes) : 1743978496.
The file has a cksum of : 2878443682 .
Ah!
Ok, I will give this a try tonight! Thanks.
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Ok, I have started my import after using the -k on my kernel line (I just did a
test dump using this method just to make sure it works ok, and it does).
I have also added the following to my /etc/system file and rebooted:
set snooping=1
According to this page:
Ok, my console is 100% completely hung, not gonna be able to enter any commands
when it freezes.
I can't even get the numlock light to change it's status.
This time I even plugged in a PS/2 keyboard instead of USB thinking maybe it
was USB dying during the hang, but not so.
I have hard
Ok, this is the script I am running (as a background process). This script
doesn't matter much, it's just here for reference, as I'm running into problems
just running the savecore command while the zpool import is running.
On 18.12.09 07:13, Jack Kielsmeier wrote:
Ok, my console is 100% completely hung, not gonna be able to enter any
commands when it freezes.
I can't even get the numlock light to change it's status.
This time I even plugged in a PS/2 keyboard instead of USB thinking maybe it
was USB dying during
Jack,
We'd like to get a crash dump from this system to
determine the root
cause of the system hang. You can get a crash dump
from a live system
like this:
# savecore -L
dumping to /dev/zvol/dsk/rpool/dump, offset 65536,
content: kernel
0:18 100% done
0% done: 49953 pages dumped,
In some cases, root logged into the console can still function,
but if not, then you'd need to shutdown the system and run sync.
I can walk you through those steps if you need them.
If you've been tortured long enough, then feel free to upload a
crash dump and let us know.
Thanks,
Cindy
On
Thanks for the update, it's no help to you of course, but I'm watching your
progress with interest. Your progress updates are very much appreciated.
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
Thanks.
I've decided now to only post when:
1) I have my zfs pool back
or
2) I give up
I should note that there are periods of time where I can ping my server
(rarely), but most of the time not. I have not been able to ssh into it, and
the console is hung (minus the little blinking cursor).
On Dec 15, 2009, at 5:50, Jack Kielsmeier jac...@netins.net wrote:
Thanks.
I've decided now to only post when:
1) I have my zfs pool back
or
2) I give up
I should note that there are periods of time where I can ping my
server (rarely), but most of the time not. I have not been able to
On Dec 15, 2009, at 5:50, Jack Kielsmeier
jac...@netins.net wrote:
Thanks.
I've decided now to only post when:
1) I have my zfs pool back
or
2) I give up
I should note that there are periods of time where
I can ping my
server (rarely), but most of the time not. I have
It's been over 72 hours since my last import attempt.
System still is non-responsive. No idea if it's doing anything
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
My system was pingable again, unfortunately I disabled all services such as
ssh. My console was still hung, but I was wondering if I had hung USB crap
(since I use a USB keyboard and everything had been hung for days).
I force rebooted and the pool was not imported :(. I started the process off
My import is still going (I hope, as I can't confirm since my system appears to
be totally locked except for the little blinking console cursor), been well
over a day.
I'm less hopeful now, but will still let it do it's thing for another couple
of days.
--
This message posted from
zpool import done! Back online.
Total downtime for 4TB pool was about 8 hours, don't know how much of this was
completing the destroy transaction.
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
zpool import done! Back online.
Total downtime for 4TB pool was about 8 hours, don't
know how much of this was completing the destroy
transaction.
Lucky You! :)
My box has gone totally unresponsive again :( I cannot even ping it now and I
can't hear the disks thrashing.
--
This message
I have disabled all 'non-important' processes (gdm, ssh, vnc, etc). I am now
starting this process locally on the server via the console with about 3.4 GB
free of RAM.
I still have my entries in /etc/system for limiting how much RAM zfs can use.
--
This message posted from opensolaris.org
On Wed, 9 Dec 2009, Markus Kovero wrote:
From what I've noticed, if one destroys dataset that is say 50-70TB and reboots
before destroy is finished, it can take up to several _days_ before it's back
up again.
So, nowadays I'm doing rm -fr BEFORE issuing zfs destroy whenever possible.
It
I have disabled all 'non-important' processes (gdm,
ssh, vnc, etc). I am now starting this process
locally on the server via the console with about 3.4
GB free of RAM.
I still have my entries in /etc/system for limiting
how much RAM zfs can use.
Going on 10 hours now, still importing.
I have disabled all 'non-important' processes
(gdm,
ssh, vnc, etc). I am now starting this process
locally on the server via the console with about
3.4
GB free of RAM.
I still have my entries in /etc/system for
limiting
how much RAM zfs can use.
Going on 10 hours now, still
I wonder if you are hitting this bug:
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6905936
Deleting large files or filesystems on a dedup=on filesystem stalls the
whole system
Cindy
On 12/09/09 16:41, Jack Kielsmeier wrote:
I have disabled all 'non-important' processes
(gdm,
Ah that could be it!
This leaves me hopeful, as it looks like that bug says it'll eventually finish!
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
Hi, are you sure zfs isnt just going thru transactions after forcibly stopping
zfs destroy?
Sometimes (always) it seems zfs/zpool commands just hang if you destroy larger
filesets, in reality zfs is just doing its job, if you reboot server during
dataset destroy it will take some time to come
I waited about 20 minutes or so. I'll try your suggestions tonight.
I didn't look at iostat. I just figured it was hung after waiting that long,
but now that I know it can take a very long time, I will watch it and make sure
it's doing something.
Thanks. I'll post my results either tonight or
On Tue, Dec 8, 2009 at 8:23 AM, Jack Kielsmeier jac...@netins.net wrote:
I waited about 20 minutes or so. I'll try your suggestions tonight.
I didn't look at iostat. I just figured it was hung after waiting that
long, but now that I know it can take a very long time, I will watch it and
make
The pool is roughly 4.5 TB (Raidz1, 4 1.5 TB Disks)
I didn't attempt to destroy the pool, only a dataset within the pool. The
dataset is/was about 1.2TB.
System Specs
Intel Q6600 (2.4 Ghz Quad Core)
4GB RAM
2x 500 GB drives in zfs mirror (rpool)
4x 1.5 TB drives in zfs raidz1 array (vault)
The
It's been about 45 minutes now since I started trying to import the pool.
I see disk activity (see below)
What concerns me is my free memory keeps shrinking as time goes on. Now have
185MB free out of 4 gigs (and 2 gigs of swap free).
Hope this doesn't exhaust all my memory and freeze my box.
On Tue, Dec 8, 2009 at 5:38 PM, Jack Kielsmeier jac...@netins.net wrote:
It's been about 45 minutes now since I started trying to import the pool.
I see disk activity (see below)
What concerns me is my free memory keeps shrinking as time goes on. Now
have 185MB free out of 4 gigs (and 2
Ah, good to know! I'm learning all kinds of stuff here :)
The command (zpool import) is still running and I'm still seeing disk activity.
Any rough idea as to how long this command should last? Looks like each disk is
being read at a rate of 1.5-2 megabytes per second.
Going worst case,
On Tue, Dec 8, 2009 at 6:36 PM, Jack Kielsmeier jac...@netins.net wrote:
Ah, good to know! I'm learning all kinds of stuff here :)
The command (zpool import) is still running and I'm still seeing disk
activity.
Any rough idea as to how long this command should last? Looks like each disk
The server just went almost totally unresponsive :(
I still hear the disks thrashing. If I press keys on the keyboard, my login
screen will not show up. I had a VNC session hang and can no longer get back in.
I can try to ssh to the server, I get prompted for my username and password,
but it
I just hard-rebooted my server. I'm moving off my VM to my laptop so it can
continue to run :)
Then, if it freezes again I'll just let it sit, as I did hear the disks
thrashing.
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
On Tue, Dec 8, 2009 at 10:16 PM, Jack Kielsmeier jac...@netins.net wrote:
I just hard-rebooted my server. I'm moving off my VM to my laptop so it can
continue to run :)
Then, if it freezes again I'll just let it sit, as I did hear the disks
thrashing.
--
As long as you've already
Ok,
When searching for how to do that, I see that it requires a modification to
/etc/system.
I'm thinking I'll limit it to 1GB, so the entry (which must be in hex) appears
to be:
set zfs:zfs_arc_max = 0x4000
Then I'll reboot the server and try the import again.
Thanks for the continued
Upon further research, it appears I need to limit both the ncsize and the
arc_max. I think I'll use:
set ncsize = 0x3000
set zfs:zfs_arc_max = 0x1000
That should give me a max of 1GB used between both.
If I should be using different values (or other settings), please let me know :)
--
Yikes,
Posted too soon. I don't want to set my ncsize that high!!! (Was thinking the
entry was memory, but it's entries).
set ncsize = 25
set zfs:zfs_arc_max = 0x1000
Now THIS should hopefully only make it so the process can take around 1GB of
RAM.
--
This message posted from
Ok, I have started the zpool import again. Looking at iostat, it looks like I'm
getting compatible read speeds (possibly a little slower):
extended device statistics
r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.00.00.00.0
On Tue, Dec 8, 2009 at 6:36 PM, Jack Kielsmeier
jac...@netins.net wrote:
Ah, good to know! I'm learning all kinds of stuff
here :)
The command (zpool import) is still running and I'm
still seeing disk activity.
Any rough idea as to how long this command should
last? Looks like each
On Wed, Dec 9, 2009 at 10:41 AM, Brent Jones br...@servuhome.net wrote:
I submitted a bug a while ago about this:
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855208
I'll escalate since I have a support contract. But yes, I see this as
a serious bug, I thought my machine had
Am in the same boat, exactly. Destroyed a large set and rebooted, with a scrub
running on the same pool.
My reboot stuck on Reading ZFS Config: * for several hours (disks were
active). I cleared the zpool.cache from single-user and am doing an import (can
boot again). I wasn't able to boot my
From what I've noticed, if one destroys dataset that is say 50-70TB and
reboots before destroy is finished, it can take up to several _days_ before
it's back up again.
So, nowadays I'm doing rm -fr BEFORE issuing zfs destroy whenever possible.
Yours
Markus Kovero
-Original Message-
68 matches
Mail list logo
