Do you have a stack trace for the crash. We are interested in the reason for the crash.
The problem was attributed to the slowness in loading the deduplication tables. When a master disk is added to a pool the initial set of deduplication tables are sequential on disk and these are quickly loaded back into memory on a reboot. The size of these tables are around 1/8 the size of the memory. The second set of deduplication tables are added on demand when more table entries are needed and the first set of tables are full. The problem is with the second set, the location of these tables can be spread across the disk. For your setup as per our calculations around 2GB are the sequential tables and 14GB are the second set. The disk is a RAID 1 disk and it seems that with just the max speed of two disks reading back 14GB of data which are mainly 4K blocks can take many minutes to hours. There are two things which we are doing. 1. Create all the deduplication tables sequentially. This will mean that only one storage pool can maintain the deduplication tables (mostly the Default pool). Most configurations are this way => This change is complete 2. For existing implementation we realign the random blocks to sequential blocks using a command line tool => This is still being implemented and ETA is the end of this month. You would need this feature On Mon, Nov 6, 2017 at 1:42 PM, <b.nauta@goldspark.solutions> wrote: > What's the status on the fix? The server just crashed and rebooting is going > to take a couple of hours again due to this bug. This is unacceptable in a > production environment. > > Op dinsdag 5 september 2017 14:07:58 UTC+2 schreef quadstor: >> >> Please send across the following files >> >> /var/log/dmesg* >> /var/log/kern.log* >> /var/log/syslog* >> >> And the output of >> dmesg >> top -b -d 1 -n 60 >> >> On Tue, Sep 5, 2017 at 5:31 PM, wrote: >> > Due to the load, I unfortunately can't access the GUI. Is there another >> > way >> > to create the diagnostics? >> > >> > The shutdown was due to a power outage. The underlying storage is a >> > hardware >> > RAID 1 on PERC 6/i; the RAID BIOS tells me everything is fine with both >> > disks. After booting the server after the power outage, the RAID >> > controller >> > was busy writing it's BBU cache to disk for approx. a minute. >> > >> > Op dinsdag 5 september 2017 13:45:54 UTC+2 schreef quadstor: >> >> >> >> Please send the diagnostics to sup...@quadstor.com >> >> >> >> What was the reason for the shutdown ? gdevq is the thread which >> >> processes incoming requests, and if that is blocked it could indicate >> >> a problem with the underlying storage. >> >> >> >> On Tue, Sep 5, 2017 at 4:54 PM, wrote: >> >> > I am running Quadstor Storage Virtualization 3.2.12.1 on Debian 7 as >> >> > iSCSI >> >> > target for my XenServer 7 servers. After an unclean shutdown, >> >> > Quadstor >> >> > suddenly slows down the entire machine to a grinding halt, up to a >> >> > point >> >> > where there are multiple messages that 'gdevq' blocked for more than >> >> > 120 >> >> > seconds. The XenServers also cannot access the files on the Quadstor >> >> > machine. >> >> > >> >> > After uninstalling Quadstor, the machine is as fast as it has always >> >> > been. >> >> > Reinstalling Quadstor didn't change that, but after rescanning the >> >> > database, >> >> > all the symptoms came back and I still can't access my data. >> >> > >> >> > Does anyone have any idea what to do? I can't access any of my data >> >> > at >> >> > the >> >> > moment. >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "QUADStor Storage Virtualization" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> > send >> >> > an >> >> > email to quadstor-vir...@googlegroups.com. >> >> > For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "QUADStor Storage Virtualization" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to quadstor-vir...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "QUADStor Storage Virtualization" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to quadstor-virt+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "QUADStor Storage Virtualization" group. To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-virt+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
