Re: ggate + gmirror write performance woes

2007-04-09 Thread Sven Willenberger 
On Fri, 2007-04-06 at 16:18 +0300, Nikolay Pavlov wrote:
 On Thursday,  5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote:
  On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
   Dmitriy Kirhlarov wrote:
On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
I am trying to set up a HA type system involving two identical boxes 
and
have gone through the following to set up the systems:
   
Slave server: 
ggated -R 196608 -S 196608
(exporting /dev/amrd1 )
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 131072

Try
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

Also, try increase this sysctls with
net.inet.tcp.rfc1323=1

I use it on FreeBSD 5.x with:
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

ggated -R 1048576 -S 1048576
ggatec -R 1048576 -S 1048576

WBR.
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
   
   
   I have seen sustained writes of 30Mb/s using the following configuration:
   
   cat /boot/loader.conf
   kern.ipc.nmbclusters=32768
   
   cat /etc/sysctl.conf
   net.inet.tcp.sendspace=1048576
   net.inet.tcp.recvspace=1048576
   
   Server:
   /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
   
   Client:
   /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
   /dev/amrd0s2
   
   The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
   adaptive read ahead and write back caching.
   
   Tom
  
  I have tried both the settings ideas suggested above but I cannot even
  get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
  anything higher that 131072 results in ggated bailing with the error:
  # ggated -v -a 10.10.0.19
  info: Reading exports file (/etc/gg.exports).
  debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
  debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
  info: Exporting 2 object(s).
  error: Cannot open stream socket: No buffer space available.
  error: Exiting.
 
 For values of net.inet.tcp.{send,recv}space more than
 524288 you also need to adjust kern.ipc.maxsockbuf
 
 Try this configuration for example:
 kern.ipc.maxsockbuf=2049152
 net.inet.tcp.recvspace=1024576
 net.inet.tcp.sendspace=1024576
 

kern.ipc.maxsockbuf was the issue here; I increased its value and now I
no longer get the buffer space error. Furthermore, the write speed issue
was also tied to a hardware raid controller issue. After fixing that
issue, and with just the following:

kern.ipc.maxsockbuf=1048576
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072

I can start ggated with -R 262144 -S 262144 as well as the ggatec and
see write speeds of 60+MB/s. I may play around with the settings more
(and see if any further speed improvements occur), but this is quite
acceptable at this point. (For the record nmbclusters is set to 32768).

The next part of the project will be writing the freevrrp failover
scripts to deal with I/O locking issues that will happen if the ggated
server fails, etc.

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-09 Thread Dmitriy Kirhlarov 
Hi!

On Mon, Apr 09, 2007 at 12:55:43PM -0400, Sven Willenberger wrote:
 The next part of the project will be writing the freevrrp failover
 scripts to deal with I/O locking issues that will happen if the ggated
 server fails, etc.

check carp(4) and ports/net/ifstated. You can use identical advbase
configuration on both nodes and prevent service migration back from
second node to first, if first node will reboot.
also, take a look one{start,stop,restart} options for rcNG designed
initscripts.

WBR
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-06 Thread Dmitriy Kirhlarov 
Hi!

On Thu, Apr 05, 2007 at 04:15:35PM -0400, Sven Willenberger wrote:

 I have tried both the settings ideas suggested above but I cannot even
 get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
 
 setting net.inet.tcp.{send,recv}space to 131072 allows me to start

Now you have good chance to check differences with other recommended
sysctl's. :)

WBR
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-06 Thread Tom Judge 

Sven Willenberger wrote:

On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:

Dmitriy Kirhlarov wrote:

On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:

I am trying to set up a HA type system involving two identical boxes and
have gone through the following to set up the systems:

Slave server: 
ggated -R 196608 -S 196608

(exporting /dev/amrd1 )
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 131072

Try
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

Also, try increase this sysctls with
net.inet.tcp.rfc1323=1

I use it on FreeBSD 5.x with:
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

ggated -R 1048576 -S 1048576
ggatec -R 1048576 -S 1048576

WBR.
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


I have seen sustained writes of 30Mb/s using the following configuration:

cat /boot/loader.conf
kern.ipc.nmbclusters=32768

cat /etc/sysctl.conf
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576

Server:
/sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports

Client:
/sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
/dev/amrd0s2


The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
adaptive read ahead and write back caching.


Tom


I have tried both the settings ideas suggested above but I cannot even
get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
anything higher that 131072 results in ggated bailing with the error:
# ggated -v -a 10.10.0.19
info: Reading exports file (/etc/gg.exports).
debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
info: Exporting 2 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.

setting net.inet.tcp.{send,recv}space to 131072 allows me to start
ggated with the default R and S values of 131072; anything higher
results in no buffer space errors. At 131072 ggated starts but then I
cannot even open a new connection (like ssh) to the server as the ssh
client bails with no buffer space available.


Did you also set kern.ipc.nmbclusters=32768 in /boot/loader.conf and 
reboot?  It sounds like you did not as this is the exact same problem I 
came across before adjusting that value.


SNIP


This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
Raid using LSiMegaRaid.


Do you have the cache BBU fitted (Batery Backup Unit) and the array 
caching set to write back?  Also have you tested writing to the array 
locally without ggate to test the write speed?




The odd thing is that even after I set the send and recvspace down to
values like 65536, I continue to get the no buffer error when trying to
connect to it remotely again.



I found that the easyest way to fix this was to reboot the system with 
good values for net.inet.tcp.{send,recv}space.




___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-06 Thread Nikolay Pavlov 
On Thursday,  5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote:
 On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
  Dmitriy Kirhlarov wrote:
   On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
   I am trying to set up a HA type system involving two identical boxes and
   have gone through the following to set up the systems:
  
   Slave server: 
   ggated -R 196608 -S 196608
   (exporting /dev/amrd1 )
   net.inet.tcp.sendspace: 65536
   net.inet.tcp.recvspace: 131072
   
   Try
   net.local.stream.recvspace=65535
   net.local.stream.sendspace=65535
   
   Also, try increase this sysctls with
   net.inet.tcp.rfc1323=1
   
   I use it on FreeBSD 5.x with:
   net.inet.tcp.sendspace=131072
   net.inet.tcp.recvspace=131072
   net.local.stream.recvspace=65535
   net.local.stream.sendspace=65535
   
   ggated -R 1048576 -S 1048576
   ggatec -R 1048576 -S 1048576
   
   WBR.
   Dmitriy
   ___
   freebsd-stable@freebsd.org mailing list
   http://lists.freebsd.org/mailman/listinfo/freebsd-stable
   To unsubscribe, send any mail to [EMAIL PROTECTED]
  
  
  I have seen sustained writes of 30Mb/s using the following configuration:
  
  cat /boot/loader.conf
  kern.ipc.nmbclusters=32768
  
  cat /etc/sysctl.conf
  net.inet.tcp.sendspace=1048576
  net.inet.tcp.recvspace=1048576
  
  Server:
  /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
  
  Client:
  /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
  /dev/amrd0s2
  
  The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
  adaptive read ahead and write back caching.
  
  Tom
 
 I have tried both the settings ideas suggested above but I cannot even
 get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
 anything higher that 131072 results in ggated bailing with the error:
 # ggated -v -a 10.10.0.19
 info: Reading exports file (/etc/gg.exports).
 debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
 debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
 info: Exporting 2 object(s).
 error: Cannot open stream socket: No buffer space available.
 error: Exiting.

For values of net.inet.tcp.{send,recv}space more than
524288 you also need to adjust kern.ipc.maxsockbuf

Try this configuration for example:
kern.ipc.maxsockbuf=2049152
net.inet.tcp.recvspace=1024576
net.inet.tcp.sendspace=1024576

 
 setting net.inet.tcp.{send,recv}space to 131072 allows me to start
 ggated with the default R and S values of 131072; anything higher
 results in no buffer space errors. At 131072 ggated starts but then I
 cannot even open a new connection (like ssh) to the server as the ssh
 client bails with no buffer space available.
 
 more information:
 # netstat -m
 514/641/1155 mbufs in use (current/cache/total)
 512/284/796/32768 mbuf clusters in use (current/cache/total/max)
 512/256 mbuf+clusters out of packet secondary zone in use
 (current/cache)
 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
 0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
 0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
 1152K/728K/1880K bytes allocated to network (current/cache/total)
 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
 0/4/6656 sfbufs in use (current/peak/max)
 0 requests for sfbufs denied
 0 requests for sfbufs delayed
 0 requests for I/O initiated by sendfile
 0 calls to protocol drain routines
 
 This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
 Raid using LSiMegaRaid.
 
 The odd thing is that even after I set the send and recvspace down to
 values like 65536, I continue to get the no buffer error when trying to
 connect to it remotely again.
 
 Sven
 
 ___
 freebsd-stable@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-stable
 To unsubscribe, send any mail to [EMAIL PROTECTED]

-- 
==  
- Best regards, Nikolay Pavlov. ---
==  

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

ggate + gmirror write performance woes

2007-04-05 Thread Sven Willenberger 
I am trying to set up a HA type system involving two identical boxes and
have gone through the following to set up the systems:

Slave server: 
ggated -R 196608 -S 196608
(exporting /dev/amrd1 )
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 131072



Master server:
ggatec create -u 0 -R 196608 -S 196608 -o rw [slaveip] /dev/amrd1
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 65536


#gmirror status
  NameStatus  Components
mirror/gm0  COMPLETE  amrd1s1
  ggate0s1

the two servers are connected to each other via their 2ndary physical
gigE interfaces using cat6 crossover cable. (Netperf shows 890 Mbps at
95% confidence).

softupdates are enable on gm0 (though this does not affect the results).

The results:
/usr/bin/time -h cp testfile64M /data1
28.62s real 0.00s user  0.16s sys

and this is very consistent ... about 3 MB/s over repeated runs 

dd if=/dev/zero of=/data1/testfile32M2 bs=32k count=1024
1024+0 records in
1024+0 records out
33554432 bytes transferred in 16.122641 secs (2081199 bytes/sec)

What else can I tune here to make this functional? If I increase
recvspace and sendspace much beyond those numbers, ggated will not start
claiming to not have enough buffer space.

Sven

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-05 Thread Dmitriy Kirhlarov 
On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
 I am trying to set up a HA type system involving two identical boxes and
 have gone through the following to set up the systems:
 
 Slave server: 
 ggated -R 196608 -S 196608
 (exporting /dev/amrd1 )
 net.inet.tcp.sendspace: 65536
 net.inet.tcp.recvspace: 131072

Try
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

Also, try increase this sysctls with
net.inet.tcp.rfc1323=1

I use it on FreeBSD 5.x with:
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

ggated -R 1048576 -S 1048576
ggatec -R 1048576 -S 1048576

WBR.
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-05 Thread Tom Judge 

Dmitriy Kirhlarov wrote:

On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:

I am trying to set up a HA type system involving two identical boxes and
have gone through the following to set up the systems:

Slave server: 
ggated -R 196608 -S 196608

(exporting /dev/amrd1 )
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 131072


Try
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

Also, try increase this sysctls with
net.inet.tcp.rfc1323=1

I use it on FreeBSD 5.x with:
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

ggated -R 1048576 -S 1048576
ggatec -R 1048576 -S 1048576

WBR.
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]



I have seen sustained writes of 30Mb/s using the following configuration:

cat /boot/loader.conf
kern.ipc.nmbclusters=32768

cat /etc/sysctl.conf
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576

Server:
/sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports

Client:
/sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
/dev/amrd0s2


The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
adaptive read ahead and write back caching.


Tom
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: ggate + gmirror write performance woes

2007-04-05 Thread Sven Willenberger 
On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
 Dmitriy Kirhlarov wrote:
  On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
  I am trying to set up a HA type system involving two identical boxes and
  have gone through the following to set up the systems:
 
  Slave server: 
  ggated -R 196608 -S 196608
  (exporting /dev/amrd1 )
  net.inet.tcp.sendspace: 65536
  net.inet.tcp.recvspace: 131072
  
  Try
  net.local.stream.recvspace=65535
  net.local.stream.sendspace=65535
  
  Also, try increase this sysctls with
  net.inet.tcp.rfc1323=1
  
  I use it on FreeBSD 5.x with:
  net.inet.tcp.sendspace=131072
  net.inet.tcp.recvspace=131072
  net.local.stream.recvspace=65535
  net.local.stream.sendspace=65535
  
  ggated -R 1048576 -S 1048576
  ggatec -R 1048576 -S 1048576
  
  WBR.
  Dmitriy
  ___
  freebsd-stable@freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-stable
  To unsubscribe, send any mail to [EMAIL PROTECTED]
 
 
 I have seen sustained writes of 30Mb/s using the following configuration:
 
 cat /boot/loader.conf
 kern.ipc.nmbclusters=32768
 
 cat /etc/sysctl.conf
 net.inet.tcp.sendspace=1048576
 net.inet.tcp.recvspace=1048576
 
 Server:
 /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
 
 Client:
 /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
 /dev/amrd0s2
 
 The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
 adaptive read ahead and write back caching.
 
 Tom

I have tried both the settings ideas suggested above but I cannot even
get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
anything higher that 131072 results in ggated bailing with the error:
# ggated -v -a 10.10.0.19
info: Reading exports file (/etc/gg.exports).
debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
info: Exporting 2 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.

setting net.inet.tcp.{send,recv}space to 131072 allows me to start
ggated with the default R and S values of 131072; anything higher
results in no buffer space errors. At 131072 ggated starts but then I
cannot even open a new connection (like ssh) to the server as the ssh
client bails with no buffer space available.

more information:
# netstat -m
514/641/1155 mbufs in use (current/cache/total)
512/284/796/32768 mbuf clusters in use (current/cache/total/max)
512/256 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
1152K/728K/1880K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/4/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
Raid using LSiMegaRaid.

The odd thing is that even after I set the send and recvspace down to
values like 65536, I continue to get the no buffer error when trying to
connect to it remotely again.

Sven

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]