Re: Truncation Data Loss

2009-11-11 Thread David Vasek

On Tue, 10 Nov 2009, Nick Guenther wrote:


[ext3 data= / FFS]
journal ~= sync (ensures consistency of both metadata and file data)
ordered ~= softdep (ensures consistency of metadata both internally
and with file data)
writeback ~= default (ensures consistency of metadata internally but
real file data may not agree, e.g. my empty file)
Additionally FFS has the async flag which turns off the internal
consistency of the metadata structures; I guess there's no equivalent
for this in ext?


Isn't it rather
default ~= async ?

For ext2, at least.

Regards,
David



Re: softraid crypto performance

2009-11-11 Thread Jan Stary
On Nov 10 23:36:42, Michael wrote:
 Hi,
 
 Am 10.11.2009 22:53, schrieb Jan Stary:
  Those 40 MB/s are limited due to the other systems read performance.
  However, softraid crypto seems (unreasonably ?) slow to me.
  
  Why do you even involve the network and some other system's
  read in evaluationg your softraid's write?
 
 Maybe, because it doesn't matter!? Softraid crypto write speed isn't
 limited by network at all, in this case.

Ah, I see. Maybe pipe the data through cat | dd | cat too.



Re: softraid crypto performance

2009-11-11 Thread Jan Stary
On Nov 10 16:21:04, Alvaro Mantilla Gimenez wrote:
 On Tue, 2009-11-10 at 21:31 +0100, Michael wrote:
  Hi,
  
  when using softraid crypto with OpenBSD 4.6-current I never get more
  than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
  GB) itself, without crypto, can do way more.
 
 Uh...that sounds wear to me. I just copy 70 Gb from a USB SATA HD to
 the local partitions under a softraid crypto device and I get 14-16 Mb/s
 all the time. Of course I don't expect more from a USB
 device

So why don't you write to the softraid device from /dev/zero,
isolating yourself from assumptions about USB or whatever?



umass0: Sometimes on ehci and sometimes on uhci

2009-11-11 Thread Rene Maroufi
Hi,

i have a IBM Thinkpad T41 running OpenBSD 4.6 stable. The Thinkpad has 2
USB Ports and supports USB 2.0 (ehci). If i plug in a umass device,
sometimes i get USB 2 speed, sometimes not. In my dmesg I can see that
uhub0 is on usb0 thats on ehci, but uhub1 - uhub3 are uhci devices. My
USB harddrive sometimes plugs in on uhub0, but sometimes not. Is there a
way to always attach a USB 2 device to an ehci port? Or maybe can I
configure the kernel to bound umass devices only to uhub0?

dmesg:

OpenBSD 4.6 (GENERIC) #3: Thu Oct 29 11:12:11 CET 2009
r...@freya.maroufi:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) M processor 1700MHz (GenuineIntel 686-class) 1.70 
GHz
cpu0: 
FPU,V86,DE,PSE,TSC,MSR,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,TM,SBF,EST,TM2
real mem  = 1072656384 (1022MB)
avail mem = 1028411392 (980MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 06/18/04, BIOS32 rev. 0 @ 0xfd750, SMBIOS 
rev. 2.33 @ 0xe0010 (61 entries)
bios0: vendor IBM version 1RETCDWW (3.06f) date 06/18/2004
bios0: IBM 2373TG5
apm0 at bios0: Power Management spec V1.2
apm0: battery life expectancy 95%
apm0: AC on, battery charge high, charging
acpi at bios0 function 0x0 not configured
pcibios0 at bios0: rev 2.1 @ 0xfd6e0/0x920
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdea0/272 (15 entries)
pcibios0: PCI Interrupt Router at 000:31:0 (Intel 82371FB ISA rev 0x00)
pcibios0: PCI bus #6 is the last bus
bios0: ROM list: 0xc/0x1 0xd/0x1000 0xd1000/0x1000 0xdc000/0x4000! 
0xe/0x1
cpu0 at mainbus0: (uniprocessor)
cpu0: Enhanced SpeedStep 1695 MHz: speeds: 1700, 1400, 1200, 1000, 800, 600 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
io address conflict 0x5800/0x8
io address conflict 0x5808/0x4
io address conflict 0x5810/0x8
io address conflict 0x580c/0x4
pchb0 at pci0 dev 0 function 0 Intel 82855PM Host rev 0x03
intelagp0 at pchb0
agp0 at intelagp0: aperture at 0xd000, size 0x1000
ppb0 at pci0 dev 1 function 0 Intel 82855PM AGP rev 0x03
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 ATI Radeon Mobility M9 rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: irq 11
drm0 at radeondrm0
uhci0 at pci0 dev 29 function 0 Intel 82801DB USB rev 0x01: irq 11
uhci1 at pci0 dev 29 function 1 Intel 82801DB USB rev 0x01: irq 11
uhci2 at pci0 dev 29 function 2 Intel 82801DB USB rev 0x01: irq 11
ehci0 at pci0 dev 29 function 7 Intel 82801DB USB rev 0x01: irq 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1
ppb1 at pci0 dev 30 function 0 Intel 82801BAM Hub-to-PCI rev 0x81
pci2 at ppb1 bus 2
mem address conflict 0xb000/0x1000
mem address conflict 0xb100/0x1000
cbb0 at pci2 dev 0 function 0 TI PCI4520 CardBus rev 0x01: irq 11
cbb1 at pci2 dev 0 function 1 TI PCI4520 CardBus rev 0x01: irq 11
em0 at pci2 dev 1 function 0 Intel PRO/1000MT (82540EP) rev 0x03: irq 11, 
address 00:0d:60:7b:15:dd
ath0 at pci2 dev 2 function 0 Atheros AR5212 (IBM MiniPCI) rev 0x01: irq 11
ath0: AR5213 5.6 phy 4.1 rf5111 1.7 rf2111 2.3, WOR1W, address 00:05:4e:4c:0e:93
cardslot0 at cbb0 slot 0 flags 0
cardbus0 at cardslot0: bus 3 device 0 cacheline 0x8, lattimer 0xb0
pcmcia0 at cardslot0
cardslot1 at cbb1 slot 1 flags 0
cardbus1 at cardslot1: bus 6 device 0 cacheline 0x8, lattimer 0xb0
pcmcia1 at cardslot1
ichpcib0 at pci0 dev 31 function 0 Intel 82801DBM LPC rev 0x01: 24-bit timer 
at 3579545Hz
pciide0 at pci0 dev 31 function 1 Intel 82801DBM IDE rev 0x01: DMA, channel 0 
configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: SAMSUNG HM160HC
wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: MATSHITA, UJDA755yDVD/CDRW, 1.72 ATAPI 5/cdrom 
removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
ichiic0 at pci0 dev 31 function 3 Intel 82801DB SMBus rev 0x01: irq 11
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 512MB DDR SDRAM non-parity PC2700CL2.5
spdmem1 at iic0 addr 0x51: 512MB DDR SDRAM non-parity PC2700CL2.5
auich0 at pci0 dev 31 function 5 Intel 82801DB AC97 rev 0x01: irq 11, ICH4 
AC97
ac97: codec id 0x41445374 (Analog Devices AD1981B)
ac97: codec features headphone, 20 bit DAC, No 3D Stereo
audio0 at auich0
Intel 82801DB Modem rev 0x01 at pci0 dev 31 function 6 not configured
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 Intel UHCI root hub rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)

Re: Truncation Data Loss

2009-11-11 Thread Janne Johansson
Nick Guenther wrote:

 So, as nicely summarized at

 http://www.h-online.com/open/news/item/Possible-data-loss-in-Ext4-740467.html
 ,
 ext4 is kind of broken. It won't honor fsync and, as a /feature/, will
 wait up to two minutes to write out data, leading to lots of files
 emptied to the great bitbucket in the sky if the machine goes down in
 that period.
 There is a very simple explanation for why things are so.
 Actual data file loss has never been what these things were coded for.
 filesystem *tree and meta-data*, ie. the structure of how things are
 knit together, is the main concern.  If you lose the filesystem tree
 structure, you've lost all your files, not just the newest ones.
 Therefore the goal is safe metadata handling.  The result is you can
 lose specific data in specific (newly written to) files, but the
 structure of the filesystem is consistant enough for fsck to not damage
 it.

 See, since it seems that BSD doesn't have this file-data consistency
 guarantee, are Linus' worries about ext4's potential data loss just
 being alarmist? It seems to me that the case described in
 https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45
 is just as likely to happen on OpenBSD--if I run KDE or GNOME and mess
 around with my settings then quickly murder the system the files will
 be resurrected empty, right?

It seems like some posters in this thread somehow misses the fact that
if you have outstanding writes and the box dies. Some of your data dies
also. New or old data, something will be missing.

From the point your app does a write(), it gets buffered in the I/O
handling, it gets buffered by the device driver for the card, it gets
buffered in the card probably, it gets buffered on the on-disk memory
cache and then it serially hits the platter one bit a a time until its
all written. If you have data in this long pipe and the power goes, you
will lose data, period.

OpenBSD has chosen to try harder to keep the metadata intact, and ext4
doesn't try at all, for the love of speed. Still, you are only moving
around the window of opportunity for fail, and sometimes making it
larger or smaller, but it is always there.

The last comment above should really only read:
If I quickly murder my system, the files might be gone. Nothing else.

If you have writes going, data loss is a reality. Sometimes more,
sometimes less, but its all games with statistics. If ext4 has a 50%
chance of killing your files and FFS on obsd has 1%, you might still get
to keep your KDE settings on either system or you may lose them all. It
shouldn't be news to anyone that Linux always went for fast-and-insecure
whereas the BSDs opted for slower-but-safer for the filesystems. Making
a fuss about how insecure the penguins are this week feels like a waste
of time to me.

If you care about your data, you have backups.

Regardless of if the probability is 1% or 50%, because for someone out
there, the percentages will be against you.



Re: Truncation Data Loss

2009-11-11 Thread Michal
Janne Johansson wrote:
 Nick Guenther wrote:
 
 So, as nicely summarized at

 http://www.h-online.com/open/news/item/Possible-data-loss-in-Ext4-740467.html
 ,
 ext4 is kind of broken. It won't honor fsync and, as a /feature/, will
 wait up to two minutes to write out data, leading to lots of files
 emptied to the great bitbucket in the sky if the machine goes down in
 that period.
 There is a very simple explanation for why things are so.
 Actual data file loss has never been what these things were coded for.
 filesystem *tree and meta-data*, ie. the structure of how things are
 knit together, is the main concern.  If you lose the filesystem tree
 structure, you've lost all your files, not just the newest ones.
 Therefore the goal is safe metadata handling.  The result is you can
 lose specific data in specific (newly written to) files, but the
 structure of the filesystem is consistant enough for fsck to not damage
 it.
 
 See, since it seems that BSD doesn't have this file-data consistency
 guarantee, are Linus' worries about ext4's potential data loss just
 being alarmist? It seems to me that the case described in
 https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45
 is just as likely to happen on OpenBSD--if I run KDE or GNOME and mess
 around with my settings then quickly murder the system the files will
 be resurrected empty, right?
 
 It seems like some posters in this thread somehow misses the fact that
 if you have outstanding writes and the box dies. Some of your data dies
 also. New or old data, something will be missing.
 
 From the point your app does a write(), it gets buffered in the I/O
 handling, it gets buffered by the device driver for the card, it gets
 buffered in the card probably, it gets buffered on the on-disk memory
 cache and then it serially hits the platter one bit a a time until its
 all written. If you have data in this long pipe and the power goes, you
 will lose data, period.
 
 OpenBSD has chosen to try harder to keep the metadata intact, and ext4
 doesn't try at all, for the love of speed. Still, you are only moving
 around the window of opportunity for fail, and sometimes making it
 larger or smaller, but it is always there.
 
 The last comment above should really only read:
 If I quickly murder my system, the files might be gone. Nothing else.
 
 If you have writes going, data loss is a reality. Sometimes more,
 sometimes less, but its all games with statistics. If ext4 has a 50%
 chance of killing your files and FFS on obsd has 1%, you might still get
 to keep your KDE settings on either system or you may lose them all. It
 shouldn't be news to anyone that Linux always went for fast-and-insecure
 whereas the BSDs opted for slower-but-safer for the filesystems. Making
 a fuss about how insecure the penguins are this week feels like a waste
 of time to me.
 
 If you care about your data, you have backups.
 
 Regardless of if the probability is 1% or 50%, because for someone out
 there, the percentages will be against you.
 

I know this is a bit off topic, but storage devices have battery's on
RAID cards for a reason. If you are worried about read/writes etc when a
system dies, there are measures you can take



Re: Truncation Data Loss

2009-11-11 Thread Russell Howe

Michal wrote, sometime around 11/11/09 11:40:


I know this is a bit off topic, but storage devices have battery's on
RAID cards for a reason. If you are worried about read/writes etc when a
system dies, there are measures you can take


Probably even more OT, but...

Although some (most?) RAID cards which have a battery option will only 
let you enable the write cache if you have a battery installed. 
Certainly the HP P400 cards we have do.


There has been endless discussion about data loss in these types of 
scenarios on the XFS mailing list - it journals metadata but not data, 
so if your application (e.g. vim) overwrites files by first truncating 
them to 0 length and then writing out the data, you'll find that the 
truncate and the resize of the file are all nicely replayed from the 
journal after the crash, but if the machine died before your data hit 
the disk, all you'll get when you read() is \0\0\0\0...


Since ext4 has started to implement similar features in similar ways to 
XFS, the ext4 folk are running into the same old problems.


--
Russell Howe, IT Manager. rh...@bmtmarinerisk.com
BMT Marine  Offshore Surveys Ltd.



Re: softraid crypto performance

2009-11-11 Thread Jacob Yocom-Piatt

Jan Stary wrote:

On Nov 10 16:21:04, Alvaro Mantilla Gimenez wrote:
  

On Tue, 2009-11-10 at 21:31 +0100, Michael wrote:


Hi,

when using softraid crypto with OpenBSD 4.6-current I never get more
than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
GB) itself, without crypto, can do way more.
  

Uh...that sounds wear to me. I just copy 70 Gb from a USB SATA HD to
the local partitions under a softraid crypto device and I get 14-16 Mb/s
all the time. Of course I don't expect more from a USB
device



So why don't you write to the softraid device from /dev/zero,
isolating yourself from assumptions about USB or whatever?

  



one sees the same sort of bottleneck using e.g. bonnie as michael 
demonstrates using his ftp transfer.


asking michael to change how he demonstrates this bottleneck is not 
productive. if you're so keen on doing it your way you should take the 
5-10 minutes to test it and post your results.




Re: softraid crypto performance

2009-11-11 Thread Marco Peereboom
Bonnie is not a realistic load, ever.  Therefore the numberis are really
not useful.  If one insists on getting an idea of what crypto can run
then do: dd if=/dev/rsd2c of=/dev/null bs=1m count=100
Where rsd2c is the raw crypto disk.

At some point I will have another look to see if I can speed it up some
more.

On Wed, Nov 11, 2009 at 06:27:00AM -0600, Jacob Yocom-Piatt wrote:
 Jan Stary wrote:
 On Nov 10 16:21:04, Alvaro Mantilla Gimenez wrote:
   
 On Tue, 2009-11-10 at 21:31 +0100, Michael wrote:
 
 Hi,

 when using softraid crypto with OpenBSD 4.6-current I never get more
 than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
 GB) itself, without crypto, can do way more.
   
 Uh...that sounds wear to me. I just copy 70 Gb from a USB SATA HD to
 the local partitions under a softraid crypto device and I get 14-16 Mb/s
 all the time. Of course I don't expect more from a USB
 device
 

 So why don't you write to the softraid device from /dev/zero,
 isolating yourself from assumptions about USB or whatever?

   


 one sees the same sort of bottleneck using e.g. bonnie as michael  
 demonstrates using his ftp transfer.

 asking michael to change how he demonstrates this bottleneck is not  
 productive. if you're so keen on doing it your way you should take the  
 5-10 minutes to test it and post your results.



Re: Truncation Data Loss

2009-11-11 Thread Marco Peereboom
On Wed, Nov 11, 2009 at 12:24:25PM +0100, Janne Johansson wrote:
 Nick Guenther wrote:
 
  So, as nicely summarized at
 
  http://www.h-online.com/open/news/item/Possible-data-loss-in-Ext4-740467.html
  ,
  ext4 is kind of broken. It won't honor fsync and, as a /feature/, will
  wait up to two minutes to write out data, leading to lots of files
  emptied to the great bitbucket in the sky if the machine goes down in
  that period.
  There is a very simple explanation for why things are so.
  Actual data file loss has never been what these things were coded for.
  filesystem *tree and meta-data*, ie. the structure of how things are
  knit together, is the main concern.  If you lose the filesystem tree
  structure, you've lost all your files, not just the newest ones.
  Therefore the goal is safe metadata handling.  The result is you can
  lose specific data in specific (newly written to) files, but the
  structure of the filesystem is consistant enough for fsck to not damage
  it.
 
  See, since it seems that BSD doesn't have this file-data consistency
  guarantee, are Linus' worries about ext4's potential data loss just
  being alarmist? It seems to me that the case described in
  https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45
  is just as likely to happen on OpenBSD--if I run KDE or GNOME and mess
  around with my settings then quickly murder the system the files will
  be resurrected empty, right?
 
 It seems like some posters in this thread somehow misses the fact that
 if you have outstanding writes and the box dies. Some of your data dies
 also. New or old data, something will be missing.

And that all SATA drives enable it or else they are glacial and that
SCSI disables it to enhance perception that one is safer.

Buy a ups! your laptop has a built in one.



Re: Truncation Data Loss

2009-11-11 Thread Marco Peereboom
EXT was and still is a joke.  I remember reading about the 2 minute
drain and I almost peed my pants.  EXT3 had the nice feature of randomly
stopping to boot after enough reboots on enough machines.  Thankfully I
no longer run any volume of this crap.

On Wed, Nov 11, 2009 at 11:55:30AM +, Russell Howe wrote:
 Michal wrote, sometime around 11/11/09 11:40:

 I know this is a bit off topic, but storage devices have battery's on
 RAID cards for a reason. If you are worried about read/writes etc when a
 system dies, there are measures you can take

 Probably even more OT, but...

 Although some (most?) RAID cards which have a battery option will only  
 let you enable the write cache if you have a battery installed.  
 Certainly the HP P400 cards we have do.

 There has been endless discussion about data loss in these types of  
 scenarios on the XFS mailing list - it journals metadata but not data,  
 so if your application (e.g. vim) overwrites files by first truncating  
 them to 0 length and then writing out the data, you'll find that the  
 truncate and the resize of the file are all nicely replayed from the  
 journal after the crash, but if the machine died before your data hit  
 the disk, all you'll get when you read() is \0\0\0\0...

 Since ext4 has started to implement similar features in similar ways to  
 XFS, the ext4 folk are running into the same old problems.

 -- 
 Russell Howe, IT Manager. rh...@bmtmarinerisk.com
 BMT Marine  Offshore Surveys Ltd.



Re: softraid crypto performance

2009-11-11 Thread Jan Stary
 when using softraid crypto with OpenBSD 4.6-current I never get more
 than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
 GB) itself, without crypto, can do way more.
   
 Uh...that sounds wear to me. I just copy 70 Gb from a USB SATA HD to
 the local partitions under a softraid crypto device and I get 14-16 Mb/s
 all the time. Of course I don't expect more from a USB
 device
 
 So why don't you write to the softraid device from /dev/zero,
 isolating yourself from assumptions about USB or whatever?

 one sees the same sort of bottleneck using e.g. bonnie as michael  
 demonstrates using his ftp transfer.

Why do you even introduce other possible bottlenecks
(such as FTP speed or bonnie or USB speed) when all you
want to measure is the r/w speed of the crypto device?



Re: 802.11 Monitor Mode in 4.6-Release

2009-11-11 Thread Tom Smith
On Tue, Nov 10, 2009 at 9:30 PM, Nick Guenther kou...@gmail.com wrote:


 A snaplen of 0 on linux really means a snaplen of 2^16-1 which is
 good enough. I'd imagine tcpdump: invalid snaplen 0 was chosen
 because technically it's true, the linux thing is just a convenience
 hack that will bite someone down the line.


I hope that you are not accusing me of using Linux. Because if you are, then
that is the ultimate insult to which I would reply how do *you* know so much
about that steaming pile of fecal matter? FreeBSD's tcpdump has a snaplen
implementation that can be set to 0 that is why I asked the question.


 What you want is to set
 your snaplen to be equal to your MTU, which is what I guess you're
 doing?


I'm sniffing packets over 802.11 and I wonder why I see some packets, but
not all.

Thomas



Re: softraid crypto performance

2009-11-11 Thread Jan Stary
On Nov 11 07:09:36, Marco Peereboom wrote:
 Bonnie is not a realistic load, ever.  Therefore the numberis are really
 not useful.  If one insists on getting an idea of what crypto can run
 then do: dd if=/dev/rsd2c of=/dev/null bs=1m count=100
 Where rsd2c is the raw crypto disk.


This is how my crypto softraid device comes to existence:

# disklabel sd0 | grep RAID
  k:406781865 83441610RAID   

# bioctl -c C -l /dev/sd0k softraid0
scsibus3 at softraid0: 1 targets
sd1 at scsibus3 targ 0 lun 0: OPENBSD, SR CRYPTO, 003 SCSI2 0/direct fixed
sd1: 198623MB, 512 bytes/sec, 406781786 sec total


This is a test of the reading speed, as adviced above:

# dd if=/dev/rsd1c of=/dev/null bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 4.696 secs (22326954 bytes/sec)

# dd if=/dev/rsd1c of=/dev/null bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 4.710 secs (22260383 bytes/sec)

# dd if=/dev/rsd1c of=/dev/null bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 4.712 secs (22249807 bytes/sec)


As opposed to reading the underlying device directly:

# dd if=/dev/rsd0k of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 1.616 secs (64876691 bytes/sec)

# dd if=/dev/rsd0k of=/dev/null bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 1.569 secs (66799470 bytes/sec)

# dd if=/dev/rsd0k of=/dev/null bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 1.569 secs (66795045 bytes/sec)


This is a test of the writing speed, in analogy with the above
(that's what's slow for the OP):

# dd of=/dev/rsd1c if=/dev/zero bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 4.728 secs (22173665 bytes/sec)

# dd of=/dev/rsd1c if=/dev/zero bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 4.699 secs (22311243 bytes/sec)

# dd of=/dev/rsd1c if=/dev/zero bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 4.691 secs (22351435 bytes/sec)


This is what the underlying device can write:

# dd of=/dev/rsd0k if=/dev/zero bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 1.630 secs (64309537 bytes/sec)

# dd of=/dev/rsd0k if=/dev/zero bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 1.630 secs (64310444 bytes/sec)

# dd of=/dev/rsd0k if=/dev/zero bs=1m count=100 
100+0 records in
100+0 records out
104857600 bytes transferred in 1.675 secs (62581676 bytes/sec)


To summarize:

readwrite
rsd0k   65M/s   64M/s
rsd1c   21M/s   21M/s


[OP:]
  when using softraid crypto with OpenBSD 4.6-current I never get more
  than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
  GB) itself, without crypto, can do way more.

How exactly do you 'write'?
What numbers do you get when you write like I do above?
How fast can the disk itself write (using the same 'measurement')?



[BLOG DECO et DESIGN] nouveautés my-deco-shop - misc@openbsd.org

2009-11-11 Thread ma-boutique-deco.com
sarablocks_imageblocks_imageblocks_imageblocks_imageblocks_imageblocks_image

Chaises | Tables | Fauteui ls | Poufs  tabourets | Bancs | Meacut
e;ridiennes  transats | Commodes et armoires | Lits | Luminai res |
Tapis  coussins | Coupes  vases | Ranger  accrocher | Deacut e;corer
| Enfants  adolescents | La dico abordable | Outdoor | Les packs | Les
designers

[  Toute la boutique  ]

blocks_imageblocks_imageblocks_imageblocks_imageblocks_imageblocks_image

) 2009 MY-DECO-SHOP.com



Re: Can't get carp to fail over all interfaces with pfsync

2009-11-11 Thread Stuart Henderson
On 2009-11-10, Daniel Ouellet dan...@presscom.net wrote:
 FW1 hostname.if files are:
 
  $ cat /etc/hostname.carp0
 
 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass
 password
  $ cat /etc/hostname.carp1
 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass
 password
  $ cat /etc/hostname.pfsync0

 Shouldn't you run different vhid ID of carp on different carp instance. 
 Here you have Carp0 and carp 1 both running with vhid 1, so how will the 
 system see them as different one?

It sees them as different, because they're on different interfaces.



Re: which raid card? [was: aac raid status]

2009-11-11 Thread William Graeber
On Wed, Nov 11, 2009 at 01:47, Theo de Raadt dera...@cvs.openbsd.org wrote:
 You say MegaRAID 8X in response to someone saying mfi?

Sorry - I was referring to ami(4) cards in response to general
discussion about LSI cards. For clarification, a quick search revealed
that some, if not all, mfi(4) cards have a 64 bit LBA.

Thanks for catching that.

-William



Re: softraid crypto performance

2009-11-11 Thread Robert

Warning, long post...

This whole performance discussion made me curious, and since I had 2
identical SATA disks available I made some tests with softraid and
crypto (dmesg at the end).

summary (MB/sec):
write   readlayout
80  80  single disk / raw
68  80  single disk / file system
30  30  single disk + softraid crypto / raw
28  30  single disk + softraid crypto / file system
21  50  single disk + VN crypto / raw
8   37  single disk + VN crypto / file system
80  78  2 disk softraid RAID1 / raw
70  76  2 disk softraid RAID1 / file system
21  32  2 disk softraid RAID1 + VN crypto / raw
20  22  2 disk softraid RAID1 + VN crypto / file system
30  22  2 disk softraid RAID1 + softraid crypto / raw
28  22  2 disk softraid RAID1 + softraid crypto / file system

One thing that I don't understand: read performance with RAID1 is
basically the same as with a single disk. I would have expected way
better results, since data can be read from both disks at the same time
(which means in theory 200% performance)?


Now for the boring ;) numbers:

*) CPU usage was measured with top -S -s 0.1
*) interrupts/sec was measured with systat -s 0.1 vmstat


1) simple disk


1.1) raw speed
# dd if=/dev/zero of=/dev/rwd1c bs=1m count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 6.470 secs (81022423 bytes/sec)
# dd if=/dev/rwd1c of=/dev/null bs=1m count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 6.499 secs (80667486 bytes/sec)

# dd if=/dev/zero of=/dev/rwd2c bs=1m count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 6.360 secs (82429854 bytes/sec)
# dd if=/dev/rwd2c of=/dev/null bs=1m count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 6.268 secs (83632786 bytes/sec)
(as expected, since the disks are identical...)

*) CPU load nearly none, interrupts ca. 1.000/sec


1.2) simple file system
# fdisk -iy wd1
# printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E wd1
# newfs wd1a
# mount -o softdep /dev/wd1a /mnt/test

# dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000


1000+0 records in
1000+0 records out
1048576000 bytes transferred in 15.412 secs (68036212 bytes/sec)
*) interrupts went up to 5.000/sec (jumping around) and CPU was aroung
10% for dd

# dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 13.079 secs (80168224 bytes/sec)
*) interrupts at 1.500/sec and 5% CPU for dd


2) softraid crypto on 1 disk

# fdisk -iy wd1
# printf a\n\n\n\nRAID\nw\nq\n\n |disklabel -E wd1
# bioctl -c C -l /dev/wd1a softraid0
scsibus1 at softraid0: 1 targets
sd0 at scsibus1 targ 0 lun 0: OPENBSD, SR CRYPTO, 003 SCSI2 0/direct fixed
sd0: 476937MB, 512 bytes/sec, 976767923 sec total
# fdisk -iy sd0

2.1) raw speed
# dd if=/dev/zero of=/dev/rsd0c bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 3.485 secs (30086175 bytes/sec)
# dd if=/dev/rsd0c of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 3.452 secs (30374596 bytes/sec)

*) crypto just decreased the speed to 37%...
*) interrupts were at ca. 500/second
*) CPU load of dd kept climbing (60%+); it seems that the CPU/encryption
speed is the limiting factor here (?)


2.2) writing a file in crypto
# printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E sd0
# newfs sd0a
# mount -o softdep /dev/sd0a /mnt/test
# dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 37.349 secs (28074519 bytes/sec)

*) little less than raw speed
*) interesting note: the interrupts (for pciide) kept jumping between
500-1700 (??); CPU load for dd was aroung 20%


2.3) reading from the same file
# sync
# dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 34.888 secs (30055309 bytes/sec)

*) CPU load was around 3%; decrypting seems to be WAY less expensive
*) interrupts were around 500


3) SVN crypto on 1 disk

# fdisk -iy wd1
# printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E wd1
# newfs wd1a
# mount -o softdep /dev/wd1a /mnt/test
# dd if=/dev/arandom of=/mnt/test/crypt.file bs=1m count=2000
# vnconfig -ck svnd0 /mnt/test/crypt.file
# fdisk -iy svnd0


3.1) raw speed
# dd if=/dev/zero of=/dev/rsvnd0c bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 49.077 secs (21365694 bytes/sec)
# dd if=/dev/rsvnd0c of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 21.019 secs (49885388 bytes/sec)
*) This used up all the CPU and interrupts jumped between 0-5.000; the
system was unusable meanwhile.


3.2) file system speed
# printf a\n\n\n\n4.2BSD\nw\nq\n\n | disklabel -E svnd0
# newfs svnd0a
# mount -o softdep /dev/svnd0a /mnt/crypto

# dd if=/dev/zero 

ftpd, OpenBSD 4.5 memory behavior

2009-11-11 Thread MK

Hello all,

recently I've noticed on my OpenBSD 4.5 Stable box strange memory behavior 
while downloading files from ftpd daemon.
It seems ftpd is somehow allocates more and more memory. Memory is not freed 
until something else needs it.
At least it is always freed after daily script runs. I've noticed this 
problem while few clients were downloading files from the box and I don't 
recall I saw something similar on OpenBSD 4.4.


top output shows something like this: Memory: Real: 53M/836M  (normal state 
should be about 53M/170M)
I was also trying to reproduce the problem by downloading files from ftpd 
and saving remaining free memory every 10 minutes.


Date | Free Memory (KB)

| 2009-11-11 18:30:01 | 807936 |
| 2009-11-11 18:40:01 | 771072 |
| 2009-11-11 18:50:02 | 561152 |
| 2009-11-11 19:00:02 | 329728 |
| 2009-11-11 19:10:02 | 214016 |
| 2009-11-11 19:20:02 | 211968 |

Is it a normal situation?

Thanks
MK



Re: locking a softraid crypto vol

2009-11-11 Thread Nick Guenther
On Tue, Nov 10, 2009 at 10:52 PM, Marco Peereboom sl...@peereboom.us wrote:

 where sd3 is the softraid crypto volume.

 On Tue, Nov 10, 2009 at 07:38:00PM -0600, c l wrote:
 Is it possible to lock a softraid crypto volume without rebooting?

 It seems bioctl -d is what I want but I'm not sure.

 What I would like to do is unlock the volume...

 bioctl -c C -l /dev/sd0d softraid0

 Mount it, then copy some data to it, then unmount it and lock again.

 bioctl -d softraid0


 Cluestick anyone?


 Not sure what locking means but -d delete it.

 The man page has an example of -d but it comes down to
 bioctl -d sd3

If Marco doesn't know what 'locking' means I would say he just wants
to make sure that the volume gets encrypted. To the OP: the volume
is always encrypted, decrypting just means that the kernel knows the
key, so as soon as you unmount it it is locked (though you have to
make sure your key is protected, of course).

-Nick



Re: softraid crypto performance

2009-11-11 Thread Aaron Poffenberger
On Nov 10, 2009, at 2:31 PM, Michael wrote:

 Hi,

 when using softraid crypto with OpenBSD 4.6-current I never get more
 than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
 GB) itself, without crypto, can do way more.

 What I see during transfer in top/systat is a high interrupt load,
 however the interrupt load when writing to a not encrypted partition is
 even higher and the speed faster, so that doesn't seem to be the issue.

 The crypto system process is shown as mostly bored in top. So, I am
 wondering why softraid crypto never exceeds ~10-11 MB/s for me?

 Ideas are welcome. It would also be nice if someone could check the
 speed with their own softraid crypto setups.

 To test the speed I downloaded a huge file (~3,5 GB) using FTP over gbit
 ethernet.

 Write performance measured:

 to mfs partition: ~40 MB/s
 to ffs partition: ~40 MB/s (doesn't matter if using softdep or not)
 to softraid crypto partition: ~11 MB/s (+- 1 MB/s)

 Those 40 MB/s are limited due to the other systems read performance.
 However, softraid crypto seems (unreasonably ?) slow to me.


 Michael



long reply follows:

I've been running softraid(4) for while and recently added the crypto
discipline to my test configuration and find the performance acceptable
but wanted numbers so I ran some tests and found that while my
throughput is better than yours, it isn't huge multiples of
better.

And yet I don't think my experience demonstrate huge performance
issues with softraid(4)'s crypto discipline. I say that because in
addition to testing crypto performance I also tested my server's
encryption performance to see whether my max write speeds were
siginificantly worse than the speed the server can encrypt data.

In summary, writing to a file using softdep or asynchronous was about
20 MB/s or approximately twice your reported speed.

The server-encryption tests results ranged from 25 MB/s to 45 MB/s
depending on whether I used /dev/zero, /dev/arandom or a raw device
as the input. Writing to a file using softdep or asynchronous mount
options gave results ranging from a 57% to 68% of theoretical. And note
the crypto device is sitting on a softraid(4) mirror device on a system
running 4.5.

So while less than system max leaving some room for improvement, based on
the system encryption peformance test, and assuming I constructed
encryption peformance test correctly, it looks like my server's softraid
crypto performance is pretty good, especially when taking into account
that softraid(4) and the crypto discpline are very new to OpenBSD.

One curious result is that my read speeds are lower than write. I was
expecting reads to be a lot faster since the crypto devices are
sitting on a softraid(4) mirror.

Assuming my system-encryption performance test is well-constructed,
perhaps you could run it on your system and see whether you're getting
similar results.

I'd appreciate feedback anyone has about the test setup, especially the
system-encryption test. I used openssl (see below) with the same data
inputs as the crypto devices.

Summary and details of the tests and results below.

--Aaron

--

System Summary:
OpenBSD 4.5
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ 2.01 GHz
1 GB RAM

Tests Summary:
# dd if=/dev/zero bs=1m count=100 | openssl enc -aes-128-cbc \
-salt [1]  /dev/null

# dd if=/dev/[2] of=[3] bs=1m count=100

[1] softraid crypto key
[2] One of zero | arandom | rwd0a
[3] One of /mnt/backup3/speed_test | /dev/rsd7a

Results Summary:
1) Openssl sytem-encryption test:
Source   MB/s
/dev/zero   44.47
/dev/arandom25.05
/dev/rwd0a  34.18

Write Tests:
2) Asynchronous disk write:
Source  MB/s% of System Test
/dev/zero   21.43   48
/dev/arandom14.84   59
/dev/rwd0a  19.44   57

3) Softdep disk write:
Source  MB/s% of System Test
/dev/zero   20.85   47
/dev/arandom14.74   59
/dev/rwd0a  23.257

4) Synchronous disk write:
Source  MB/s% of System Test
/dev/zero   5.8 13
/dev/arandom5.0720
/dev/rwd0a  5.3516

5) Raw device write:
Source  MB/s% of System Test
/dev/zero   19.59   44
/dev/arandom14.32   57
/dev/rwd0a  16.55   48

6) Read Tests:
Source  MB/s
Mounted Read15.12
Raw Read15.72

--

Tests and Configuration:
I tried several configuration for the hardware tests: write to the raw
device and write to a file on a mounted drive using softdep, asynchronous,
synchronous and raw. The write tests used data from /dev/zero,
/dev/arandom the IDE system boot disk. The write tests are probably
overkill but it didn't take much longer to run them and I was curious.

Read test were performed from a file on the mounted drive and raw device.

The drives are internal SATA II with 32MB of cache spinning at 7200
RPM. The chipset on the motherboard supports the full 3Gb/s speed of
SATA II. The server is low to moderately taxed.

Re: softraid crypto performance

2009-11-11 Thread Marco Peereboom
You defeated read ahead with your RAID test and found out that kernel
crypto is slow.  Oh and you tested softdep pretty well too because that
is what you were trying to measure, right?

Lies, damn lies and benchmarks...

On Wed, Nov 11, 2009 at 08:09:40PM +0100, Robert wrote:
 Warning, long post...

 This whole performance discussion made me curious, and since I had 2
 identical SATA disks available I made some tests with softraid and
 crypto (dmesg at the end).

 summary (MB/sec):
 write readlayout
 8080  single disk / raw
 6880  single disk / file system
 3030  single disk + softraid crypto / raw
 2830  single disk + softraid crypto / file system
 2150  single disk + VN crypto / raw
 8 37  single disk + VN crypto / file system
 8078  2 disk softraid RAID1 / raw
 7076  2 disk softraid RAID1 / file system
 2132  2 disk softraid RAID1 + VN crypto / raw
 2022  2 disk softraid RAID1 + VN crypto / file system
 3022  2 disk softraid RAID1 + softraid crypto / raw
 2822  2 disk softraid RAID1 + softraid crypto / file system

 One thing that I don't understand: read performance with RAID1 is
 basically the same as with a single disk. I would have expected way
 better results, since data can be read from both disks at the same time
 (which means in theory 200% performance)?


 Now for the boring ;) numbers:

 *) CPU usage was measured with top -S -s 0.1
 *) interrupts/sec was measured with systat -s 0.1 vmstat


 1) simple disk


 1.1) raw speed
 # dd if=/dev/zero of=/dev/rwd1c bs=1m count=500
 500+0 records in
 500+0 records out
 524288000 bytes transferred in 6.470 secs (81022423 bytes/sec)
 # dd if=/dev/rwd1c of=/dev/null bs=1m count=500
 500+0 records in
 500+0 records out
 524288000 bytes transferred in 6.499 secs (80667486 bytes/sec)

 # dd if=/dev/zero of=/dev/rwd2c bs=1m count=500
 500+0 records in
 500+0 records out
 524288000 bytes transferred in 6.360 secs (82429854 bytes/sec)
 # dd if=/dev/rwd2c of=/dev/null bs=1m count=500
 500+0 records in
 500+0 records out
 524288000 bytes transferred in 6.268 secs (83632786 bytes/sec)
 (as expected, since the disks are identical...)

 *) CPU load nearly none, interrupts ca. 1.000/sec


 1.2) simple file system
 # fdisk -iy wd1
 # printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E wd1
 # newfs wd1a
 # mount -o softdep /dev/wd1a /mnt/test

 # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000


 1000+0 records in
 1000+0 records out
 1048576000 bytes transferred in 15.412 secs (68036212 bytes/sec)
 *) interrupts went up to 5.000/sec (jumping around) and CPU was aroung
 10% for dd

 # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
 1000+0 records in
 1000+0 records out
 1048576000 bytes transferred in 13.079 secs (80168224 bytes/sec)
 *) interrupts at 1.500/sec and 5% CPU for dd


 2) softraid crypto on 1 disk

 # fdisk -iy wd1
 # printf a\n\n\n\nRAID\nw\nq\n\n |disklabel -E wd1
 # bioctl -c C -l /dev/wd1a softraid0
 scsibus1 at softraid0: 1 targets
 sd0 at scsibus1 targ 0 lun 0: OPENBSD, SR CRYPTO, 003 SCSI2 0/direct fixed
 sd0: 476937MB, 512 bytes/sec, 976767923 sec total
 # fdisk -iy sd0

 2.1) raw speed
 # dd if=/dev/zero of=/dev/rsd0c bs=1m count=100
 100+0 records in
 100+0 records out
 104857600 bytes transferred in 3.485 secs (30086175 bytes/sec)
 # dd if=/dev/rsd0c of=/dev/null bs=1m count=100
 100+0 records in
 100+0 records out
 104857600 bytes transferred in 3.452 secs (30374596 bytes/sec)

 *) crypto just decreased the speed to 37%...
 *) interrupts were at ca. 500/second
 *) CPU load of dd kept climbing (60%+); it seems that the CPU/encryption
 speed is the limiting factor here (?)


 2.2) writing a file in crypto
 # printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E sd0
 # newfs sd0a
 # mount -o softdep /dev/sd0a /mnt/test
 # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
 1000+0 records in
 1000+0 records out
 1048576000 bytes transferred in 37.349 secs (28074519 bytes/sec)

 *) little less than raw speed
 *) interesting note: the interrupts (for pciide) kept jumping between
 500-1700 (??); CPU load for dd was aroung 20%


 2.3) reading from the same file
 # sync
 # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
 1000+0 records in
 1000+0 records out
 1048576000 bytes transferred in 34.888 secs (30055309 bytes/sec)

 *) CPU load was around 3%; decrypting seems to be WAY less expensive
 *) interrupts were around 500


 3) SVN crypto on 1 disk

 # fdisk -iy wd1
 # printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E wd1
 # newfs wd1a
 # mount -o softdep /dev/wd1a /mnt/test
 # dd if=/dev/arandom of=/mnt/test/crypt.file bs=1m count=2000
 # vnconfig -ck svnd0 /mnt/test/crypt.file
 # fdisk -iy svnd0


 3.1) raw speed
 # dd if=/dev/zero of=/dev/rsvnd0c bs=1m count=1000
 1000+0 records in
 1000+0 records out
 1048576000 bytes transferred in 49.077 secs (21365694 bytes/sec)
 # dd if=/dev/rsvnd0c of=/dev/null bs=1m count=1000
 1000+0 records in
 

Re: locking a softraid crypto vol

2009-11-11 Thread Aaron Poffenberger
On Nov 11, 2009, at 2:27 PM, Nick Guenther wrote:

 On Tue, Nov 10, 2009 at 10:52 PM, Marco Peereboom sl...@peereboom.us
wrote:

 where sd3 is the softraid crypto volume.

 On Tue, Nov 10, 2009 at 07:38:00PM -0600, c l wrote:
 Is it possible to lock a softraid crypto volume without rebooting?

 It seems bioctl -d is what I want but I'm not sure.

 What I would like to do is unlock the volume...

 bioctl -c C -l /dev/sd0d softraid0

 Mount it, then copy some data to it, then unmount it and lock again.

 bioctl -d softraid0


 Cluestick anyone?


 Not sure what locking means but -d delete it.

 The man page has an example of -d but it comes down to
 bioctl -d sd3

 If Marco doesn't know what 'locking' means I would say he just wants
 to make sure that the volume gets encrypted. To the OP: the volume
 is always encrypted, decrypting just means that the kernel knows the
 key, so as soon as you unmount it it is locked (though you have to
 make sure your key is protected, of course).

 -Nick


umount-ing a softraid(4) crypto device does not flush the key from bioctl. I
can umount and mount a crypto device as often as I want. bioctl -d and halt
are the only ways to lock the device.

--Aaron



Re: softraid crypto performance

2009-11-11 Thread Robert
*) Softdep was put there on purpose, because this is most probably how 
the mount will be done by most of the users.


*) My intention was simply to find out how those combinations will work 
on my system in the way that I would (from my knowledge so far) 
configure them - any advice on how to improve this (my) setup is very 
welcome, I'm actually happy to learn.


*) Lies, benchmarks, statistics - sure, no arguing here; that's why I 
made my own ;)


I've often seen questions from people who try to improve their disk or 
network speed (me too of course). So I was just curious and tried out 
some things, maybe someone else might find this useful. Or add some 
hints on how to improve.


Again, any advice is very welcome.

regards,
Robert

Marco Peereboom wrote:

You defeated read ahead with your RAID test and found out that kernel
crypto is slow.  Oh and you tested softdep pretty well too because that
is what you were trying to measure, right?

Lies, damn lies and benchmarks...




E17 wiki page for OpenBSD

2009-11-11 Thread sda
hello,

http://trac.enlightenment.org/e/wiki/OpenBSD

welcome to correct, improve, advise, etc...

regards,
sda

[demime 1.01d removed an attachment of type application/pgp-signature]



Re: softraid crypto performance

2009-11-11 Thread Kenneth R Westerback
On Wed, Nov 11, 2009 at 02:30:25PM -0600, Marco Peereboom wrote:
 You defeated read ahead with your RAID test and found out that kernel
 crypto is slow.  Oh and you tested softdep pretty well too because that
 is what you were trying to measure, right?
 
 Lies, damn lies and benchmarks...

Lies, damn lies, statistics, and **%^* benchmarks I think is the correct
ordering.

 Ken

 
 On Wed, Nov 11, 2009 at 08:09:40PM +0100, Robert wrote:
  Warning, long post...
 
  This whole performance discussion made me curious, and since I had 2
  identical SATA disks available I made some tests with softraid and
  crypto (dmesg at the end).
 
  summary (MB/sec):
  write   readlayout
  80  80  single disk / raw
  68  80  single disk / file system
  30  30  single disk + softraid crypto / raw
  28  30  single disk + softraid crypto / file system
  21  50  single disk + VN crypto / raw
  8   37  single disk + VN crypto / file system
  80  78  2 disk softraid RAID1 / raw
  70  76  2 disk softraid RAID1 / file system
  21  32  2 disk softraid RAID1 + VN crypto / raw
  20  22  2 disk softraid RAID1 + VN crypto / file system
  30  22  2 disk softraid RAID1 + softraid crypto / raw
  28  22  2 disk softraid RAID1 + softraid crypto / file system
 
  One thing that I don't understand: read performance with RAID1 is
  basically the same as with a single disk. I would have expected way
  better results, since data can be read from both disks at the same time
  (which means in theory 200% performance)?
 
 
  Now for the boring ;) numbers:
 
  *) CPU usage was measured with top -S -s 0.1
  *) interrupts/sec was measured with systat -s 0.1 vmstat
 
 
  1) simple disk
 
 
  1.1) raw speed
  # dd if=/dev/zero of=/dev/rwd1c bs=1m count=500
  500+0 records in
  500+0 records out
  524288000 bytes transferred in 6.470 secs (81022423 bytes/sec)
  # dd if=/dev/rwd1c of=/dev/null bs=1m count=500
  500+0 records in
  500+0 records out
  524288000 bytes transferred in 6.499 secs (80667486 bytes/sec)
 
  # dd if=/dev/zero of=/dev/rwd2c bs=1m count=500
  500+0 records in
  500+0 records out
  524288000 bytes transferred in 6.360 secs (82429854 bytes/sec)
  # dd if=/dev/rwd2c of=/dev/null bs=1m count=500
  500+0 records in
  500+0 records out
  524288000 bytes transferred in 6.268 secs (83632786 bytes/sec)
  (as expected, since the disks are identical...)
 
  *) CPU load nearly none, interrupts ca. 1.000/sec
 
 
  1.2) simple file system
  # fdisk -iy wd1
  # printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E wd1
  # newfs wd1a
  # mount -o softdep /dev/wd1a /mnt/test
 
  # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
 
 
  1000+0 records in
  1000+0 records out
  1048576000 bytes transferred in 15.412 secs (68036212 bytes/sec)
  *) interrupts went up to 5.000/sec (jumping around) and CPU was aroung
  10% for dd
 
  # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
  1000+0 records in
  1000+0 records out
  1048576000 bytes transferred in 13.079 secs (80168224 bytes/sec)
  *) interrupts at 1.500/sec and 5% CPU for dd
 
 
  2) softraid crypto on 1 disk
 
  # fdisk -iy wd1
  # printf a\n\n\n\nRAID\nw\nq\n\n |disklabel -E wd1
  # bioctl -c C -l /dev/wd1a softraid0
  scsibus1 at softraid0: 1 targets
  sd0 at scsibus1 targ 0 lun 0: OPENBSD, SR CRYPTO, 003 SCSI2 0/direct fixed
  sd0: 476937MB, 512 bytes/sec, 976767923 sec total
  # fdisk -iy sd0
 
  2.1) raw speed
  # dd if=/dev/zero of=/dev/rsd0c bs=1m count=100
  100+0 records in
  100+0 records out
  104857600 bytes transferred in 3.485 secs (30086175 bytes/sec)
  # dd if=/dev/rsd0c of=/dev/null bs=1m count=100
  100+0 records in
  100+0 records out
  104857600 bytes transferred in 3.452 secs (30374596 bytes/sec)
 
  *) crypto just decreased the speed to 37%...
  *) interrupts were at ca. 500/second
  *) CPU load of dd kept climbing (60%+); it seems that the CPU/encryption
  speed is the limiting factor here (?)
 
 
  2.2) writing a file in crypto
  # printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E sd0
  # newfs sd0a
  # mount -o softdep /dev/sd0a /mnt/test
  # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
  1000+0 records in
  1000+0 records out
  1048576000 bytes transferred in 37.349 secs (28074519 bytes/sec)
 
  *) little less than raw speed
  *) interesting note: the interrupts (for pciide) kept jumping between
  500-1700 (??); CPU load for dd was aroung 20%
 
 
  2.3) reading from the same file
  # sync
  # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
  1000+0 records in
  1000+0 records out
  1048576000 bytes transferred in 34.888 secs (30055309 bytes/sec)
 
  *) CPU load was around 3%; decrypting seems to be WAY less expensive
  *) interrupts were around 500
 
 
  3) SVN crypto on 1 disk
 
  # fdisk -iy wd1
  # printf a\n\n\n\n4.2BSD\nw\nq\n\n |disklabel -E wd1
  # newfs wd1a
  # mount -o softdep /dev/wd1a /mnt/test
  # dd if=/dev/arandom of=/mnt/test/crypt.file bs=1m count=2000
  # vnconfig -ck 

Sun Fire x2270 experience

2009-11-11 Thread Predrag Punosevac
Dear All,

I was wondering if I could get some input on Sun Fire x2270. Our 
department is getting ready to purchase one for running statistical
software R. Application will be provided via local network to our 
faculty and students. 

Thank You,
Predrag Punosevac

P.S. I really gave a hard push to run OpenBSD but it is probably going
to run SuSE enterprise edition due to the University requirements.
Could anybody give a quick update on bigmem status. One of the reasons
I could not push harder is that will come with 16Gb of RAM.



OpenBSD platform of choice?

2009-11-11 Thread Daniel Gracia Garallar

Hi there!

Now that I have to change my little server farm and I'm able to choose a 
new platform, I would like to choose wisely.


It's a matter of fact that Intel x86 is bogus-prone, and after 
experimenting a lot with OpenBSD and listening about the different archs 
since several years ago, I tend to think that most of the delevopers 
have a taste for Sparc derived machines as being more... predictable. 
But of course, no machine is bug free.


So thinking about security and stability, what would be your OpenBSD 
platform of choice?


Keep in mind that in this question price is not a factor. I'm just 
curious about preferences based on CPU features and their implementation 
on OpenBSD.


Regards!

Dani



CURESE DE SUS ENFERMEDADES, YA NO REGALE SU DINERO¡¡¡

2009-11-11 Thread Bioresonanse Integral S.

ESTA CANSADO DE BUSCAR SOLUCISN A SUS PROBLEMAS DE SALUD?
AHORA EN MEXICO USTED PUEDE REALIZARSE ANALISIS MEDICO Y TRATAMIENTO CON LO
ULTIMO DE LA TECNOLOGIA A NIVEL MUNDIAL.
ALIVIESE DE MUCHAS ENFERMEDADES QUE HASTA EL DIA DE HOY PARECIAN INCURABLES
POR NO SER DETECTADAS CON LOS ESTUDIOS CLINICOS EN LABORATORIOS
CONVENCIONALES O PEOR AUN, POR UN DIAGNOSTICO CLINICO EQUIVOCADO.
EN BIORESONANSE INTEGRAL SISTEM, LE OFRECEMOS DIAGNOSTICO Y TRAQTAMIENTO
INTEGRAL CON UN ALTO PORCENTAJE DE EFECTIVIDAD, ADEMAS CONTAMOS CON LO
ULTIMO EN TECNOLOGIA EN MEXICO.
   ENFERMEDADES:

  DIABETES, ARTRITIS, HIPERTENSION,ASMA,MIGRAQA,HEPATITIS, MIOMAS, CANCER,
SIDA (VIH)
  NERVIOS, GASTRITIS, COLITIS, RIQON, TUBERCULOSIS, CANSANCIO, ESCLEROSIS
MULTIPLE, STREES,
  ALERGIAS, HEMORROIDES, PROSTATA.

SI PARA USTED NO ES UTIL NUESTRA INFORMACION, POR FAVOR COMPARTALA CON ALGUIEN
QUE LA NECESITE GRACIAS.

AV. PASEO DE LOS JARDINES 222 LOCAL 61  PLAZA TAXQUEQA
COL. PASEOS DE TAXQUEQA  MEXICO, D.F.

  TELEFONOS:  56 70 02 14   Y  55 81 82 06
  DESDE TODA LA REP. MEXICANA01 800 83 22 333

O A NUESTRO CORREO  conta...@bioresonanse.com.mx



Re: 802.11 Monitor Mode in 4.6-Release

2009-11-11 Thread Tom Smith
On Wed, Nov 11, 2009 at 3:35 PM, Nick Guenther kou...@gmail.com wrote:


 Well if you were using monitor mode on some other card I would say
 it's because as a 'security measure' the firmware is blocking it, but
 it's a Ralink and they're the open ones so hmm. Sorry, I think I'm
 spent.


I figured it out. The commands I posted *do* capture the 802.11 traffic
that I'm looking for. I just did not realize that until I loaded the output
from tcpdump into a better analyzer. While tcpdump does a fine job capturing
packets, it's not the best analysis tool.

Thanks for the responses,

Thomas



Re: Truncation Data Loss

2009-11-11 Thread Nick Guenther
On Wed, Nov 11, 2009 at 3:35 AM, David Vasek va...@fido.cz wrote:
 On Tue, 10 Nov 2009, Nick Guenther wrote:

 [ext3 data= / FFS]
 journal ~= sync (ensures consistency of both metadata and file data)
 ordered ~= softdep (ensures consistency of metadata both internally
 and with file data)
 writeback ~= default (ensures consistency of metadata internally but
 real file data may not agree, e.g. my empty file)
 Additionally FFS has the async flag which turns off the internal
 consistency of the metadata structures; I guess there's no equivalent
 for this in ext?

 Isn't it rather
 default ~= async ?

 For ext2, at least.


Well I'm not sure because no one seems to really know. Linux's
mount(1) has this to say:
  writeback
 Data ordering is not preserved - data may be written
into
 the  main file system after its metadata has been
commitb
 ted to the journal.  This is rumoured to be the
highest-
 throughput  option.   It  guarantees internal file
system
 integrity, however it can allow old  data  to  appear
in
 files after a crash and journal recovery.
which seems to imply that metadata is written synchronously (because
it only talks about data appearing in files, not about the whole
filesystem getting trashed).

And BSD's mount(1) says:
 async   Metadata I/O to the file system should be done asyn-
 chronously.  By default, only regular data is read/writ-
 ten asynchronously.

 This is a dangerous flag to set since it does not
guaran-
 tee to keep a consistent file system structure on the
 disk.  You should not use this flag unless you are pre-
 pared to recreate the file system should your system
 crash.  The most common use of this flag is to speed up
 restore(8) where it can give a factor of two speed in-
 crease.



Re: Truncation Data Loss

2009-11-11 Thread Nick Guenther
On Wed, Nov 11, 2009 at 1:16 PM, Ted Unangst ted.unan...@gmail.com wrote:
 On Tue, Nov 10, 2009 at 10:50 PM, Nick Guenther kou...@gmail.com wrote:
 See, since it seems that BSD doesn't have this file-data consistency
 guarantee, are Linus' worries about ext4's potential data loss just
 being alarmist? It seems to me that the case described in

https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45
 is just as likely to happen on OpenBSD--if I run KDE or GNOME and mess
 around with my settings then quickly murder the system the files will
 be resurrected empty, right?

 Yes, if you cut power before things are written to disk, they will not
 be written to disk.  Snark aside, it really is that simple.  Different
 filesystems have different definitions of what written to disk
 means, or more accurately, *when*, but in all cases, if you cared you
 used fsync or tried a little harder to not crash.

 What is the reason softdep isn't on by default?

 It changes the expected behavior.  FFS without softdep is a lot
 closer to the semantics people and most applications expect.



Okay, one last question: one of the original softdep papers
(http://www.usenix.org/publications/library/proceedings/bsdcon02/mckusick.htm
l)
is all about how softdeps can avoid fsck, but I just set softdep on
all my filesystems, rebooted (to start fresh), wrote some files, wrote
some more files, edited the first files, and jacked the power plug
right after it said wrote. When the system came up fsck ran, what
gives? Does OpenBSD only implement softdep for the write speedups?

I'm just really confused about what softdep -is- I guess. What
semantics get changed? Do all the BSDs use the same softdep code? Did
they pick and choose ideas from the original softdep papers?

Thanks for letting me pick your brain, Ted,
-Nick



Re: Truncation Data Loss

2009-11-11 Thread David Vasek

On Wed, 11 Nov 2009, Nick Guenther wrote:


On Wed, Nov 11, 2009 at 3:35 AM, David Vasek va...@fido.cz wrote:

On Tue, 10 Nov 2009, Nick Guenther wrote:


[ext3 data= / FFS]
journal ~= sync (ensures consistency of both metadata and file data)
ordered ~= softdep (ensures consistency of metadata both internally
and with file data)
writeback ~= default (ensures consistency of metadata internally but
real file data may not agree, e.g. my empty file)
Additionally FFS has the async flag which turns off the internal
consistency of the metadata structures; I guess there's no equivalent
for this in ext?


Isn't it rather
default ~= async ?

For ext2, at least.



Well I'm not sure because no one seems to really know. Linux's
mount(1) has this to say:
 writeback
Data ordering is not preserved - data may be written
into
the  main file system after its metadata has been
commitb
ted to the journal.  This is rumoured to be the
highest-
throughput  option.   It  guarantees internal file
system
integrity, however it can allow old  data  to  appear
in
files after a crash and journal recovery.
which seems to imply that metadata is written synchronously (because
it only talks about data appearing in files, not about the whole
filesystem getting trashed).


The paragraph from Linux's mount(1?) you cited is related to ext3/ext4, 
which, as tedu@ has already written, uses journaling. Ext2, in contrast, 
is mounted async by default.


If still unsure, look at the following document about ext2 from Linux 
(kernel) documentation, section Metadata (line 281 and below).


http://www.mjmwired.net/kernel/Documentation/filesystems/ext2.txt

Regards,
David



Re: 802.11 Monitor Mode in 4.6-Release

2009-11-11 Thread Tom Smith
 After this, no more noise from me. Perhaps this will help some other old
 fool some day:


1. Get an 802.11 wireless adapter that supports monitor mode. If you don't
know what adapter to use, from a -current OpenBSD release run 'apropos
wireless' and then man the chipsets.

2. To capture 802.11 packets, you *should not* have an IP address or be
associated with an Access Point. ACLs and MAC address restictions have no
impact on your ability to capture packets.

3. Run this command to get the channel and the nwid of the Access Point
(replace if0 with your 802.11 device name):

ifconfig if0 scan

4. Now, configure the adapter like so:

ifconfig if0 chan 6
ifconfig if0 nwid TheAP
ifconfig if0 mediaopt monitor
ifconfig if0 up

5. In a seperate terminal, run tcpdump to capture what the adapter sees:

tcpdump - -s 1514 -i rum0 -y IEEE802_11 -w wireless.capture

6. After a few hours (or whatever your time window is), load the tcpdump
output file into a packet analyzer for analysis.



Re: Truncation Data Loss

2009-11-11 Thread Theo de Raadt
 Okay, one last question: one of the original softdep papers
 (http://www.usenix.org/publications/library/proceedings/bsdcon02/mckusick.htm
 l)
 is all about how softdeps can avoid fsck, but I just set softdep on
 all my filesystems, rebooted (to start fresh), wrote some files, wrote
 some more files, edited the first files, and jacked the power plug
 right after it said wrote. When the system came up fsck ran, what
 gives? Does OpenBSD only implement softdep for the write speedups?
 
 I'm just really confused about what softdep -is- I guess. What
 semantics get changed? Do all the BSDs use the same softdep code? Did
 they pick and choose ideas from the original softdep papers?

You are misreading the fsck manual page.  Let me take you through it:

 fsck - file system consistency check and interactive repair

Note what it says carefully. It says file system.  It does not
say file consistency check and interactive repair.

If you want your files to not lose a byte, use the sync option.
Come on.  Just do it.  Give it a try.  Then you will understand.



Re: Truncation Data Loss

2009-11-11 Thread Ted Unangst
On Wed, Nov 11, 2009 at 7:08 PM, Nick Guenther kou...@gmail.com wrote:
 Okay, one last question: one of the original softdep papers
 (http://www.usenix.org/publications/library/proceedings/bsdcon02/mckusick.html)
 is all about how softdeps can avoid fsck, but I just set softdep on
 all my filesystems, rebooted (to start fresh), wrote some files, wrote
 some more files, edited the first files, and jacked the power plug
 right after it said wrote. When the system came up fsck ran, what
 gives? Does OpenBSD only implement softdep for the write speedups?

All the softdep code comes from FreeBSD, but at various points, not to
mention that things that work in papers never work the same for people
who aren't writing papers.  Nobody added the background fsck code to
OpenBSD because it didn't exist when we imported softdep and nobody
has cared enough to do so since.



Re: Source for LENOVO parts?

2009-11-11 Thread Frank Bax

Thanks!!  Excellent service.  fan replaced and all is good now.

Home renovations mean that my laptop was subjected to some concrete dust 
a few months ago; so I'm thinking my fan failure might have had 
something to do with that.




Eric Elena wrote:

The same thing happened to me 2 months ago. Maybe this fan is not very
reliable.
Anyway, I bought mine online on the IBM France website. For Canada, it
seems you have to contact the Toronto Parts Order Centre 
https://www-304.ibm.com/shop/americas/content/home/store_IBMPublicCanada/en_CA/parts.html



Le samedi 31 octobre 2009 C  08:31 -0400, Frank Bax a C)crit :

I have a Lenovo T60p (8744-J2U L3-CM199-07/07).
I have been running OpenBSD on it since I purchased it (Summer 2007).

The CPU fan is getting noisy and I'd like to replace it.

The store where I purchased my laptop cannot order the replacement part.

Can anyone suggest a supplier for this part?  I live 200km west of 
Toronto, ON, Canada.




Translators needed for upcoming ComixWall 4.6

2009-11-11 Thread Soner Tari
I am planning to release ComixWall 4.6 in December. (Please see further
below for a summary of upcoming release announcement.) I am happy to
announce that I have frozen the web user interface strings as one of the
final few stages of the release process.

The ComixWall ISG project needs your help. Please contribute to the
project by translating the web user interface into your native language.

BENEFITS OF LOCALIZATION:

If you are reading these lines, you probably do not need any
translations from English to your native language. However, the benefit
of a localized firewall user interface may be two folds, at least:

1. Unprivileged user on the web user interface is for your boss, and
s/he may not speak English
2. Government organizations in your country may require or prefer IT
systems with localized user interfaces

After all, I was able to complete 50% of translations into Turkish in
1-2 hours, and this percent completion is enough for most purposes.

NEW TRANSLATION SCHEME:

A new scheme is in place to help you with translations. Strings on the
web user interface are now divided into files based on where they are
used. There are mainly four benefits of this scheme:

1. Translator knows what kind of user interface item s/he is translating
(menu, button, title, help box, etc.) even without knowing anything
about the web user interface
2. Files have priorities, and completing high priority translations is
easy, and also enough for most purposes
3. Translator may choose files based on how much time s/he can dedicate
for translations
4. Clear separation enables us to have multiple translators working on
the same language very easily without any conflicts (there are only a
few overlaps, which are easy to handle using gettext tools).

Po files to be translated can be downloaded at this link:
http://comixwall.org/dmdocuments/translations/

Following information and the list of current translators can be found
on the Translations page too:
http://comixwall.org/index.php?option=com_contenttask=viewid=58Itemid=51

Explanations for each file are as follows:

  _MENU: Left and top menus, 94 x mostly 1-word strings
  _CONTROL: Controls such as buttons, 38 x mostly 1-word strings
  _NOTICE: Warnings, 24 strings
  _TITLE: Important titles or captions, 101 x mostly 2-word strings
  _STATS: Statistics, 105 x mostly 2-word strings
  _HELPBOX: Important help boxes, 24 strings
  _TITLE2: Secondary titles, 393 x mostly 2-word strings

If above translations are complete and help boxes are disabled (on 4.6
help boxes can be disabled), the web interface can be considered as
localized. 

  _HELPBOX2: Secondary help boxes, 236 strings
  _HELPWINDOW: Help windows, 185 strings

If above translations are complete, the web interface can be considered
as localized, even when the help boxes are enabled. 

  _: Rest, 246 strings 

These files are combined into comixwall.po file before the final mo file
is created.

Please contact me if you want to be a translator. All languages are
welcome. I hope to have all translations ready by the end of November.

UPCOMING RELEASE ANNOUNCEMENT:

If you are interested in what to expect in the upcoming ComixWall 4.6,
here is a quite short summary:

Software updates:

  OpenBSD 4.6-stable, i.e. with -stable patches as of December
  Snort 2.8.5.1
  SnortIPS 4.6 with the new Priority and Keyword based blocking
  ClamAV 0.95.3
  IMSpector 0.9 with patches from cvs
  Dante 0.12.0
  Pmacct 0.12.0rc3
  OpenVPN 2.1_rc20
  PHP 5.2.11 with exec() patch
  Better ports

Web Administration Interface:

  MVC-like design pattern
  Controller validates all input from View
  Controller accepts commands defined by Model only
  OOP in the Model, and some in the View
  Common PHP code base used by Model, View, Controller, and Installer
  New statistics pages
  New logs pages
  New configuration pages
  Incremental statistics, stats are saved and updated incrementally
  New login method over HTTPs without problematic HTTP authentication
  Sessions timeout, and help boxes can be disabled now
  Pfw understands new match action now
  Much faster, more robust, stable, and consistent user interface
  Higher quality source code

Full changes to the new web user interface is impossible to list here.
The new look-and-feel may be relatively familiar, but the current user
interface is the result of a 4 months of intense refactoring and
development effort. In short, you should have a much pleasant experience
using the web user interface on 4.6.

Installer:

  OpenBSD installation is modified to customize and ease ComixWall
installation. Thanks to a modified auto-partitioner of OpenBSD 4.6, the
disk is partitioned according to ComixWall needs, so you don't need to
use the label editor at all.
  All install sets including siteXY.tgz are selected by default, so you
cannot 'not' install ComixWall by mistake now.
  OpenBSD installation questions are modified according to ComixWall
needs. For example, X11 related questions are never 

IP Aliasing with DHCP

2009-11-11 Thread Hugo Osvaldo Barrera
I want to set up an HTTPS server which serves two domains. I know this
is pretty much impossible with one IP, due to how SSL works.

However, my ISP throws me an Ethernet cable, and I can use as many IPs
as I want. - If I connect a switch to that cable, and 5 PCs, they each
get 5 REAL internet IPs.

I'v already seen the alias option for ifconfig, however, it always
refers to static IPs, and I've found no reference to this being
possible with dynamic IPs.
Is this possible? A single interface, with TWO dynamic IPs?



POOR support for layer 7 security in OBSD. Options or another OS?

2009-11-11 Thread David Taveras
I love OpenBSD focused security in many areas, and in the ones not
included in base there are always options in packages.

However specifically speaking about the options to complement as an
application level firewall seems it is truly underestimated the way I
see it:

What is the option for a web based IDS/IPS in OBSD? ModSecurity dated 2006
What rules could be applied for this? Same year outdated rules from
gotroot (not supported) because modsecurity.org doesnt even have an
old copy.
Could I install from source the newest ModSecurity 2.5 with the
ModSecurity Core Rules v2.0 ? No, because its not compatible with
apache1, unless you want to be more unsecure with apache2 from ports.


Reading this thread:
http://www.mail-archive.com/po...@openbsd.org/msg24615.html
It seems the conclusion is The only way that modsecurity increases
security is if your web applications where already insecure so the
first step would be to secure the web application then modsecurity
would not be needed.

Saying that to my opinion is the same as saying Why configure packet
filter to close incoming ports on the firewall if one could just
correctly configure the respective daemons  to listen to certain ports
and only to certain IPs.

Scenarios of importance for a WAF:

-- 10 programmers 10 modules ---
a.) Why assume that the sysadmin is the programmer, and why assume
there is just one knowledgeable programmer when there might be 10
programmers each coding a seperate module of a project which will get
uploaded? A code audit can only conclude (best effort) that the code
is secure in a specific time in history.

--- statistics of alerts 
b.) Why assume that a thread is just one threat part of a massive
effort for million of IPs, a thread can have a hacker behind it who if
he did not succeed one way will continue to work in other methods.
Having statistical information per day of threats categorized by a
level of risk will give a sysadmin leads to who he is, what he is
after, and any other pattern that will give time to act accordingly
for a future event targeted differently. The newest modsecurity does
this.

-- DoS to an application --
c.) Even if I trust every day the programmers, there is still the risk
of a application level DoS. PF can put a limit of maximum requests per
minute for an IP... but a DoS these days can be done with dozens of
thousands of different IPs each doing making a single burst POST to a
search form that will hog down the database. A WAF can examine the
payload of that and a custom rule can be set if one regex a pattern.

-- Information Leakage---

Lets just assume that a user is able to exploit a script... sure I can
block all ports on the server so he cannot scp or transfer data out,
but what if he tried to request the data from port 80... old
mod-security does inspect outgoing data for credit card information
but why stay there when the new modsecurity uses improved methods
block this?

I also tried looking at SNORT, but i dont think a sniffer would be
oriented in looking specifically at payloads of web requests based on
what I see, or one would have to be very creative of signatures.

Also: creating a reverse proxy in OBSD with Apache2 would be similar
to running windows virtually on top of OpenBSD. Apache2 port patches
are non priority and may take a while to be pushed. Forcing me to
compile from source and thus be on top of bleeding edge versions.


Do I have an alternative?


--David


P.D Iam not running a shared webhosting service.



tcpdump dhcp6 interpretation out of date

2009-11-11 Thread Philip Higgins
I noticed while doing some debugging on an ipv6 connection that the
included version of tcpdump uses an old draft version of dhcp6 for
its output.

src/usr.sbin/tcpdump/dhcp6.h was last edited almost 10 years, and
is based on draft 14.

For a while I was a little confused why wide-dhcpdc was sending a
release packet every half hour, but it was really a renew packet.

I know I can just grab the latest code from tcpdump.org and build it,
but what's the best way to get this updated in tree?

Thanks
Philip Higgins



Re: POOR support for layer 7 security in OBSD. Options or another OS?

2009-11-11 Thread Jason Dixon
On Wed, Nov 11, 2009 at 09:25:45PM -0600, David Taveras wrote:
 I love OpenBSD focused security in many areas, and in the ones not
 included in base there are always options in packages.
 
 However specifically speaking about the options to complement as an
 application level firewall seems it is truly underestimated the way I
 see it:

snip

 Do I have an alternative?

There are plenty of L7 tools in OpenBSD base and ports/packages to help
you reach your goals.  It's up to you to deploy and configure them
properly for your environment.  Just a few off the top of my head:

relayd(8)
authpf(8)
net/snort
www/mod_security

Indeed, mod_security is only currently available for apache-1.3.  But I
think the lack of modsecurity-2.x is only because nobody has stepped up
to complete the port, not because of any technical hurdles.

HTH.

-- 
Jason Dixon
DixonGroup Consulting
http://www.dixongroup.net/



Re: POOR support for layer 7 security in OBSD. Options or another OS?

2009-11-11 Thread Theo de Raadt
  Indeed, mod_security is only currently available for apache-1.3.  But I
  think the lack of modsecurity-2.x is only because nobody has stepped up
  to complete the port, not because of any technical hurdles.
 
 As i said, modsecurity 2 is only compatible with apache2, otherwise I
 would be able to install modsecurity2 on top of apache1 and that is
 not the case because of library differences.

Well perhaps more people should have gotten upset when Apache started
adding contract law language to their copyright notice.



Re: Problems with 4.5 as a KVM guest

2009-11-11 Thread Philip Higgins
On Sun, Nov 1, 2009 at 5:54 PM, Garry Dolley gdol...@arpnetworks.com wrote:
 On Sat, Oct 31, 2009 at 09:42:58AM +0100, Michiel van Baak wrote:

 I tried to upgrade my 4.5 and got the same.
 Sorry, have no way around it for the moment. I reverted the vm back to
 it's previous working state.

 This is how I got OpenBSD 4.5 working:

 http://scie.nti.st/2009/10/4/running-openbsd-4-5-in-kvm-on-ubuntu-linux-9-04


FWIW, this procedure works for 4.6 on Xen as well. The issue is probably
somewhere in the common QEMU code.

--
Philip Higgins



Re: POOR support for layer 7 security in OBSD. Options or another OS?

2009-11-11 Thread David Taveras
Hi,

On Wed, Nov 11, 2009 at 9:38 PM, Jason Dixon ja...@dixongroup.net wrote:
 There are plenty of L7 tools in OpenBSD base and ports/packages to help
 you reach your goals.  It's up to you to deploy and configure them
 properly for your environment.  Just a few off the top of my head:

 relayd(8)
 authpf(8)
 net/snort
 www/mod_security

The first two do not examine web application payloads originated from
requests.
Snort is not oriented either for this type of detection/prevention..
starting only for the fact that blocking this would have to interface
with pf instead of giving a 400 error page in the browser of the
client by apache. Correct me if iam wrong?




 Indeed, mod_security is only currently available for apache-1.3.  But I
 think the lack of modsecurity-2.x is only because nobody has stepped up
 to complete the port, not because of any technical hurdles.

As i said, modsecurity 2 is only compatible with apache2, otherwise I
would be able to install modsecurity2 on top of apache1 and that is
not the case because of library differences.



amavisd-new broken?

2009-11-11 Thread Fernando Quintero
Hi all,
Im trying to install amavisd-new on release 4.6 / 386, and i got an
error with the freeze-2.5p0 package.
Im looking for it in the packages list on the openbsd's mirrors and
can't find it.

Some idea?

# uname -a
OpenBSD correo.ejemplo.com 4.6 GENERIC#58 i386
#
# pkg_add -F conflicts  -v
ftp://ftp.openbsd.org/pub/OpenBSD/4.6/packages/i386/amavisd-new-2.6.3.tgz
parsing
ftp://ftp.openbsd.org/pub/OpenBSD/4.6/packages/i386/amavisd-new-2.6.3.tgz
Dependencies for amavisd-new-2.6.3 resolve to: arc-5.21op1,
p5-Archive-Zip-1.26, freeze-2.5p0, p5-Net-Server-0.97, clamav-0.95.2,
p5-Mail-DKIM-0.35, unzip-5.52p0, bzip2-1.0.5,
p5-Mail-SpamAssassin-3.2.5p1, p5-Convert-TNEF-0.17p0,
p5-Unix-Syslog-1.1p0, rpm2cpio-1.2, unarj-2.43, zoo-2.10.1p1,
p5-MIME-tools-5.427, lha-1.14i.ac20050924.1, ripole-0.2.0p0,
cabextract-1.2p0, lzop-1.01p0, p5-Convert-UUlib-1.09p0, unrar-3.85,
p5-BerkeleyDB-0.34p1 (todo:
freeze-2.5p0,lha-1.14i.ac20050924.1,lzop-1.01p0,p5-Archive-Zip-1.26,ripole-0.
2.0p0,unarj-2.43,unrar-3.85,unzip-5.52p0,zoo-2.10.1p1,p5-Convert-TNEF-0.17p0,
p5-Convert-UUlib-1.09p0,rpm2cpio-1.2,p5-BerkeleyDB-0.34p1,p5-Net-Server-0.97,
p5-MIME-tools-5.427,p5-Mail-DKIM-0.35,p5-Mail-SpamAssassin-3.2.5p1,clamav-0.9
5.2,p5-Unix-Syslog-1.1p0)
Error from
ftp://ftp.openbsd.org/pub/OpenBSD/4.6/packages/i386/freeze-2.5p0.tgz:
550 freeze-2.5p0.tgz: No such file or directory.
amavisd-new-2.6.3:Can't find freeze-2.5p0
/usr/sbin/pkg_add: freeze-2.5p0:Fatal error
#


Thanks.
--
--

Fernando Quintero
http://nonroot.blogspot.com/
*Just a nonroot User*



Re: POOR support for layer 7 security in OBSD. Options or another OS?

2009-11-11 Thread David Taveras
Hello Theo,

On Wed, Nov 11, 2009 at 10:15 PM, Theo de Raadt dera...@cvs.openbsd.org wrote:
 Well perhaps more people should have gotten upset when Apache started
 adding contract law language to their copyright notice.

Yes, I understand the fundamentals of this decision which in turn
gives us an operating system aimed truly on free available open source
software, and my goal is only to seek alternatives or complement to my
existing BSD network.

David



Re: IP Aliasing with DHCP

2009-11-11 Thread Matthew Dempsky
On Wed, Nov 11, 2009 at 6:19 PM, Hugo Osvaldo Barrera
h...@osvaldobarrera.com.ar wrote:
 I'v already seen the alias option for ifconfig, however, it always
 refers to static IPs, and I've found no reference to this being
 possible with dynamic IPs.
 Is this possible? A single interface, with TWO dynamic IPs?

This is completely untested (and using very recently added to
-current), but could you create a bridge(4) connecting the primary
interface to several vether(4) interfaces, and then run dhclient on
each vether with a dhclient.conf with 'supercede network-mask
255.255.255.255' for each?

No idea if this would actually work though...



Re: IP Aliasing with DHCP

2009-11-11 Thread Theo de Raadt
 On Wed, Nov 11, 2009 at 6:19 PM, Hugo Osvaldo Barrera
 h...@osvaldobarrera.com.ar wrote:
  I'v already seen the alias option for ifconfig, however, it always
  refers to static IPs, and I've found no reference to this being
  possible with dynamic IPs.
  Is this possible? A single interface, with TWO dynamic IPs?
 
 This is completely untested (and using very recently added to
 -current), but could you create a bridge(4) connecting the primary
 interface to several vether(4) interfaces, and then run dhclient on
 each vether with a dhclient.conf with 'supercede network-mask
 255.255.255.255' for each?
 
 No idea if this would actually work though...

It should, but I think a few more things need to get fixed before
that.  The bridge is not very efficient, though.