ixl driver very poor network performance
Dear @misc! We have several more complex networks where openbsd is the router. Structure of the network: * OpenBSD redundant routers - two OpenBSD - CARP - pfsync - LACP trunks for LAN (2x 10Gbit) (1 side switch #1, 2 side switch #2 + VPC ) use OpenBSD aggr device * Cisco Nexus 3K switch-es - VPC (2x40Gbit) - redundant LACP links (1 side switch #1, 2 side switch #2 + VPC ) * many VLANs * PF default block all trafic, and allowed traffic only * the servers connected usually 2x10Gbit LACP *hardware:* * we updated this system in one place to OpenBSD 7.4 hardware: Dell PE 640 (2x Xeon Gold 6134 CPU, 64Gb RAM, Intel X710 network cards) * we migrated the settings from the previous system (OpenBSD 7.0) the previous hardware was different! (2x Xeon E5-2650, 64Gb RAM, Intel X520 network cards) *Problem:* After upgrade with hardware change, we have very poor network performance!! Example: A simple veeam backup restore that goes through the openbsd router hangs the network completely (very big lag) In this case, the SSH connection on the router is have lag! But OpenBSD dont have high CPU usage. If i make simple iperf speed test from OpenBSD to other server (all device have 10Gbit LACP link): [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 171 MBytes 1.44 Gbits/sec [ 5] 1.00-2.00 sec 313 MBytes 2.63 Gbits/sec [ 5] 2.00-3.00 sec 398 MBytes 3.34 Gbits/sec [ 5] 3.00-4.00 sec 384 MBytes 3.22 Gbits/sec [ 5] 4.00-5.00 sec 419 MBytes 3.51 Gbits/sec [ 5] 5.00-6.00 sec 376 MBytes 3.16 Gbits/sec [ 5] 6.00-7.00 sec 325 MBytes 2.73 Gbits/sec [ 5] 7.00-8.00 sec 337 MBytes 2.82 Gbits/sec [ 5] 8.00-9.00 sec 339 MBytes 2.85 Gbits/sec [ 5] 9.00-10.00 sec 332 MBytes 2.78 Gbits/sec [ 5] 10.00-10.19 sec 62.5 MBytes 2.75 Gbits/sec Between other devices, servers, etc ... , the speed is perfectly fine (stable 9-10 Gbits/sec) Only routed performace is very-very slow. if I make a speed test between two OpenBSDs (master router, backup router) Better value but not perfect: [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 740 MBytes 6.20 Gbits/sec [ 5] 1.00-2.00 sec 781 MBytes 6.55 Gbits/sec [ 5] 2.00-3.00 sec 784 MBytes 6.58 Gbits/sec [ 5] 3.00-4.00 sec 783 MBytes 6.57 Gbits/sec [ 5] 4.00-5.00 sec 786 MBytes 6.59 Gbits/sec [ 5] 5.00-6.00 sec 796 MBytes 6.68 Gbits/sec [ 5] 6.00-7.00 sec 779 MBytes 6.54 Gbits/sec [ 5] 7.00-8.00 sec 774 MBytes 6.49 Gbits/sec [ 5] 8.00-9.00 sec 780 MBytes 6.55 Gbits/sec [ 5] 9.00-10.00 sec 786 MBytes 6.59 Gbits/sec [ 5] 10.00-10.00 sec 640 KBytes 10.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 7.61 GBytes 6.54 Gbits/sec receiver PF have ~2000 rules, but If i disabled PF on tested OpenBSD router, nothing changes. we've run out of ideas, what would be worth watching? -- Regards Gábor Szél email:gabor.s...@wantax.eu
WAB veb vs PF NAT
Dear @misc We want to implement a transparent router. We have some different public subnets ( /27, /28). All public subnet comes in on one interface - "WAN". We want to "forward" incoming traffic to an internal VLAN. Every device in this VLAN has a public IP address. We don't want to routing for this VLAN! We want to filter the traffic to the VLAN - (with PF) _We came up with the following solution for this:__ _ * em0 - public WAN interface * em1 - filtered "WAN" interface internal VLAN is connected here - switch port, native VLAN * veb0 - we made a bridge (VEB) between em0 and em1 we set it up: - link0 - it is necessary to get to the PF of the traffic * vport0 - Public IP address * PF have global block rule "block log all" _hostname.veb0 _ add em0 add em1 add vport0 maxaddr 256 link1 up _hostname.vport0 _ inet x.x.213.222 255.255.255.224 description "WAN-xxx" -inet6 up pf.conf This works fine! if I allow the traffic to the public IP address in VLAN, then it works. This requires "link1" setting, if link0 is set, it still works, only the traffic does not enter PF. That's fine, it's normal operation. - https://man.openbsd.org/veb.4 But! :) I want to NAT from local subnet(s), the traffic does not go out. _The problem:__ _ Simple NAT rule: pass out log on vport0 from $MGMT_LAN to ! nat-to xxx.xxx.213.222 is working, packets go to the vport interface: 13:12:47.401285 xxx.xxx.213.222 > 8.8.8.8: icmp: echo request (id:af4c seq:1283) [icmp cksum ok] (DF) (ttl 63, id 9289, len 84) 13:12:48.425303 xxx.xxx.213.222 > 8.8.8.8: icmp: echo request (id:af4c seq:1284) [icmp cksum ok] (DF) (ttl 63, id 9385, len 84) But, only to the vport interface!!! VEB bridge does not work from this direction! The packets don't move to em0, em1. PF not blocked this packets (pflog), no traffic in em0, em1 ... This traffic nated to vport0, and dont move out (to em0) if I change link1 option to link0 option on veb0. NAT is working, but incoming public traffic is not included to PF. How could the veb bridge be used for outgoing traffic with the link1 option? -- Regards Gábor Szél email:gabor.s...@wantax.eu
Re: PF pass not working (on complex "firewall")
Dear @misc
We found the error!
This is not PF problem.
I found this:
http://undeadly.org/cgi?action=article=20090127205841
If i modify an ipsec config *from:*
ike active esp from 172.20.123.0/24 to 172.20.122.0/24 \
*to:*
ike active esp from 172.20.123.0/24 *(192.168.123.0/24)* to
172.20.122.0/24 \
PF rules working correctly.
--
Regards
Gábor Szél
email:gabor.s...@wantax.hu
2022. 03. 05. 23:08 keltezéssel, Szél Gábor írta:
Dear @misc
We have an stupid problem.
On a complex firewall (currently PF rules 1200 row), one PASS rule not
working.
I do not know why.
There are many VLANs, WAN, LAN interfaces, many ipsec VPNs, CARP
(master-backup), pfsync, etc ...
PF main rules:
# set
#.
set block-policy drop
set loginterface $ext_wan1_if
set skip on { lo $pfsync_if }
set reassemble no
set timeout { tcp.established 600, tcp.closing 60 }
set optimization aggressive
set ruleset-optimization none
set limit { states 10, src-nodes 10, tables 10,
table-entries 10 }
# scrub
# -
match on $ext_wan1_if all scrub ( no-df max-mss 1440 random-id )
#. antispof
#.
antispoof quick for { $ext_wan1_if } inet
# anchors
# -
anchor "ftp-proxy/*"
# Block(s)
#.
block quick proto udp to port { 1985 8116 } # neighbours
HSRP & ...
block quick log on $ext_wan1_if from { }
label IPBlackList
block log inet6 all
block log all
So all interface traffic are basically forbidden (block).
Each traffic is allowed separately
We have one ipsec VPN, where there are NAT on both sides. (on both
sides have 192.168.x.x subnets, there is a subnet collision)
we want to solve a simple thing:
* comes in the packet on VPN tunnel to "virtual" IP address -
172.20.123.54 (bind to oBSD vlan interface)
* from this address PF redirect packet to destination server -
192.168.123.54
* destination server make return package, and send back
* the response packet comes in oBSD VLAN interface (vlan141)
* PF NAT-ed this packate to 172.20.123.54
* NAT-ed package return to source address in VPN
rules:
match in log on enc0 proto tcp from 172.20.122.0/24 to
172.20.123.54 port 5240 rdr-to 192.168.123.54 port 5240
pass in log on enc0 proto tcp from 172.20.122.0/24 to
192.168.123.54
pass out log on vlan141 from 172.20.122.0/24 to 192.168.123.54
match in log on vlan141 from 192.168.123.54 to
172.20.122.0/24 nat-to 172.20.123.54
pass in log on vlan141 from 172.20.123.54 to 172.20.122.0/24
pass in log on vlan141 from 192.168.123.54 to
172.20.122.0/24 (not needed, but ... :)
return package tcpdump:
nat-to, okay:
Mar 05 23:01:09.418806 rule 410/(match) [uid 0, pid 32543] match in on
vlan141: [orig src 192.168.123.54:5240, dst 172.20.122.10:39322]
172.20.123.54.51958 > 172.20.122.10.39322: S [bad tcp cksum 5166! ->
af7b] 966412712:966412712(0) ack 437277320 win 65160 1460,sackOK,timestamp 452766647 201794907,nop,wscale 7> (DF) (ttl 64,
id 0, len 60, bad ip cksum d8be! -> ed52)
and, PF block this packet:
Mar 05 23:01:09.418820 rule 9/(match) [uid 0, pid 32543]*block in on
vlan141:* [orig src 192.168.123.54:5240, dst 172.20.122.10:39322]
172.20.123.54.51958 > 172.20.122.10.39322: S [bad tcp cksum 5166! ->
af7b] 966412712:966412712(0) ack 437277320 win 65160 1460,sackOK,timestamp 452766647 201794907,nop,wscale 7> (DF) (ttl 64,
id 0, len 60, bad ip cksum d8be! -> ed52)
If i modify pass rule, to match rule:
match in log on vlan141 from 172.20.123.54
i see, match it works, but pass rule not works!
I've tried a lot of things already, without match rules, without nat
(okay, no route, but ...), it is always blocked.
Why can't i override the block rule?
Everywhere else goes ...
--
Regards
Gábor Szél
email:gabor.s...@wantax.hu
PF pass not working (on complex "firewall")
Dear @misc
We have an stupid problem.
On a complex firewall (currently PF rules 1200 row), one PASS rule not
working.
I do not know why.
There are many VLANs, WAN, LAN interfaces, many ipsec VPNs, CARP
(master-backup), pfsync, etc ...
PF main rules:
# set
#.
set block-policy drop
set loginterface $ext_wan1_if
set skip on { lo $pfsync_if }
set reassemble no
set timeout { tcp.established 600, tcp.closing 60 }
set optimization aggressive
set ruleset-optimization none
set limit { states 10, src-nodes 10, tables 10,
table-entries 10 }
# scrub
# -
match on $ext_wan1_if all scrub ( no-df max-mss 1440 random-id )
#. antispof
#.
antispoof quick for { $ext_wan1_if } inet
# anchors
# -
anchor "ftp-proxy/*"
# Block(s)
#.
block quick proto udp to port { 1985 8116 } # neighbours
HSRP & ...
block quick log on $ext_wan1_if from { }
label IPBlackList
block log inet6 all
block log all
So all interface traffic are basically forbidden (block).
Each traffic is allowed separately
We have one ipsec VPN, where there are NAT on both sides. (on both sides
have 192.168.x.x subnets, there is a subnet collision)
we want to solve a simple thing:
* comes in the packet on VPN tunnel to "virtual" IP address -
172.20.123.54 (bind to oBSD vlan interface)
* from this address PF redirect packet to destination server -
192.168.123.54
* destination server make return package, and send back
* the response packet comes in oBSD VLAN interface (vlan141)
* PF NAT-ed this packate to 172.20.123.54
* NAT-ed package return to source address in VPN
rules:
match in log on enc0 proto tcp from 172.20.122.0/24 to
172.20.123.54 port 5240 rdr-to 192.168.123.54 port 5240
pass in log on enc0 proto tcp from 172.20.122.0/24 to
192.168.123.54
pass out log on vlan141 from 172.20.122.0/24 to
192.168.123.54
match in log on vlan141 from 192.168.123.54 to
172.20.122.0/24 nat-to 172.20.123.54
pass in log on vlan141 from 172.20.123.54 to 172.20.122.0/24
pass in log on vlan141 from 192.168.123.54 to
172.20.122.0/24 (not needed, but ... :)
return package tcpdump:
nat-to, okay:
Mar 05 23:01:09.418806 rule 410/(match) [uid 0, pid 32543] match in on
vlan141: [orig src 192.168.123.54:5240, dst 172.20.122.10:39322]
172.20.123.54.51958 > 172.20.122.10.39322: S [bad tcp cksum 5166! ->
af7b] 966412712:966412712(0) ack 437277320 win 65160 1460,sackOK,timestamp 452766647 201794907,nop,wscale 7> (DF) (ttl 64, id
0, len 60, bad ip cksum d8be! -> ed52)
and, PF block this packet:
Mar 05 23:01:09.418820 rule 9/(match) [uid 0, pid 32543]*block in on
vlan141:* [orig src 192.168.123.54:5240, dst 172.20.122.10:39322]
172.20.123.54.51958 > 172.20.122.10.39322: S [bad tcp cksum 5166! ->
af7b] 966412712:966412712(0) ack 437277320 win 65160 1460,sackOK,timestamp 452766647 201794907,nop,wscale 7> (DF) (ttl 64, id
0, len 60, bad ip cksum d8be! -> ed52)
If i modify pass rule, to match rule:
match in log on vlan141 from 172.20.123.54
i see, match it works, but pass rule not works!
I've tried a lot of things already, without match rules, without nat
(okay, no route, but ...), it is always blocked.
Why can't i override the block rule?
Everywhere else goes ...
--
Regards
Gábor Szél
email:gabor.s...@wantax.hu
Re: aggr+vlan lost packets
Dear List!
We make some tests, i think this is intel em driver (82571EB) bug!
* if i move aggr0 from em devices to bnx devices, everything will be fine!
(only change trunkport from em to bnx)
* if i change intel network card to other intel network card with
82571EB chipset, not working.
* if i copy network interfaces config to another server (clear openbsd
6.8 install) with 6x Intel I210 network cards, everything will be fine!
* if i move SSD from working intel configuration server (I210) to
PE210 (82571EB), not working.
* i tested with oBSD 6.7, the problem exists ., but before reinstall
this server, on oBSD 6.1, LACP + 82571EB is working correctly.
we have many-many OpenBSD (router, firewall) installations, but we have
not yet experienced this problem. If possible, we use intel network cards.
--
Üdvözlettel,
Szél Gábor
WanTax Kft.
tel.: +36 20 3838 171
fax: +36 82 357 585
email:gabor.s...@wantax.hu
web:http://wantax.hu
web:http://halozatom.hu
2021. 03. 22. 12:06 keltezéssel, Szél Gábor írta:
Dear List!
We have very interesting problem!
We are reinstalling a OpenBSD firewall (6.1 -> 6.8), and we connect
new servers to firewall.
We replaced firewall for the duration of the update, is not currently
production use.
Test configuration:
- Dell PE210 II (Firewall) 2x Broadcom BCM5716 (bnx) integrated for
WANs, 4x Intel 82571EB (em) PCIexp, for LANs
- Dell PE740 (Proxmox) - 2x 1G, 2x10G (currently 1G links) (node1)
- Dell PE740 (Proxmox) - 2x 1G, 2x10G (currently 1G links) (node2)
- Cisco 2960s-48ts-s switch
All devices connected to switch, witch LACP:
- firewall 4x em{0|1|2|3} - aggr0
- nodes 2x1G (eno)
- we configured only one VLAN, VLAN2 for managment (10.110.2.0/24)
- All device used tagged VLAN for managment. (but we tested, untaged,
native VLAN, no difference)
- no PF rules! clear OpenBSD install!
I describe the configurations at the end of the letter!
IPs:
- FW - 10.110.2.1 (on managment vlan if)
- switch - 10.110.2.11 (on managment vlan if)
- node 1 - 10.110.2.51 (on vmbr2 if tagged, or vmbr0 if untagged)
- node 1 - 10.110.2.52 (on vmbr2 if tagged, or vmbr0 if untagged)
Problem:
- all LACP is UP, no problems reported
- nodes see each other nodes
- nodes see switch managment VLAN IP address
- firewall see switch managment VLAN IP address
but,
- node 1 see firewall IP address
- node 2 NOT see firewall IP address
- if we changed bond parameters on nodes - from 2x10G network if to
2x1G network if, node 2 see friewall, node 1 NOT see firewall
(different MAC address)
interesting problem:
- if a start tcpdump on firewall vlan2 or aggr0 interface, everything
will be fine!!! if i stoped tcpdump, bad again! - what??? :)
- if node 2 have packet lost, i start tcpdump in node 2, a see ICMP
request, and reply packets from/to firewall!
We use a lot of oBSD 6.8 firewalls with LACP + VLANs, we have not
experienced this.
_
__Configurations:_
oBSD:
*/etc/hostname.aggr0 *
trunkport em0
trunkport em1
trunkport em2
trunkport em3
172.19.253.1 netmask 255.255.255.255
description "c1 LACP"
up
*/etc/hostname.vlan2*
inet 10.110.2.1 255.255.255.0 10.110.2.255 vnetid 2 parent aggr0
description "Managment"
*sysctl.conf*
net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4
packets
net.inet.carp.log=3 # log level of carp(4) info, default 2
machdep.kbdreset=1 # permit console CTRL-ALT-DEL to do a
nice halt
ddb.panic=0 # do not enter ddb console on kernel
panic, reboot if possible
kern.bufcachepercent=90 # Allow the kernel to use up to 90% of
the RAM for cache (default 10%)
net.inet.ip.forwarding=1 # Permit forwarding (routing) of
packets through the firewall
net.inet.ip.mtudisc=0 # TCP MTU (Maximum Transmission Unit)
discovery off since our mss is small enough
net.inet.tcp.rfc3390=1 # Enable RFC3390 TCP window increasing
so larger CWND can take affect
vm.swapencrypt.enable=1 # encrypt pages that go to swap
machdep.kbdreset=1 # permit console CTRL-ALT-DEL to do a
nice halt
hw.allowpowerdown=1 # 0=Disable power button shutdown
hw.smt=1 # HT
*Cisco 2960S
*interface Port-channel1
description FW
switchport mode trunk
switchport nonegotiate
!
interface Port-channel2
description n1.pve
switchport mode trunk
switchport nonegotiate
!
interface Port-channel3
description n2.pve
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/1
description n1.pve
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
channel-group 2 mode active
!
interface GigabitEthernet0/2
description n1.pve
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
channel-group 2 mode active
!
interface GigabitEthernet0/3
description n2.pve
switchport mode trunk
switchport nonegotiate
channel-group 3 mode active
!
interface GigabitEthernet0/4
descrip
aggr+vlan lost packets
Dear List!
We have very interesting problem!
We are reinstalling a OpenBSD firewall (6.1 -> 6.8), and we connect new
servers to firewall.
We replaced firewall for the duration of the update, is not currently
production use.
Test configuration:
- Dell PE210 II (Firewall) 2x Broadcom BCM5716 (bnx) integrated for
WANs, 4x Intel 82571EB (em) PCIexp, for LANs
- Dell PE740 (Proxmox) - 2x 1G, 2x10G (currently 1G links) (node1)
- Dell PE740 (Proxmox) - 2x 1G, 2x10G (currently 1G links) (node2)
- Cisco 2960s-48ts-s switch
All devices connected to switch, witch LACP:
- firewall 4x em{0|1|2|3} - aggr0
- nodes 2x1G (eno)
- we configured only one VLAN, VLAN2 for managment (10.110.2.0/24)
- All device used tagged VLAN for managment. (but we tested, untaged,
native VLAN, no difference)
- no PF rules! clear OpenBSD install!
I describe the configurations at the end of the letter!
IPs:
- FW - 10.110.2.1 (on managment vlan if)
- switch - 10.110.2.11 (on managment vlan if)
- node 1 - 10.110.2.51 (on vmbr2 if tagged, or vmbr0 if untagged)
- node 1 - 10.110.2.52 (on vmbr2 if tagged, or vmbr0 if untagged)
Problem:
- all LACP is UP, no problems reported
- nodes see each other nodes
- nodes see switch managment VLAN IP address
- firewall see switch managment VLAN IP address
but,
- node 1 see firewall IP address
- node 2 NOT see firewall IP address
- if we changed bond parameters on nodes - from 2x10G network if to 2x1G
network if, node 2 see friewall, node 1 NOT see firewall (different MAC
address)
interesting problem:
- if a start tcpdump on firewall vlan2 or aggr0 interface, everything
will be fine!!! if i stoped tcpdump, bad again! - what??? :)
- if node 2 have packet lost, i start tcpdump in node 2, a see ICMP
request, and reply packets from/to firewall!
We use a lot of oBSD 6.8 firewalls with LACP + VLANs, we have not
experienced this.
_
__Configurations:_
oBSD:
*/etc/hostname.aggr0 *
trunkport em0
trunkport em1
trunkport em2
trunkport em3
172.19.253.1 netmask 255.255.255.255
description "c1 LACP"
up
*/etc/hostname.vlan2*
inet 10.110.2.1 255.255.255.0 10.110.2.255 vnetid 2 parent aggr0
description "Managment"
*sysctl.conf*
net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4
packets
net.inet.carp.log=3 # log level of carp(4) info, default 2
machdep.kbdreset=1 # permit console CTRL-ALT-DEL to do a
nice halt
ddb.panic=0 # do not enter ddb console on kernel
panic, reboot if possible
kern.bufcachepercent=90 # Allow the kernel to use up to 90% of
the RAM for cache (default 10%)
net.inet.ip.forwarding=1 # Permit forwarding (routing) of packets
through the firewall
net.inet.ip.mtudisc=0 # TCP MTU (Maximum Transmission Unit)
discovery off since our mss is small enough
net.inet.tcp.rfc3390=1 # Enable RFC3390 TCP window increasing so
larger CWND can take affect
vm.swapencrypt.enable=1 # encrypt pages that go to swap
machdep.kbdreset=1 # permit console CTRL-ALT-DEL to do a
nice halt
hw.allowpowerdown=1 # 0=Disable power button shutdown
hw.smt=1 # HT
*Cisco 2960S
*interface Port-channel1
description FW
switchport mode trunk
switchport nonegotiate
!
interface Port-channel2
description n1.pve
switchport mode trunk
switchport nonegotiate
!
interface Port-channel3
description n2.pve
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/1
description n1.pve
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
channel-group 2 mode active
!
interface GigabitEthernet0/2
description n1.pve
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
channel-group 2 mode active
!
interface GigabitEthernet0/3
description n2.pve
switchport mode trunk
switchport nonegotiate
channel-group 3 mode active
!
interface GigabitEthernet0/4
description n2.pve
switchport mode trunk
switchport nonegotiate
channel-group 3 mode active
!
*
***interface GigabitEthernet0/45
description FW-LACP
switchport mode trunk
switchport nonegotiate
channel-group 1 mode active
!
interface GigabitEthernet0/46
description FW-LACP
switchport mode trunk
switchport nonegotiate
channel-group 1 mode active
!
interface GigabitEthernet0/47
description FW-LACP
switchport mode trunk
switchport nonegotiate
channel-group 1 mode active
!
interface GigabitEthernet0/48
description FW-LACP
switchport mode trunk
switchport nonegotiate
channel-group 1 mode active
--
Üdvözlettel,
Szél Gábor
WanTax Kft.
tel.: +36 20 3838 171
fax: +36 82 357 585
email: gabor.s...@wantax.hu
web: http://wantax.hu
web: http://halozatom.hu
PF divert-packet reinjection
Dear @misc We test OpenBSD with Suricata in IPS mode. IPS mode requires PF divert-packet. simple rule to divert: pass in log quick on $_if proto tcp from ! to any divert-packet port 700 At first look everything is good! The packet goes to suricata, suricata check packet, if packet is "bad", throw away. But, not working good! if suricata does not drop packet, packet does not reinjected to PF! After divert-packet rule, any rule nothing works. a simple example: pass in log quick on $_if proto tcp from ! to any divert-packet port 700 block log all I'm trying to connect to host with SSH, divert to suricata is okay, and SSH connect is successful. Why? Next rule is block all! i thought apples wouldn't reinject packets, so i made a simple test: https://man.openbsd.org/divert.4 - example C code If i replaced suricate with example C code, the situation is the same! I see the packet in example log, and SSH connection is successfully. I think PF divert-packet does not reinject packets to PF. Is there a solution for this? -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu
PPTP NAT passthrough
Dear @misc Our customer need more parallel outgoing PPTP session. I know PPTP is no security VPN, but our client not have any options. (our customer remote partner accept only PPTP VPN ...) OpenBSD PF can't use parallel PPTP session. First session is NAT-ed, but second session is broken. I know OpenBSD not supported PPTP NAT passthrough. I found two, very old PPTP proxy for openbsd: * https://github.com/crvv/pptp-proxy This is ftp-proxy fork(?) * https://sourceforge.net/projects/frickin/ frickin 1.x working only fix remote PPTP address, not good for me. frickin 2.x (beta) not compiled on oBSD 6.6. pptp-proxy is compiled, and started, but not working. We tested very simple pf.conf (NAT, and some rules) pass in quick log on $int_if proto gre from any to ! $int_if:0 rdr-to 127.0.0.1 pass in quick log on $int_if proto tcp from any to ! $int_if:0 port 1723 rdr-to 127.0.0.1 port 2317 pptp-proxy is accepted session, but not working. (in tcpdump only 2 outgoing, 1 inbound packet found) Does anyone know a working solution for PPTP NAT passthrough? In openbsd based securityrouter.org firewall a found PPTP-Proxy support: https://securityrouter.org/wiki/Comparison But I don't know what to use. -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu
Re: aggr vs Nexus 3K
Sorry!, I was not attentive. in earlier oBSD versions, i think it was not necessary "up" in hostname.* configuration file when interface have IP address. if I add it to hostname.aggr0: trunkport oce0 trunkport oce1 172.20.253.12 netmask 255.255.255.255 description "." up LACP trunk is working! but interesting! hostname.trunk* is working without up! :) -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu 2019. 11. 14. 17:45 keltezéssel, Szél Gábor írta: Yes of course! :) if aggr up, Nexus suspend port channel: "not receiving any LACP BPDUs suspending" If trunk up, nexus not suspend port channel. -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email:gabor.s...@wantax.hu web:http://wantax.hu web:http://halozatom.hu 2019. 11. 14. 17:35 keltezéssel, hrv...@srce.hr írta: Can you do ifconfig aggr1 up ... Is it active then? Dana 14. stu 2019. 04:51 PM osoba Szél Gábor napisala je: Dear @misc We want to try use aggr pseudo LACP device. We have a working setup with trunk interface: oBSD: hostname.trunk0 trunkproto lacp trunkport oce0 trunkport oce1 172.20.253.11 netmask 255.255.255.255 description "" trunk0: flags=8943 mtu 9000 lladdr 28:80:23:e0:ab:40 description: C1 LACP index 25 priority 0 llprio 3 trunk: trunkproto lacp trunk id: [(8000,28:80:23:e0:ab:40,40CD,,), (7F9B,00:23:04:ee:be:0a,8002,,)] trunkport oce1 active,collecting,distributing trunkport oce0 active,collecting,distributing groups: trunk media: Ethernet autoselect status: active inet 172.20.253.11 netmask 0x Nexus 3K: Cisco LACP|vPC config is very simple: interface Ethernet1/2 description FW2-oce1 switchport mode trunk channel-group 2 mode active interface port-channel2 speed 1 description FW2 switchport mode trunk vpc 2 lacp rate normal vPC domain id : 10 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 consistency status : success vPC role : primary Number of vPCs configured : 8 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled Auto-recovery status : Disabled vPC Peer-link status - id Port Status Active vlans -- -- -- 1 Po99 up 1-2,10,12-13,20,30,40,50-53,60,62,101 vPC status id Port Status Consistency Reason Active vlans -- --- -- --- -- --- 2 Po2 up success success 1-2,10,12-1 3,20,30,40, 50-53,60,62 ,101 If we change trunk driver to aggr driver: oBSD: hostname.aggr1 trunkport oce0 trunkport oce1 172.20.253.12 netmask 255.255.255.255 description "" LACP is not working: oce0: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:cc:92 index 1 priority 0 llprio 3 trunk: trunkdev aggr1 media: Ethernet autoselect (10GbaseSR full-duplex,rxpause,txpause) status: active oce1: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:cc:92 index 2 priority 0 llprio 3 trunk: trunkdev aggr1 media: Ethernet autoselect (10GbaseSR full-duplex,rxpause,txpause) status: active aggr1: flags=8802 mtu 1500 lladdr fe:e1:ba:d0:cc:92 description: C1 LACP index 7 priority 0 llprio 7 trunk: trunkproto lacp trunk id: [(8000,fe:e1:ba:d0:cc:92,0007,,), (,00:00:00:00:00:00,,,)] oce0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:cc:92, key 0x7, port pri 0x8000 number 0x1 oce0 lacp actor state activity,aggregation,defaulted oce0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 oce0 lacp partner state activity,sync oce0 port oce1 lacp actor system pri 0x8000 mac fe:e1:ba:d0:cc:92, key 0x7, port pri 0x8000 number 0x2 oce1 lacp actor state activity,aggregation,defaulted oce1 lacp partner system pri 0x0 mac 00:00
Re: SPAM: Re: aggr vs Nexus 3K
Yes of course! :) if aggr up, Nexus suspend port channel: "not receiving any LACP BPDUs suspending" If trunk up, nexus not suspend port channel. -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu 2019. 11. 14. 17:35 keltezéssel, hrv...@srce.hr írta: Can you do ifconfig aggr1 up ... Is it active then? Dana 14. stu 2019. 04:51 PM osoba Szél Gábor napisala je: Dear @misc We want to try use aggr pseudo LACP device. We have a working setup with trunk interface: oBSD: hostname.trunk0 trunkproto lacp trunkport oce0 trunkport oce1 172.20.253.11 netmask 255.255.255.255 description "" trunk0: flags=8943 mtu 9000 lladdr 28:80:23:e0:ab:40 description: C1 LACP index 25 priority 0 llprio 3 trunk: trunkproto lacp trunk id: [(8000,28:80:23:e0:ab:40,40CD,,), (7F9B,00:23:04:ee:be:0a,8002,,)] trunkport oce1 active,collecting,distributing trunkport oce0 active,collecting,distributing groups: trunk media: Ethernet autoselect status: active inet 172.20.253.11 netmask 0x Nexus 3K: Cisco LACP|vPC config is very simple: interface Ethernet1/2 description FW2-oce1 switchport mode trunk channel-group 2 mode active interface port-channel2 speed 1 description FW2 switchport mode trunk vpc 2 lacp rate normal vPC domain id : 10 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 consistency status : success vPC role : primary Number of vPCs configured : 8 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled Auto-recovery status : Disabled vPC Peer-link status - id Port Status Active vlans -- -- -- 1 Po99 up 1-2,10,12-13,20,30,40,50-53,60,62,101 vPC status id Port Status Consistency Reason Active vlans -- --- -- --- -- --- 2 Po2 up success success 1-2,10,12-1 3,20,30,40, 50-53,60,62 ,101 If we change trunk driver to aggr driver: oBSD: hostname.aggr1 trunkport oce0 trunkport oce1 172.20.253.12 netmask 255.255.255.255 description "" LACP is not working: oce0: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:cc:92 index 1 priority 0 llprio 3 trunk: trunkdev aggr1 media: Ethernet autoselect (10GbaseSR full-duplex,rxpause,txpause) status: active oce1: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:cc:92 index 2 priority 0 llprio 3 trunk: trunkdev aggr1 media: Ethernet autoselect (10GbaseSR full-duplex,rxpause,txpause) status: active aggr1: flags=8802 mtu 1500 lladdr fe:e1:ba:d0:cc:92 description: C1 LACP index 7 priority 0 llprio 7 trunk: trunkproto lacp trunk id: [(8000,fe:e1:ba:d0:cc:92,0007,,), (,00:00:00:00:00:00,,,)] oce0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:cc:92, key 0x7, port pri 0x8000 number 0x1 oce0 lacp actor state activity,aggregation,defaulted oce0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 oce0 lacp partner state activity,sync oce0 port oce1 lacp actor system pri 0x8000 mac fe:e1:ba:d0:cc:92, key 0x7, port pri 0x8000 number 0x2 oce1 lacp actor state activity,aggregation,defaulted oce1 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 oce1 lacp partner state activity,sync oce1 port groups: aggr media: Ethernet autoselect status: no carrier inet 172.20.253.12 netmask 0x N3K logs: 2019 Nov 14 15:28:16 s1b %ETHPORT-5-SPEED: Interface Ethernet1/2, operational speed changed to 10 Gbps 2019 Nov 14 15:28:16 s1b %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/2, operational duplex mode changed to Full 2019 Nov 14 15:28:16 s1b %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/2, operational Receive Flow Control
aggr vs Nexus 3K
nnel group is 1 port channel is Po2 PDUs sent: 398962 PDUs rcvd: 381753 Markers sent: 0 Markers rcvd: 40 Marker response sent: 40 Marker response rcvd: 0 Unknown packets rcvd: 0 Illegal packets rcvd: 0 Lag Id: [ [(0, 0-0-0-0-0-0, 0, 0, 0), (0, 0-0-0-0-0-0, 0, 0, 0)] ] Operational as aggregated link since Thu Nov 14 14:11:10 2019 Local Port: Eth1/2 MAC Address= 0-23-4-ee-be-a System Identifier=0x8000,0-23-4-ee-be-a Port Identifier=0x8000,0x101 Operational key=32769 LACP_Activity=active LACP_Timeout=Long Timeout (30s) Synchronization=IN_SYNC Collecting=true Distributing=true Partner information refresh timeout=Long Timeout (90s) Actor Admin State=(Ac-1:To-1:Ag-1:Sy-0:Co-0:Di-0:De-0:Ex-0) Actor Oper State=(Ac-1:To-0:Ag-1:Sy-1:Co-1:Di-1:De-1:Ex-0) Neighbor: 0x0 MAC Address= 0-0-0-0-0-0 System Identifier=0x0, Port Identifier=0x0,0x0 Operational key=0 LACP_Activity=unknown LACP_Timeout=Long Timeout (30s) Synchronization=NOT_IN_SYNC Collecting=false Distributing=false Partner Admin State=(Ac-0:To-1:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0) Partner Oper State=(Ac-0:To-0:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0) Aggregate or Individual(True=1)= 1 If I try disable LACP BPDUs check on port channel interface: no lacp suspend-individual or enabe "lacp graceful-convergence". still not working, if i removed aggr interface, and make new trunk interface with LACP, it works great. I dont know whats wrong :( Anyone have any ideas? -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu
Re: 10Gbit network work only 1Gbit
alright, old hardver, but network speed is limited only 1 gbit! not more! (not per VLAN, global 1 gbit limit!) This is not hardvare preformance problem, i think this hardvare have enough performace to 2x1Gbit parallel network connection. We monitoring CPU, load, etc max load is 2-3, but not permanently We reinstall backup firewall oBSD 6.6, and make some test. Thx your reply! -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu 2019. 11. 12. 20:46 keltezéssel, Diana Eichert írta: Someone else mentioned your underlying system, Dell 2950, is ancient. It really is ancient, just because it links up @ 10G doesn't mean you are going to see anywhere near 10G On Tue, Nov 12, 2019 at 3:10 AM Szél Gábor wrote: Dear Hrvoje, Theo, Thank you for your answers! answers to the questions: - who is parent interface for carp? -> vlan ( carp10 interface parent vlan10 -> vlan10 interface parent -> trunk0 ) - why vlan interfaces don't have ip address ? -> it wasn't needed! i think vlan interface need only tag packages. Carp (over vlan) interface have IP address. - vether implies that you have bridge? -> yes whe have only one bridge for bridget openvpn clients, but we will eliminate it. we will do the following: - refresh our backup firewall to oBSD 6.6 - replace trunk interface with aggr - remove bridge interface if there was an update finised, I'll write again! -- Regars Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu 2019. 11. 11. 23:42 keltezéssel, Hrvoje Popovski írta: On 11.11.2019. 13:42, Szél Gábor wrote: Hello @misc, Hi, We have an interesting problem, we run a lot of OpenBSD router/firewalls in many places. We have a larger network than our client, 300-400 local wired or wireless endpoint, 20+ VLAN, 20+ switches. Network structure: * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated 2x40Gbit Peer link, keepalive link) * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module. some 3750x stacked (2 or 3 switch) * Main and access switches have redundant 10Gbit fiber link (LACP) * when is possible jumbo frame is enabled (mtu 9000) Firewall/router: * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+ network card hardware is really old, if you can, buy something newer * redundant design - CARP, pfsync, ifstated, etc master-backup> configuration * HP NC550SFP network card, oce driver (mtu 9000) if you can change oce with ix. ixl is not so bad .. . * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access link) - use openbsd trunk interface * all vlan used openbsd pseudo-device over trunk interface (VLANs not have have IP address, only up) update to openbsd 6.6 or snapshot and insted of trunk use aggr. why vlan interfaces don't have ip address ? * all network subnets defined in CARP interfaces, only managment VLAN have address on VLAN interface. who is parent interface for carp ? * some vether virtual interface for VPN, DNS, etc ... vether implies that you have bridge? bridge is slow.. * some tun and tap interface for VPN * enc interface for ipsec ipsec is performance killer big time ... even for traffic that doesn't go through ipsec tunnel .. if you can move ipsec or any vpn stuff to other boxes that you speed up your firewalls ... * one bridge interface for openVPN (during termination) vether is in that bridge? * OpenBSD 6.3 64bit please, update boxes regularly.. you have carp and pfsync, you can do that without any problem .. PF: * global block rule (block all) * ruleset-optimization none * optimization aggressive * reassemble no * block-policy drop * scrub enabled * antispoof enabled * regulating traffic between subnets with pf pass in/out rules * pf.conf currently 1500+ lines * the number of connections during the day in PF 10 000+ Problem: We see that network traffic is limited to 1Gbit on firewall. Not in one link, not IP-to-IP, to the whole firewall! yes ... ipsec, trunk, pf are for whole firewall .. and even if you have fastest box in the world you will not get performance that you want .. example: * i make test traffic form VLAN 2 to VLAN 12 witch iperf. test PC-s have 1Gbit ethernet cards. Speed is okay, ~800Mbit/sec * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another PC-s (they also have 1gbit ethernet cards) speed is not good! ~60-80Mbit/sec * if i stopped first speed test (2->12), second test speed is okay! (2->20) * but i make test from completely different VLANs, 2->12 and 20->30, the result is so. if you disable pf on vlan intefaces (set skip on vlan2/vlan12
Re: 10Gbit network work only 1Gbit
Dear Hrvoje, Theo, Thank you for your answers! answers to the questions: - who is parent interface for carp? -> vlan ( carp10 interface parent vlan10 -> vlan10 interface parent -> trunk0 ) - why vlan interfaces don't have ip address ? -> it wasn't needed! i think vlan interface need only tag packages. Carp (over vlan) interface have IP address. - vether implies that you have bridge? -> yes whe have only one bridge for bridget openvpn clients, but we will eliminate it. we will do the following: - refresh our backup firewall to oBSD 6.6 - replace trunk interface with aggr - remove bridge interface if there was an update finised, I'll write again! -- Regars Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu 2019. 11. 11. 23:42 keltezéssel, Hrvoje Popovski írta: On 11.11.2019. 13:42, Szél Gábor wrote: Hello @misc, Hi, We have an interesting problem, we run a lot of OpenBSD router/firewalls in many places. We have a larger network than our client, 300-400 local wired or wireless endpoint, 20+ VLAN, 20+ switches. Network structure: * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated 2x40Gbit Peer link, keepalive link) * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module. some 3750x stacked (2 or 3 switch) * Main and access switches have redundant 10Gbit fiber link (LACP) * when is possible jumbo frame is enabled (mtu 9000) Firewall/router: * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+ network card hardware is really old, if you can, buy something newer * redundant design - CARP, pfsync, ifstated, etc master-backup> configuration * HP NC550SFP network card, oce driver (mtu 9000) if you can change oce with ix. ixl is not so bad .. . * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access link) - use openbsd trunk interface * all vlan used openbsd pseudo-device over trunk interface (VLANs not have have IP address, only up) update to openbsd 6.6 or snapshot and insted of trunk use aggr. why vlan interfaces don't have ip address ? * all network subnets defined in CARP interfaces, only managment VLAN have address on VLAN interface. who is parent interface for carp ? * some vether virtual interface for VPN, DNS, etc ... vether implies that you have bridge? bridge is slow.. * some tun and tap interface for VPN * enc interface for ipsec ipsec is performance killer big time ... even for traffic that doesn't go through ipsec tunnel .. if you can move ipsec or any vpn stuff to other boxes that you speed up your firewalls ... * one bridge interface for openVPN (during termination) vether is in that bridge? * OpenBSD 6.3 64bit please, update boxes regularly.. you have carp and pfsync, you can do that without any problem .. PF: * global block rule (block all) * ruleset-optimization none * optimization aggressive * reassemble no * block-policy drop * scrub enabled * antispoof enabled * regulating traffic between subnets with pf pass in/out rules * pf.conf currently 1500+ lines * the number of connections during the day in PF 10 000+ Problem: We see that network traffic is limited to 1Gbit on firewall. Not in one link, not IP-to-IP, to the whole firewall! yes ... ipsec, trunk, pf are for whole firewall .. and even if you have fastest box in the world you will not get performance that you want .. example: * i make test traffic form VLAN 2 to VLAN 12 witch iperf. test PC-s have 1Gbit ethernet cards. Speed is okay, ~800Mbit/sec * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another PC-s (they also have 1gbit ethernet cards) speed is not good! ~60-80Mbit/sec * if i stopped first speed test (2->12), second test speed is okay! (2->20) * but i make test from completely different VLANs, 2->12 and 20->30, the result is so. if you disable pf on vlan intefaces (set skip on vlan2/vlan12) do you get better performace? and after that for disable ipsec and try testing again... do you see differences ? This is firewall (openbsd) limitation, but we don't understand why? I know openbsd VLAN interface has a speed problem, this is it? not in OpenBSD 6.6 I know it's so difficult to make a mistake from some information, what should we look at? OpenBSD is great router and firewall that can do so much for you .. but please you really need to rethink your hardware and setup ..
10Gbit network work only 1Gbit
Hello @misc, We have an interesting problem, we run a lot of OpenBSD router/firewalls in many places. We have a larger network than our client, 300-400 local wired or wireless endpoint, 20+ VLAN, 20+ switches. Network structure: * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated 2x40Gbit Peer link, keepalive link) * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module. some 3750x stacked (2 or 3 switch) * Main and access switches have redundant 10Gbit fiber link (LACP) * when is possible jumbo frame is enabled (mtu 9000) Firewall/router: * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+ network card * redundant design - CARP, pfsync, ifstated, etc master-backup configuration * HP NC550SFP network card, oce driver (mtu 9000) * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access link) - use openbsd trunk interface * all vlan used openbsd pseudo-device over trunk interface (VLANs not have have IP address, only up) * all network subnets defined in CARP interfaces, only managment VLAN have address on VLAN interface. * some vether virtual interface for VPN, DNS, etc ... * some tun and tap interface for VPN * enc interface for ipsec * one bridge interface for openVPN (during termination) * OpenBSD 6.3 64bit PF: * global block rule (block all) * ruleset-optimization none * optimization aggressive * reassemble no * block-policy drop * scrub enabled * antispoof enabled * regulating traffic between subnets with pf pass in/out rules * pf.conf currently 1500+ lines * the number of connections during the day in PF 10 000+ Problem: We see that network traffic is limited to 1Gbit on firewall. Not in one link, not IP-to-IP, to the whole firewall! example: * i make test traffic form VLAN 2 to VLAN 12 witch iperf. test PC-s have 1Gbit ethernet cards. Speed is okay, ~800Mbit/sec * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another PC-s (they also have 1gbit ethernet cards) speed is not good! ~60-80Mbit/sec * if i stopped first speed test (2->12), second test speed is okay! (2->20) * but i make test from completely different VLANs, 2->12 and 20->30, the result is so. This is firewall (openbsd) limitation, but we don't understand why? I know openbsd VLAN interface has a speed problem, this is it? I know it's so difficult to make a mistake from some information, what should we look at? -- Üdvözlettel, Szél Gábor WanTax Kft. tel.: +36 20 3838 171 fax: +36 82 357 585 email: gabor.s...@wantax.hu web: http://wantax.hu web: http://halozatom.hu
