Hi Jan, but exactly this Setup works at Hetzner on our Rrootservers.
Hetzner sends the complete traffic to the MAIN IP: admin_ip=138.XXX.XX.XXA The vnic0 gets the first IP from the SUBNET: 88.XXX.XXX.XXA For IPv6 use vnic1 and plumb 2a01:AAAA:AAAA:AAAA::EEEA/64 Hetzner specifically you push the entire traffic over fe80::1 [root@root1 /zones/ass.de/template]# cat vm01-root1-fw1-opnsense.json { "brand": "kvm", "alias": "root1-fw1-opnsense", "resolvers": [ "8.8.8.8", "8.8.4.4" ], "ram": "4096", "vcpus": "2", "nics": [ { "__comment" : "hetzner: 88.XXX.XXX.XXB", "nic_tag": "admin", "allowed_ips": [ "2a01:AAAA:AAAA:AAAA::B:CCCC" ], "ip": "88.XXX.XXX.XXB", "ips": ["88.XXX.XXX.XXB/29", "addrconf"], "netmask": "255.255.255.248", "gateway": "88.XXX.XXX.XXA", "model": "virtio", "primary": true }, { "__comment" : "internal: 10.XXX.XXX.XXD", "nic_tag": "vswitch0", "ip": "10.XXX.XXX.XXD", "ips": ["10.XXX.XXX.XXD/22", "addrconf"], "netmask": "255.255.252.0", "gateway": "10.XXX.XXX.XXE", "model": "virtio" } ], "disks": [ { "boot": true, "model": "virtio", "compression": "lz4", "size": 16384, "block_size": 8192 } ] } [root@root1 /zones/ass.de/template]# For security reasons create firewall rules at Hetzner Robot for the Root Servers. (DROP all traffic to the MAIN IP, but allow all other for the Subnet IPs) vmadm update UUID vnc_port=ZZZA (this only activates the vnc port on the MAIN IP) / to disable the vnc access -> use: vmadm update UUID vnc_port=-1 And with ssh -p XXXX -i /home/fuu/.ssh/id_bar -L 9999: 138.XXX.XX.XXA:ZZZA r...@138.xxx.xx.xxa<mailto:r...@138.xxx.xx.xxa> you can tunnel the plain vnc access locally Works like a charm. From linux I recognize that you can rewrite mac addresses on the bridge (proxyarp) but I did not try this under smartos. I have used a lot of network stuff in my LXC-to-GO Project: https://github.com/plitc/lxc-to-go/blob/master/content/README.DIAGRAM.md Or crazy stuff on FreeBSD with up to 256 Bridges: https://blog.plitc.eu/2014/freebsd-10-komplexe-bridge-zones-mit-lacp-uplink/ But my impression is, the more one uses complicated techniques, the more cumbersome it becomes to the conclusion to debug (like proxyarp, multiple source & destination nat between vms on the same host) 😉 Mit freundlichen Grüßen DANIEL PLOMINSKI Leiter – IT / Head of IT Telefon 09265 808-151 | Mobil 0151 58026316 | d...@ass.de<mailto:d...@ass.de> PGP Key: http://pgp.ass.de/2B4EB20A.key [cid:C17DB6FB-5F79-4BCC-AAB4-CAB59266BC29@localdomain] ASS-Einrichtungssysteme GmbH ASS-Adam-Stegner-Straße 19 | D-96342 Stockheim Geschäftsführer: Matthias Stegner, Michael Stegner, Stefan Weiß Amtsgericht Coburg HRB 3395 | Ust-ID: DE218715721 [cid:E40AEC87-91EE-472A-901A-ECAD3F5801FB@localdomain] Von: Ján Poctavek [mailto:jan.pocta...@erigones.com] Gesendet: Dienstag, 12. September 2017 13:08 An: smartos-discuss@lists.smartos.org Betreff: Re: AW: [smartos-discuss] smartos in dedicated hosting Thank you Daniel for sharing your setup. I use your scenario in some installations, also with etherstubs and GZ routing. But: 1. this is exactly I'd like to avoid - need for creating an own custom script for networking 2. you are creating a vnic0 interface over e1000g0. It will not work with e.g. Hetzner or OVH because you are changing the external MAC. Jan On 12. 9. 2017 11:17, Daniel Plominski wrote: Hi Poctavek, Example: DATACENTER <=> DC Switch <=> Rootserver (SmartOS + VMs) SmartOS has 1 ADMIN interfac e with an additional /29 Subnet [root@root1 /usbkey]# cat config # # This file was auto-generated and must be source-able by bash. # ### ### ### ASS // ### ### ### admin_nic=AA:BB:CC:DD:EE:00 admin_ip=dhcp headnode_default_gateway=138.XXX.XX.XXF dns_resolvers=8.8.8.8,8.8.4.4 dns_domain=ass.de ntp_hosts=0.smartos.pool.ntp.org compute_node_ntp_hosts=dhcp ... … … ### ### ### // ASS ### ### ### # EOF [root@root1 /usbkey]# [root@root1 /opt/custom/smf]# cat subnet-routing-setup.xml <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'> <service_bundle type='manifest' name='export'> <service name='site/subnet-routing-setup' type='service' version='0'> <create_default_instance enabled='true'/> <single_instance/> <dependency name='network' grouping='require_all' restart_on='error' type='service'> <service_fmri value='svc:/milestone/network:default'/> </dependency> <dependency name='filesystem' grouping='require_all' restart_on='error' type='service'> <service_fmri value='svc:/system/filesystem/local'/> </dependency> <exec_method name='start' type='method' exec='/opt/custom/scripts/subnet-routing-setup' timeout_seconds='60'> <method_context> <method_credential user='root' group='staff'/> <method_environment> <envvar name='PATH' value='/usr/bin:/usr/sbin:/bin'/> </method_environment> </method_context> </exec_method> <exec_method name='restart' type='method' exec=':kill' timeout_seconds='60'> <method_context> <method_credential user='root' group='staff'/> </method_context> </exec_method> <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'> <method_context> <method_credential user='root' group='staff'/> </method_context> </exec_method> <property_group name='startd' type='framework'> <propval name='duration' type='astring' value='transient'/> <propval name='ignore_error' type='astring' value='core,signal'/> </property_group> <property_group name='application' type='application'/> <stability value='Evolving'/> <template> <common_name> <loctext xml:lang='C'>subnet-routing-setup</loctext> </common_name> </template> </service> </service_bundle> [root@root1 /opt/custom/smf]# [root@root1 /opt/custom/scripts]# cat subnet-routing-setup #!/bin/sh . /lib/svc/share/smf_include.sh #// disable services svcadm disable svc:/network/rpc/bind:default #// HOST: ipv6 #/dladm create-vnic -l e1000g0 vnic1 ifconfig e1000g0 inet6 plumb ifconfig e1000g0 inet6 addif 2a01:AAAA:AAAA:AAAA::EEEA/64 up route add -inet6 fe80::1 2a01:AAAA:AAAA:AAAA::EEEA -interface route add -inet6 default fe80::1 svcadm enable ipv6-forwarding routeadm -e ipv6-forwarding routeadm -e ipv6-routing routeadm -u #// VM: ipv4 dladm create-vnic -l e1000g0 vnic0 ifconfig vnic0 plumb 88.XXX.XXX.XXA netmask 255.255.255.248 up svcadm enable route routeadm -e ipv4-forwarding routeadm -e ipv4-routing routeadm -u #// VM: internal vswitch (intern / ass vpn) # create a etherstub dladm create-etherstub vswitch0 dladm set-linkprop -p mtu=1500 vswitch0 #// VM: internal vswitch (intern / coorp vpn) # create a etherstub dladm create-etherstub vswitch1 dladm set-linkprop -p mtu=1500 vswitch1 exit $SMF_EXIT_OK [root@root1 /opt/custom/scripts]# Now use the SmartOS IP: 88.XXX.XXX.XXA as default gateway for ALL your Zone / KVM Machines Another method would be: NAT Mit freundlichen Grüßen DANIEL PLOMINSKI Leiter – IT / Head of IT Telefon 09265 808-151 | Mobil 0151 58026316 | d...@ass.de<mailto:d...@ass.de> PGP Key: http://pgp.ass.de/2B4EB20A.key [cid: C17DB6FB-5F79-4BCC-AAB4-CAB59266BC29@localdomain] ASS-Einrichtungssysteme GmbH ASS-Adam-Stegner-Straße 19 | D-96342 Stockheim Geschäftsführer: Matthias Stegner, Michael Stegner, Stefan Weiß Amtsgericht Coburg HRB 3395 | Ust-ID: DE218715721 [cid: E40AEC87-91EE-472A-901A-ECAD3F5801FB@localdomain] -----Ursprüngliche Nachricht----- Von: Ján Poctavek [mailto:jan.pocta...@erigones.com] Gesendet: Dienstag, 12. September 2017 10:45 An: smartos-discuss@lists.smartos.org<mailto:smartos-discuss@lists.smartos.org> Betreff: [smartos-discuss] smartos in dedicated hosting Hi guys, I have a bit of complications installing SmartOS in the dedicated hosting. Many hosting providers have an additional security (network filter) that allows a dedicated server to communicate to the internet only by using the assigned IP address *together* with the default MAC address. But when I configure the external interface with IP address in the config file, the IP address is created over external0 vnic. And this new vnic has a new MAC address that is different from default HW NIC address. As a result, all communication gets dropped. Is there a way to solve this using a config file? The workarounds I can come with: 1. add a new SMF service that manually adds the IP address over the physical NIC 2. modify the network/physical script 3. add <nictag>_preserve_mac config property to add IP address directly to physical NIC The thing is that the first two options do not scale and I don't want to implement the third if it already exists. Thanks for hints. Jan smartos-discuss | Archives<https://www.listbox.com/member/archive/184463/=now> [https://www.listbox.com/images/feed-icon-10x10.jpge18b463.jpg?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2ZlZWQtaWNvbi0xMHgxMC5qcGc] <https://www.listbox.com/member/archive/rss/184463/29198361-7a6753c0> | Modify<https://www.listbox.com/member/?&> Your Subscription [https://www.listbox.com/images/listbox-logo-small.pnge18b463.png?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2xpc3Rib3gtbG9nby1zbWFsbC5wbmc]<http://www.listbox.com> ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com