I was able to see, however, that you have re-introduced the WIDE_TC_MARKS
option to the default shorewall.conf. Don't do that - that option has
been superseded for some time.
Thanks for clarification. I got the error message "ERROR: shorewall.conf does not
set option WIDE_TC_MARKS at /root/shorewall/tools/build/annotate.pl line 37."
when trying to build, so I gave it a shot. Wanted to remove it afterwards, as
it's not directly related.
I believe that:
1) The order in which Docker and Shorewall are started should not matter.
2) Either product should be able to be started, stopped, restarted or
reloaded without effecting the other product.
Yes that would be ideal. Seems like something each product would need to take
care of.
As Docker decided to add ACCEPT rules which are hit if the jumps in the FORWARD
chain aren't removed, Shorewall
needs to have the last word. Thus in our current approach Shorewall needs to be
reloaded after a start/restart of
Docker, as the jumps in the FORWARD chain would be re-added.
When Shorewall executes a 'start', 'stop', or 'reload' ('restart' is a
'stop' followed by a 'start'), it saves all Docker-related chains and
rules, then integrates them into the new configuration.
Actually it looks like Shorewall-5.2.3 drops rules (that were added after
start) on restart, while conserving them on reload.
The problem with that approach is that when Docker adds something new, that
new content doesn't get saved/restored by Shorewall. That fundamental
weakness in the current design caused me to consider dropping Docker
support entirely in Shorewall 5.2.
Maybe that's the problem in the issue above. Or do you mean the short window
between export/re-import of existing rules on start (or is that
performed atomically like mentioned). It might be possible to not re-import the
state before Shorewall was started, but clean the current
state when Shorewall is stopped.
Generally it's probably not ideal to blow up Shorewall with special
integrations for other iptables using products.
Maybe the general conserving integration with hooks (like already given with
extension scripts) is an adequate choice.
We went for further integration as it's partially there and implementing the
missing parts in extension scripts
also felt unsatisfactory and it looked like just a few changes.
If I ran Docker ...
We thought about scenarios like that, but didn't want to add the virtualization
layer at first. Maybe we could also reconsider that.
If an integration could fulfill your two precepts I'd prefer that.
Thanks,
Niko
Am 06.03.19 um 02:33 schrieb Tom Eastep:
Let's move this discussion to the Development list -- it is not
appropriate for the users list (I've added shorewall-devel to the To: list).
My hotel's internet access is too poor to look at your tree tonight. I
was able to see, however, that you have re-introduced the WIDE_TC_MARKS
option to the default shorewall.conf. Don't do that - that option has
been superseded for some time.
Note: And when an option is added to the default shorewall.conf, it must
also be added to the Sample .confs and all of the Shorewall6 .confs, and
to the shorewall.conf manpage. Otherwise, the Shorewall build process fails.
I believe that:
1) The order in which Docker and Shorewall are started should not matter.
2) Either product should be able to be started, stopped, restarted or
reloaded without effecting the other product.
Those two precepts led to the design of the current Docker support in
Shorewall.
When Shorewall executes a 'start', 'stop', or 'reload' ('restart' is a
'stop' followed by a 'start'), it saves all Docker-related chains and
rules, then integrates them into the new configuration. I assume the
'save' part of that algorithm is what the code in Chains.pm that you
refer to below is doing (Note that Shorewall atomically replaces the
entire Netfilter configuration during each of these operations). The
problem with that approach is that when Docker adds something new, that
new content doesn't get saved/restored by Shorewall. That fundamental
weakness in the current design caused me to consider dropping Docker
support entirely in Shorewall 5.2. The alternative, which I settled on,
is to continue to add new logic to Shorewall each time that Docker adds
new chains (and jumps to those chains). And that new logic must detect
whether the new Docker feature is in use or not, and must not change the
generated ruleset unless the new feature is present.
If I ran Docker (which I don't - I run LXC on my Shorewall system), I
would not run Shorewall on the same system. I would rather front-end the
Docker system with a box running Shorewall. Note that since Docker can
run in a VM, you can still run Docker and Shorewall on the same physical
system by running Shorewall in the host and Docker in a VM.
-Tom
On 3/5/19 2:20 AM, Niko Mayr wrote:
Hi!
As mentioned in a previous post
(https://sourceforge.net/p/shorewall/mailman/message/36599490/) we are
experimenting with Docker 18.09 in swarm mode and Shorewall 5.2.3, and the
current integration doesn't work.
We decided to try to quickly implement a patch, which can be found here
https://sourceforge.net/u/nikm/shorewall/ci/master/tree/. We tried to
extend the existing integration to reflect the current chain and
interface configuration utilized in Docker in swarm mode. In general it
would be nice if we could get some feedback on that sometime to be able
to prepare a merge request if wanted, but there is one thing that would
be especially interesting. Docker 18.09 in swarm mode applies some sort
of firewall strategy and it would be interesting where you'd see the
ideal Shorewall integration.
Docker iptables usage:
1) In the nat table Docker adds the two chains DOCKER and
DOCKER-INGRESS. Jump rules added to the PREROUTING chain direct to those
chains on dst-type LOCAL.
PREROUTING chain:
DOCKER-INGRESS all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
DOCKER all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
In those chains Docker dynamically adds DNAT rules to direct the
registered task ports to the concerned interface. Which chain it uses
depends on the utilization of the routing-mesh.
Sample DOCKER chain:
RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
RETURN all -- docker_gwbridge * 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- !docker0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:10001 to:172.17.0.2:443
Sample DOCKER-INGRESS chain:
DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:10002 to:172.18.0.2:10002
RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
2) The DNAT rules above don't direct to local interfaces directly
(docker0: inet 172.17.0.1/16, docker_gwbridge: inet 172.18.0.1/16) so
that takes us to the FORWARD chain in the filter table.
In the filter table Docker adds five chains DOCKER-USER,
DOCKER-ISOLATION-STAGE-1, DOCKER-ISOLATION-STAGE-2, DOCKER,
DOCKER-INGRESS. Jump rules in the FORWARD chain distribute
traffic to those.
FORWARD chain:
DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0
0.0.0.0/0
ACCEPT all -- * docker0
0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- * docker0
0.0.0.0/0 0.0.0.0/0
ACCEPT all -- docker0 !docker0
0.0.0.0/0 0.0.0.0/0
ACCEPT all -- docker0 docker0
0.0.0.0/0 0.0.0.0/0
ACCEPT all -- * docker_gwbridge 0.0.0.0/0
0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- docker_gwbridge !docker_gwbridge
0.0.0.0/0 0.0.0.0/0
DROP all -- docker_gwbridge docker_gwbridge
0.0.0.0/0 0.0.0.0/0
We can see here that all traffic passes the DOCKER-USER chain to
allow custom rules. Afterwards all traffic passes the DOCKER-INGRESS
chain which applies ACCEPT rules for NEW connections
as well as RELATED,ESTABLISHED connections for back traffic to
registered task ports utilizing the routing-mesh.
Sample DOCKER-INGRESS chain:
ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:10002
ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED tcp spt:10002
RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 are there for
docker network isolation. And finally the DOCKER chain applies ACCEPT
rules for registered task ports that are running locally.
Sample DOCKER chain:
ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2
tcp dpt:10001
Possible solutions:
So for now we are wondering where the best place for integrating
Shorewall could be. One thing to keep in mind is that rules in
DOCKER/DOCKER-INGRESS chain get applied dynamically on scheduling of tasks.
1) As a first approach we decided to simply remove the jump rules to
DOCKER/DOCKER-INGRESS chains from the FORWARD chain, as those rules
pretty much seem to be what we do want to set explicitly, and doing so
allows package flow to directly enter the Shorewall logic. As those
rules are created explicitly (see
https://sourceforge.net/u/nikm/shorewall/ci/master/tree/Shorewall/Perl/Shorewall/Misc.pm[681-ff])
in the Shorewall integration for Docker, this is also easy to do. (We
saw that in
https://sourceforge.net/u/nikm/shorewall/ci/master/tree/Shorewall/Perl/Shorewall/Chains.pm[8720-ff]
there is some sort of export logic for forward rules which isn't used in
the end. Is there something missing?)
But there might also be a scenario where we add a custom rule to the
DOCKER-USER chain that would allow hooking in with the Shorewall logic.
Any thoughts are greatly appreciated.
Thanks and best regards,
Niko
_______________________________________________
Shorewall-users mailing list
shorewall-us...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users
_______________________________________________
Shorewall-users mailing list
shorewall-us...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users
--
Kouba & Partner
Wien 1, Mölker Bastei 3/4
Niko Mayr
m: +43 650 3240305
e: niko.m...@kouba.at
_______________________________________________
Shorewall-devel mailing list
Shorewall-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-devel
