Let's move this discussion to the Development list -- it is not appropriate for the users list (I've added shorewall-devel to the To: list).
My hotel's internet access is too poor to look at your tree tonight. I was able to see, however, that you have re-introduced the WIDE_TC_MARKS option to the default shorewall.conf. Don't do that - that option has been superseded for some time. Note: And when an option is added to the default shorewall.conf, it must also be added to the Sample .confs and all of the Shorewall6 .confs, and to the shorewall.conf manpage. Otherwise, the Shorewall build process fails. I believe that: 1) The order in which Docker and Shorewall are started should not matter. 2) Either product should be able to be started, stopped, restarted or reloaded without effecting the other product. Those two precepts led to the design of the current Docker support in Shorewall. When Shorewall executes a 'start', 'stop', or 'reload' ('restart' is a 'stop' followed by a 'start'), it saves all Docker-related chains and rules, then integrates them into the new configuration. I assume the 'save' part of that algorithm is what the code in Chains.pm that you refer to below is doing (Note that Shorewall atomically replaces the entire Netfilter configuration during each of these operations). The problem with that approach is that when Docker adds something new, that new content doesn't get saved/restored by Shorewall. That fundamental weakness in the current design caused me to consider dropping Docker support entirely in Shorewall 5.2. The alternative, which I settled on, is to continue to add new logic to Shorewall each time that Docker adds new chains (and jumps to those chains). And that new logic must detect whether the new Docker feature is in use or not, and must not change the generated ruleset unless the new feature is present. If I ran Docker (which I don't - I run LXC on my Shorewall system), I would not run Shorewall on the same system. I would rather front-end the Docker system with a box running Shorewall. Note that since Docker can run in a VM, you can still run Docker and Shorewall on the same physical system by running Shorewall in the host and Docker in a VM. -Tom On 3/5/19 2:20 AM, Niko Mayr wrote: > > Hi! > > As mentioned in a previous post > (https://sourceforge.net/p/shorewall/mailman/message/36599490/) we are > experimenting with Docker 18.09 in swarm mode and Shorewall 5.2.3, and the > current integration doesn't work. > > > We decided to try to quickly implement a patch, which can be found here > https://sourceforge.net/u/nikm/shorewall/ci/master/tree/. We tried to > extend the existing integration to reflect the current chain and > interface configuration utilized in Docker in swarm mode. In general it > would be nice if we could get some feedback on that sometime to be able > to prepare a merge request if wanted, but there is one thing that would > be especially interesting. Docker 18.09 in swarm mode applies some sort > of firewall strategy and it would be interesting where you'd see the > ideal Shorewall integration. > > > Docker iptables usage: > > > 1) In the nat table Docker adds the two chains DOCKER and > DOCKER-INGRESS. Jump rules added to the PREROUTING chain direct to those > chains on dst-type LOCAL. > > PREROUTING chain: > > DOCKER-INGRESS all -- * * 0.0.0.0/0 > 0.0.0.0/0 ADDRTYPE match dst-type LOCAL > DOCKER all -- * * 0.0.0.0/0 > 0.0.0.0/0 ADDRTYPE match dst-type LOCAL > > In those chains Docker dynamically adds DNAT rules to direct the > registered task ports to the concerned interface. Which chain it uses > depends on the utilization of the routing-mesh. > > > Sample DOCKER chain: > > RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0 > RETURN all -- docker_gwbridge * 0.0.0.0/0 0.0.0.0/0 > DNAT tcp -- !docker0 * 0.0.0.0/0 > 0.0.0.0/0 tcp dpt:10001 to:172.17.0.2:443 > > > Sample DOCKER-INGRESS chain: > > DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 > tcp dpt:10002 to:172.18.0.2:10002 > RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 > > > 2) The DNAT rules above don't direct to local interfaces directly > (docker0: inet 172.17.0.1/16, docker_gwbridge: inet 172.18.0.1/16) so > that takes us to the FORWARD chain in the filter table. > > In the filter table Docker adds five chains DOCKER-USER, > DOCKER-ISOLATION-STAGE-1, DOCKER-ISOLATION-STAGE-2, DOCKER, > DOCKER-INGRESS. Jump rules in the FORWARD chain distribute > > traffic to those. > > > FORWARD chain: > > DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0 > DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0 > DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 > 0.0.0.0/0 > ACCEPT all -- * docker0 > 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED > DOCKER all -- * docker0 > 0.0.0.0/0 0.0.0.0/0 > ACCEPT all -- docker0 !docker0 > 0.0.0.0/0 0.0.0.0/0 > ACCEPT all -- docker0 docker0 > 0.0.0.0/0 0.0.0.0/0 > ACCEPT all -- * docker_gwbridge 0.0.0.0/0 > 0.0.0.0/0 ctstate RELATED,ESTABLISHED > DOCKER all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0 > ACCEPT all -- docker_gwbridge !docker_gwbridge > 0.0.0.0/0 0.0.0.0/0 > DROP all -- docker_gwbridge docker_gwbridge > 0.0.0.0/0 0.0.0.0/0 > > > We can see here that all traffic passes the DOCKER-USER chain to > allow custom rules. Afterwards all traffic passes the DOCKER-INGRESS > chain which applies ACCEPT rules for NEW connections > > as well as RELATED,ESTABLISHED connections for back traffic to > registered task ports utilizing the routing-mesh. > > > Sample DOCKER-INGRESS chain: > > ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 > tcp dpt:10002 > ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED tcp spt:10002 > RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 > > > DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 are there for > docker network isolation. And finally the DOCKER chain applies ACCEPT > rules for registered task ports that are running locally. > > > Sample DOCKER chain: > > ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 > tcp dpt:10001 > > > Possible solutions: > > So for now we are wondering where the best place for integrating > Shorewall could be. One thing to keep in mind is that rules in > DOCKER/DOCKER-INGRESS chain get applied dynamically on scheduling of tasks. > > > 1) As a first approach we decided to simply remove the jump rules to > DOCKER/DOCKER-INGRESS chains from the FORWARD chain, as those rules > pretty much seem to be what we do want to set explicitly, and doing so > allows package flow to directly enter the Shorewall logic. As those > rules are created explicitly (see > https://sourceforge.net/u/nikm/shorewall/ci/master/tree/Shorewall/Perl/Shorewall/Misc.pm[681-ff]) > in the Shorewall integration for Docker, this is also easy to do. (We > saw that in > https://sourceforge.net/u/nikm/shorewall/ci/master/tree/Shorewall/Perl/Shorewall/Chains.pm[8720-ff] > there is some sort of export logic for forward rules which isn't used in > the end. Is there something missing?) > > > But there might also be a scenario where we add a custom rule to the > DOCKER-USER chain that would allow hooking in with the Shorewall logic. > Any thoughts are greatly appreciated. > > > Thanks and best regards, > > Niko > > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users -- Tom Eastep \ Q: What do you get when you cross a mobster with Shoreline, \ an international standard? Washington, USA \ A: Someone who makes you an offer you can't http://shorewall.org \ understand \_______________________________________________
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users