Hi Daniel - Thanks for providing feedback. I have some comments (and additional questions) based on your responses below.
On Tue, Nov 5, 2013 at 12:15 PM, Daniel Wagner <[email protected]> wrote: > Hi Glenn, > > > On 11/04/2013 08:37 PM, Glenn Schmottlach wrote: >> >> I have a proposal which I hope the Connman developers will consider. > > > Sure :) > > >> As I understand it, Connman sessions are tracked, in part, by marking >> packets associated with the session's UID, GID, or SELinux context >> information. This is translated into iptable rules to which "mark" the >> connections as described in session-overview.txt. >> >> Per session iptables rules: >> >> iptables -t mangle -A OUTPUT -m owner [--uid-owner|--gid-owner] $OWNER \ >> -j MARK --set-mark $MARK >> >> iptables -t filter -A INPUT -m mark --mark $MARK \ >> -m nfacct --nfacct-name session-input-$MARK >> iptables -t filter -A OUTPUT -m mark --mark $MARK \ >> -m nfacct --nfacct-name session-output-$MARK > > > BTW, the nfacct is going away. We are going to use NFQUEUE in future. > Though we still need the MARK unless we can convince the netdev > guys that the lookup for a policy routing table could be something else > e.g. cgroup id. For the time being I keep the assumption we need > the marker. > I'm curious how you anticipate using NFQUEUE in the future unless you're considering re-directing the network traffic to a user-space application (like Connman). Are you trying to do a transparent proxy or intercept the packets in some way? If so, have you considered TPROXY or even using NFLOG to get at the these packets? My only concern/confusion here is the intent for using NFQUEUE since it seems you want to funnel matching packets into user-space. Can you elaborate on your plans? Wouldn't this have a negative impact on the performance of the connections managed by Connman if you are moving in that direction? > >> These rules instruct iptables to collect basic usage metrics >> (bytes/packets sent/received) on behalf of Connman. >> >> I have a requirement to do additional filtering and statistics >> collection using my own set of rules and would like to filter on these >> same MARK'ed packets while adding additional "marks" (or fields) to >> the packets so that I can separate them into additional categories >> needed by my application. This is where I need some additional >> information from Connman and a slight modification to how packets are >> MARK'ed and compared. >> >> As I understand it, each netfilter packet has an associated 32-bit >> mark value which is being set to $MARK in the above rules. When >> setting a MARK value you can provide a mask which will only allow >> specific bits to be set. Looking at the session.c code (line 39) I see >> that new sessions are initialized starting at 256 and are incremented >> upwards from there. So my assumption is that Connman assumes it "owns" >> the upper 24-bits of the netfiler mark value. Likewise, it would seem >> from the iptables rules that it does not expect other rules in the >> iptable to use these same bits for marking packets. > > > ConnMan starts counting at 256 because that is the first 'unmanaged' > ID for a policy routing table. To simplify I used the same value > for the routing table as for the marker. Obviously the current > implementation doesn't really play nicely by assuming it has complete > control. I have no problem to change that. > > >> What I am proposing is the following: >> >> 1) Modify Connman to provide a mechanism to retrieve the iptable >> "MARK" value associated with a session as well as the bitmask used to >> filter/extract those bits from the underlying U32 netfilter value. >> Perhaps both the session MARK value and bitmask could be considered >> read-only "settings" of the session object and would be updated when >> the iptable rule is written. > > > I don't really follow here. Could you please elaborate it a bit? Let me try to explain what I'm trying to accomplish. On my target platform there exist applications that run as separate processes. My intention is to run each "app" as a separate Linux user. These apps might still be further divided into groups with similar features (e.g. streaming apps, music apps, etc...). My thought was to assign each app to a Linux group, or as you indicate, perhaps a specific cgroup if I utilize that Linux feature. I need to enforce both a per-application network data quota (bytes sent + received < data limit) as well as an overall application "group" data limit. My thought is that I could create a Connman policy (or session configuration) for *each* application. Connman seems to already provide the mechanism to provide a prioritized Bearer list and the necessary indications to tell an application whether the underlying Bearer/service is connected (or not) and to request a connection through the Session API. It seems that when a Session is created new rules are added to the iptable to "mark" these packets from a particular application based on the UID, GID, or SELinux context. I want to set up additional filters on these same marked packets except I also want to add a field to mark a GID association in the same U32 netfilter value. I need to know the netfilter mask I can use to mask off the fields Connman uses for each packet. My thought was to do this outside of Connman since it's not clear this is something everyone needs/desires in their architectures. Also, I want to use netfilter to collect metrics for a combined send/receive byte count since currently Connman tracks sent/received bytes separately. So in my solution, I add additional rules for each app (using the Connman "mark" value) to collect a send/receive application statistic. I also use my own netfilter "mark" value to track the application's Linux "group" so I can do group statistics and impose a quota at that level as well (which could override the quota at the application level). I can do all of this directly with iptables (I've prototyped it by hand) but need Connman to provide me with the per-session/application "mark" value and the filter I can used to mask off that value for my own iptable rules. Retrieving this information as a read-only Session property seemed the most convenient way. I'm open to other suggestions however. > >> 2) Provide a configuration option (either at compile-time or via >> configuration file item), that specifies the offset/mask for the >> session's MARK value. For instance, currently the inferred offset (by >> examining the code) is 8 bits with 0xFFFFFFFF as the default mask. >> This doesn't allow the Connman markings to co-exist with any other >> external markings. For my application I would like to reserve the >> upper 16 bits for the Connman session value and the lower 16 bits for >> my own use. The Connman code that generates the iptables rule might be >> modified to look like this: >> >> iptables -t mangle -A OUTPUT -m owner [--uid-owner|--gid-owner] $OWNER \ >> -j MARK --set-mark $MARK/0x0000FFFF >> >> iptables -t filter -A INPUT -m mark --mark $MARK/0xFFFF0000 \ >> -m nfacct --nfacct-name session-input-$MARK >> iptables -t filter -A OUTPUT -m mark --mark $MARK/0xFFFF0000 \ >> -m nfacct --nfacct-name session-output-$MARK > > > That sounds reasonable to me. I don't think we need 2^24 ids, we > should be happy with 2^16 applications :) > I thought so too . . . I can't imagine (at least for the near-term) having more the 2^16 apps installed. In the process of testing how Sessions work in Connman I believe I've stumbled upon either a bug in the Sessions implementation or an intended (purposeful) behavior. I rather hope it's a bug since otherwise it will force me to put a "shim" in front of the Connman Session API. What I have observed is that for *every* session that is created, a new set of iptable rules is created . . . even if the same user/group/SELinux context is used to create the session. For instance, if I have two applications (both the same user - with a session policy keyed to the UID) and they create a Session instance then in the iptable's "mangle" table (OUTPUT) looks something like this: Chain connman-OUTPUT (1 references) pkts bytes target prot opt in out source destination 20 1134 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1000 MARK set 0x100 20 1134 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1000 MARK set 0x101 So it would appear the *last* session that is created will effectively mark (and clobber) the first marking for the packet (0x101 mark clobbers the earlier 0x100 mark since it's last in the chain). I hope this is *not* what was intended. I expected that iptable rules for Session policies for the same UID/GID/SELinux context would "share" the same session (and thus statistics). So if two applications running as the same UID/GID/SELinux context requested a Connman Session, the first requestor would create it while the second application would get a "reference" to that session. This session would remain valid until the last application (with the same UID/GID/SELinux context) destroyed the session. In a sense the Sessions become reference counted per session policy. As it stands today it would appear Connman creates far more iptable rules than necessary to support multiple applications running as the same user (or in the same group or SELinux context). Am I correct in assuming this is a design error in the current implementation of Sessions? > >> As you can see, setting the Connman MARK is no longer destructive to a >> value that might already exist in the lower 16-bits of the 32-bit >> word. Likewise, when reading and comparing the value the lower 16-bits >> of the 32-bits value are effectively masked out so it won't impact the >> Connman session match test. >> >> Part of my problem is that I need to create additional iptable rules >> that place data-caps (e.g. a quota) on certain sessions (apps) which >> are then optionally partitioned into additional groups. I can use the >> power/flexibility of iptables to implement this logic but I need to >> cooperatively use the same netfilter 32-bit value to store additional >> information about the connection and their associated packets. This >> change would help Connman behave as a better netfilter citizen while >> allowing me to piggy-back on top of the existing marking done by >> Connman. > > > Well, the quota thing is something we want also to support natively > with sessions. That is also why count the packets (currently with > nfacct but we don't use that information yet). > I'm sure you realize that you don't explicitly need nfacct to collect statistics. Granted, nfacct gives you nicely named buckets you can access with the nfacct utility and likely over netlink as well. But each iptable's rule has a byte/packet count associated with it. So if a packet merely passes through a rule (a simple "do nothing" rule) it will increment that rule's counters. Plus, iptables gives give you a way to query/reset the stats for a specific rule if I'm not mistaken. > >> I'm willing to make the necessary code changes to implement this >> solution. I'm just seeking feedback of whether a better solution >> exists (that I'm unaware of) or perhaps recommendations on my >> approach. Of course any patches I make would be available to the >> community and hopefully could eventually be integrated into the main >> baseline. > > > I am happy to see patches! > > cheers, > daniel I'm really interested in the direction you're headed with Connman Sessions. I'd like to leverage this work and perhaps contribute the project's generic parts for the solutions I devise. I realize not everyone has similar needs but it would appear (at least so far) Connman's goals are fairly closely aligned with mine. Thanks, Glenn _______________________________________________ connman mailing list [email protected] https://lists.connman.net/mailman/listinfo/connman
