Forwarded by permission of the author.
Juliusz
--- Begin Message ---
Greetings! Thank you for designing and implementing Babel. In my opinion, it's
head-and-shoulders above everything else in this space.
I have some questions that I hope you can help with, but I wasn't sure what the
most appropriate forum would be, so I'm writing you directly. If you'd like me
to re-route this somewhere else, I'll happily do so.
I'm prototyping a hardware + software stack for metro-scale use. The plan
largely follows a traditional WISP model of centrally-managed tower sites and
centrally-controlled client hardware. Towers consist of sector antennas to
serve clients, plus dishes to backhaul to other towers. Each antenna is plugged
into a different device, each running on different (fixed) frequencies, all
plugged into a regular Ethernet switch, all running Babel on Linux. Clients get
radios with strongly directional antennas, also running babel, exposing normal
TCP/IP over Ethernet to the client's systems.
This deployment model, plus a proprietary TDMA-ish option provided by the
hardware vendor, strongly suggests running in managed/infra modes rather than
ad-hoc. babeld works fine in this scenario, but Managed's default bridging
semantics means Babel doesn't quite make correct decisions: two clients
attached to the same tower sector believe they're directly connected to each
other, when in fact packets must go up to the tower and again back down.
I think I want to enable client isolation (defeating the implicit bridge) and
disable ICMP redirects on that interface to make it two hops and thus properly
reflect reality. Do others run Babel in this configuration?
Another vendor extension allows me to retrieve a connection quality metric on a
per-node basis, combining factors like usable transmit rates and physical layer
retransmit probability into a percentage-of-theoretical-maximum link quality
value. I want babeld to be aware of this, rather than just user-visible packet
loss: two links can be perfect according to ETX, while one of them is noisy and
hidden by the wireless stack, and the other is entirely clear.
Can you offer any thoughts on how to best include this value in the computed
metrics? (I was thinking neighbour_cost() would return the ETX-computed cost +
(100% - quality) * network->cost.) Also, would you be interested in merging
this functionality upstream, perhaps as a compile-time option?
Infra/managed would also let babeld quickly react to disassociation events
substantially faster than waiting for missed hellos, and should improve
convergence speed in the not-uncommon case of a station switching from one AP
to another. For example, IEEE80211_IOCTL_STA_INFO (madwifi `80211stats -a`)
returns a list of all associated MAC addresses. Called periodically, the
results could be converted to modified EUI-64 and cross-referenced with the
neighbor list, updating a timestamp; any neighbor with a nonzero timestamp in
the past could then be flushed immediately. I don't see any downsides to
turning that on globally, but again, your thoughts and interest level in a
patch are welcome.
Unrelated, I think I've seen some sort of bug in one of my testing
environments. The topology in question looks like node A <-> node B <-> nodes
[C,D], each babeld advertising a global /128. After power-cycling B, A's
routing table included C & D via B's link-local address, but no route to B's
global address. This lasted several minutes, until I bounced babeld on A, at
which point B's route appeared. This has occurred at least twice, but appears
to be intermittent. Assuming I can reproduce this a third time, what sort of
debugging information should I gather, from which node(s)?
Thanks for everything!
--Will Glynn
--- End Message ---
--- Begin Message ---
Hi Will,
(Your name is familiar, I'm sure we've interacted before, but I'm not
sure under what circumstances.)
> In my opinion, it's head-and-shoulders above everything else in this space.
Thanks for the kind words.
> I have some questions that I hope you can help with, but I wasn't sure
> what the most appropriate forum would be, so I'm writing you
> directly. If you'd like me to re-route this somewhere else, I'll
> happily do so.
I usually prefer Babel-related correspondence to go to the Babel-users
mailing list, where there's a lot of knowledgeable people, notably the
Debian and OpenWRT packagers as well as the students who wrote parts of
babeld. Please let me know if I'm welcome to forward your mail to them.
> two clients attached to the same tower sector believe they're directly
> connected to each other, when in fact packets must go up to the tower
> and again back down.
Hmm... ETX is only suitable when using the 802.11 MAC. With anything
else than 802.11, you'll want to tweak the metric being used.
> I think I want to enable client isolation (defeating the implicit
> bridge) and disable ICMP redirects on that interface to make it two
> hops and thus properly reflect reality.
Alternatively, you could tweak Babel to use higher metrics for other
clients of the same AP. How can you automaticallyt identify other
clients?
The simplest would be to identify them by router-id:
in id whatever metric 256
although this would require as many entries in your config file as there
are clients.
> Another vendor extension allows me to retrieve a connection quality
> metric on a per-node basis, combining factors like usable transmit
> rates and physical layer retransmit probability into
> a percentage-of-theoretical-maximum link quality value.
Yep.
> Can you offer any thoughts on how to best include this value in the
> computed metrics? (I was thinking neighbour_cost() would return the
> ETX-computed cost + (100% - quality) * network->cost.)
Hmm... I'd give up on ETX altogether. I'd simply use the 2-3 metric, as
with wired links, and multiply it by a smoothed value of the link-layer
quality.
Note that you're most likely going to want to smooth the link-layer
value -- link layers tend to give extremely unstable values.
Note further that you do want to preserve the behaviour of the 2-3
metric, since it will allow you to discard associations that work at the
link layer but not at the network layer (e.g. due to a mis-configured
firewall).
> Also, would you be interested in merging this functionality upstream,
> perhaps as a compile-time option?
No promises -- but probably yes. Please make sure that you implement
your code as a new kind of interface (in addition to wired and wireless).
> Infra/managed would also let babeld quickly react to disassociation
> events substantially faster than waiting for missed hellos, and should
> improve convergence speed in the not-uncommon case of a station
> switching from one AP to another.
Flushing on disassociation is not a good idea, since I've found it
pretty common for a station to disassociate and then reassociate
straight away -- you don't want to reacquire all routes when that
happens. Increasing the link cost upon disassociation to a very large
value might be more useful.
> Unrelated, I think I've seen some sort of bug in one of my testing
> environments. The topology in question looks like node A <-> node
> B <-> nodes [C,D], each babeld advertising a global /128. After
> power-cycling B, A's routing table included C & D via B's link-local
> address, but no route to B's global address. This lasted several
> minutes, until I bounced babeld on A, at which point B's route
> appeared.
How many is ``several minutes''? More than 200 seconds? Can you
reproduce that after decreasing SOURCE_GC_TIME in source.h? (Note that
decreasing this value below 30 seconds or so will most likely cause
routing loops.)
Juliusz
--- End Message ---
--- Begin Message ---
On Oct 18, 2010, at 1:03 PM, Juliusz Chroboczek wrote:
> Hi Will,
>
> (Your name is familiar, I'm sure we've interacted before, but I'm not
> sure under what circumstances.)
Hmm, I don't think so, though I have been following most of your projects since
I first found one of them ages ago.
> I usually prefer Babel-related correspondence to go to the Babel-users
> mailing list, where there's a lot of knowledgeable people, notably the
> Debian and OpenWRT packagers as well as the students who wrote parts of
> babeld. Please let me know if I'm welcome to forward your mail to them.
Ah, okay. There's nothing secretive here; would you like me to re-send the
initial message there, so you can re-reply, and thus reconstruct both threading
and authorship?
>> two clients attached to the same tower sector believe they're directly
>> connected to each other, when in fact packets must go up to the tower
>> and again back down.
>
> Hmm... ETX is only suitable when using the 802.11 MAC. With anything
> else than 802.11, you'll want to tweak the metric being used.
Well, I'm intending to use 802.11, but in infrastructure mode (AP/station)
rather than ad-hoc. (And with overhyped but valuable pseudo-TDMA stuff built
into the wireless driver, all running on top of a standard 802.11n PHY.) Packet
loss happens, and with TCP/IP traffic, should be avoided -- but like city
driving, sometimes the best route is congested and of questionable quality, and
sometimes it's the only route to your destination.
>> I think I want to enable client isolation (defeating the implicit
>> bridge) and disable ICMP redirects on that interface to make it two
>> hops and thus properly reflect reality.
>
> Alternatively, you could tweak Babel to use higher metrics for other
> clients of the same AP. How can you automaticallyt identify other
> clients?
>
> The simplest would be to identify them by router-id:
>
> in id whatever metric 256
>
> although this would require as many entries in your config file as there
> are clients.
Hmm... I suppose I could identify other clients on the same broadcast domain by
comparing router-ids to the currently-associated IBSS. If they match, default
metric, otherwise 2x default. On the other hand, the various tweaks I'm
investigating would probably all work best if the various stations on same AP
weren't aware of each other, since e.g. stations don't know each others'
connection qualities, while the AP connecting them all does.
I'll have to measure the CPU impact of routing versus bridging, but it's all
software forwarding anyway (on my platform...), so I expect it'll be acceptable.
>> Can you offer any thoughts on how to best include this value in the
>> computed metrics? (I was thinking neighbour_cost() would return the
>> ETX-computed cost + (100% - quality) * network->cost.)
>
> Hmm... I'd give up on ETX altogether. I'd simply use the 2-3 metric, as
> with wired links, and multiply it by a smoothed value of the link-layer
> quality.
>
> Note that you're most likely going to want to smooth the link-layer
> value -- link layers tend to give extremely unstable values.
Good point. I think I also need it to be aware of traffic levels -- the 802.11
rate algorithms I've tried seem to make bad decisions when the link is
near-idle, causing retransmissions that count against the quality metric but
generally disappear under load. 80% means nothing if you've only transmitted 5
packets.
> Note further that you do want to preserve the behaviour of the 2-3
> metric, since it will allow you to discard associations that work at the
> link layer but not at the network layer (e.g. due to a mis-configured
> firewall).
Agreed.
>> Also, would you be interested in merging this functionality upstream,
>> perhaps as a compile-time option?
>
> No promises -- but probably yes. Please make sure that you implement
> your code as a new kind of interface (in addition to wired and wireless).
Hmm... I was thinking it'd be a sub-case of wireless, automatically enabling
this functionality if the underlying platform and configuration supports it.
Like:
#ifdef CRAZY_WIRELESS_IOCTL
if (ioctl(...)) {
/* we're an access point, not in ad-hoc mode */
if (ioctl(...)) {
/* we can probe link quality, hooray! */
/* use link quality related metric */
}
}
#endif
if (metric_isnt_set_yet) {
/* use default mechanics */
}
And yeah, I'm not looking for a commitment to merge my work into trunk, just a
general feeling of if I should try. I code better when I'm planning to submit
it somewhere, anyway.
>> Infra/managed would also let babeld quickly react to disassociation
>> events substantially faster than waiting for missed hellos, and should
>> improve convergence speed in the not-uncommon case of a station
>> switching from one AP to another.
>
> Flushing on disassociation is not a good idea, since I've found it
> pretty common for a station to disassociate and then reassociate
> straight away -- you don't want to reacquire all routes when that
> happens. Increasing the link cost upon disassociation to a very large
> value might be more useful.
I haven't observed that on this particular hardware, but then again, I haven't
been looking too closely. Bumping the cost is a better idea in any case. I'm
glad I asked first :-)
>> Unrelated, I think I've seen some sort of bug in one of my testing
>> environments. The topology in question looks like node A <-> node
>> B <-> nodes [C,D], each babeld advertising a global /128. After
>> power-cycling B, A's routing table included C & D via B's link-local
>> address, but no route to B's global address. This lasted several
>> minutes, until I bounced babeld on A, at which point B's route
>> appeared.
>
> How many is ``several minutes''? More than 200 seconds? Can you
> reproduce that after decreasing SOURCE_GC_TIME in source.h? (Note that
> decreasing this value below 30 seconds or so will most likely cause
> routing loops.)
You're exactly right. I just saw this again a couple hours ago, saw 204 seconds
of missed pings. A-ha! So... this would be because that node's sequence number
is reset between boots, while its router-id remains unchanged.
Unfortunately, I don't have a classically writeable filesystem. I could
serialize the babel-state file into nonvolatile storage myself every shutdown,
but that would require mixing runtime with configuration data. I'm not sure how
many write cycles I get, and rewriting the board's config all the time
generally makes me nervous.
A booting node's neighbors likely know its last-boot sequence number -- would
it make sense to solicit it from one of them, as an alternative to a)
persisting it locally or b) picking new random values? It seems like a failure
of any type in that process could fall back to the current behavior (pick new
numbers and deal with incomplete reachability until SOURCE_GC_TIME elapses),
while success would allow a node to pick up where it left off without delay and
without local persistence.
Thanks,
--Will Glynn
--- End Message ---
_______________________________________________
Babel-users mailing list
[email protected]
http://lists.alioth.debian.org/mailman/listinfo/babel-users
