Re: Netflix hates IPv6

[Phil Mayers]

On 12/06/2016 22:02, Jens Link wrote:


[1] Those are also the people putting "copying DVDs is illegal" videos
on DVDs which you are forced to watch using a normal TV/DVD player
combination. People who are just copying the DVD will leave out such
trailer.


You wouldn't steal a BABY!

https://www.youtube.com/watch?v=ALZZx1xmAzg for the uninitiated.

Re: v6 naming and shaming - *.europa.eu

[Phil Mayers]

On 18/05/16 15:32, Tim Chown wrote:


The flip side is what evidence do we have that its a problem that is
common enough to care about?


This is a fair point. Perhaps I'm overreacting - we don't get too many 
of these.

Re: v6 naming and shaming - *.europa.eu

[Phil Mayers]

On 18/05/16 15:03, Jeroen Massar wrote:


The best advice for getting IPv6 fixed is for a large well used network
(google, facebook) to stop providing IPv4. Then suddenly people will fix
things as they won't have working "Internet" and their users will
complain really really loud.


Ok so basically, if more/most access networks were IPv6-enabled (because 
big or vital providers are IPv6 only) then all service networks would 
have to get it working?


Not unreasonable, but that's a very long term prospect I guess.

I'd be curious to know if people have suggestions that work shorter term.

I'm in agreement that shaming is not effective; but I'm frustrated and 
it just seemed so ironic that their public claims were so pro-v6.


Question for any access network providers: if/when you run into these 
issues, how do you plan to proceed? Leave the site broken and force the 
site owner to fix, or work around at your end and hide the problem?


No judgement either way, just curious.

Regards,
Phil

Re: v6 naming and shaming - *.europa.eu

[Phil Mayers]

On 18/05/16 14:45, Matthew Ford wrote:


Many moons ago, europa.eu IPv6 ‘service’ was a reverse-proxy operated
by BT. I have no idea what the current kludge is.


Ah, BT. The obvious choice of provider for an IPv6 implementation /sarcasm

Whoever runs it, they've broken it a bunch of times before.

I've "fixed" it at our end on this and previous occasions using bind RPZ 
to convert  replies containing their /48 to NODATA.


This makes me feel dirty :o(

Re: v6 naming and shaming - *.europa.eu

[Phil Mayers]

On 18/05/16 14:29, Jeroen Massar wrote:


Really, you cannot keep on telling people to finally deploy IPv6, it
does not have any effect whatsoever, only their pocket books care and
those will only notice when it is too late...


So it's hopeless and we should just give up?

That doesn't seem like the most encouraging advice ever, but thanks for 
the reply.


Anyone else got thoughts on how to discourage half-working/half-broken 
setups which create negative externalities?


I'm specifically not asking about encouraging people who haven't 
deployed; rather people who have and who have broken or abandoned their 
efforts.

v6 naming and shaming - *.europa.eu

[Phil Mayers]

Broken over IPv6:

https://webcast.ec.europa.eu/281715cafa675bf359ebaa42cb44fa17

(Webserver has , returns 404 over v6, fine over v4)

And yet:

https://ec.europa.eu/digital-single-market/en/blog/ipv6-more-than-a-reality-a-necessity

I'm sick and tired of people doing tickbox IPv6 and then well-run 
networks getting the blowback: "It works on my 4G and home ADSL, it must 
be your network".


I really, really, really wish there was some incentive to do it right or 
not at all.


So, for discussion - what can the operational community do to discourage 
half-measures that create blowback / moral hazard?


Cheers,
Phil

Re: push apps failing in Android until you disable IPv6

[Phil Mayers]

On 10/05/16 14:10, JORDI PALET MARTINEZ wrote:

Understood, thanks !

I just read all the Doze thing :-) I also recall something published
by Lorenzo about power saving in IPv6, etc., however, I still fail to
see if there is no GUA, why Android is affected using only IPv4.


Well, it is probably a bug. Just could be a bug in a lot of places - 
user, kernel, wifi driver, wifi firmware, even wifi hardware. Some of 
those are Google-supplied (but possibly manufacturer modified?) but 
others are IHV-supplied IIUC.


The fact that IPv6 pulls in a lot more multicast and thus potential wifi 
complexity might be related.


It might be interesting to know whether the issue is a failure of e.g. 
WMM, so the notify packets aren't being queued to / reaching the 
handset, or whether they're being send over the air, but the handset 
isn't waking up. Could possibly determine that with a wireshark aircap 
on an affected CPE/handset pair if you know the WPA-PSK.


Does the behaviour change if the device is on charge?

Re: push apps failing in Android until you disable IPv6

[Phil Mayers]

On 10/05/16 13:57, JORDI PALET MARTINEZ wrote:

Hi Phil,

Not sure if you have seen the previous message with the rdisc6. Your
network may be not having a “broken” CPE.


I did.

You'd asked:

"""
Right, but how this is affecting IPv4 push notifications ?
"""

I was trying to convey that the wakeup system is complex, and that the 
broken v6 might be causing the entire wakeup system to fail, including 
the v4 bit, and to suggest some (wild guess) reasons as to the cause.

Re: Curious situation - not urgent, but I'd like to know more

[Phil Mayers]

On 23/12/15 10:54, Seth Mos wrote:


We use OpenVPN on pfSense with Viscosity on the clients, or the Android
OpenVPN app. It is a complete Dual-Stack solution for both the servers
and the clients, and because we push more specific IPv6 routes it takes


What happens if the client has no local, non-VPN IPv6 traffic? Doesn't 
it break things, because even though you're pushing more-specifics, the 
device now thinks it has a global IPv6 address and breaks address selection?

Re: [dns-operations] DNSSec and GoDaddy and IPv6 (cross-posted)

[Phil Mayers]

On 08/12/15 07:53, Nico CARTRON wrote:

On 08 Dec 2015, at 01:26, David Conrad  wrote:



On Dec 7, 2015, at 3:50 PM, Frank Bulk  wrote:
Anyone know of a registrar that supports both IPv6 and DNSsec?


https://www.gkg.net


GANDI (http://www.gandi.net) also does.


+1 GANDI are excellent.

Re: Looking for information on IGP choice in dual-stack networks

[Phil Mayers]

On 05/06/2015 11:00, Tore Anderson wrote:

* Philip Matthews philip_matth...@magma.ca


We are looking particularly at combinations of the following IGPs:
IS-IS, OSPFv2, OSPFv3, EIGRP.


We're using OSPFv2 and OSPFv3 as ships in the night for IPv4 and IPv6,



We do the same, FWIW. Not large numbers - 27 OSPFv2 and 25 OSPFv3 
routers, mix of IOS and JunOS. Works fine, without any real caveats. Bit 
more typing with two protocols, but meh, not significantly.

Re: IPv6 QUIC traffic

[Phil Mayers]

On 04/06/2015 18:51, Ca By wrote:


UDP 80 and 443 are very commonly associated with DDoS in my experience.
I think it is common used as a reflection source port.


Sadly true. We see this a lot.

Not saying I agree with blocking it, but UDP 80/443 is deeply suspicious 
traffic in my experience.

Re: Google no longer returning AAAA records?

[Phil Mayers]

On 15/04/15 16:05, Brian Rak wrote:

We noticed that we're no longer getting  results back for google.com
when we do queries from a few of our recursive servers (other ones are
fine).

A bit of searching revealed that a few of our servers are listed here
http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_.txt
(there are 4 listed for AS20473, which are the ones I'm referring to)

However, I can't seem to find any information about why we'd be listed
there, nor can I find anything telling me how to get delisted.


Yes. They don't provide that info, nor do they provide a way to request 
a de-listing AFAIK.


You'll be removed automatically when the problem goes away.

There's a lot of discussion in the archives about this, but I believe 
that, as far as is known publicly:


 1. There's a web-bug in the google search page
 2. They correlate IPv6 failures of this against the DNS lookup
 3. When the DNS source IP hits a certain threshold of correlatd 
failures, they stop serving  records for a time (one week?).


Subjectively it's irritating that Google don't provide info to operators 
as to the specifics of the cause, but look at it from their PoV - 
there's a lot of info, some of it potentially personal / data-protected, 
that they'd have to communicate securely to operators when they ask.


It would be a lot of work for them and I'm somewhat sympathetic on that 
basis (although I wasn't when it was happening to us ;o)


My guess: you've got some form of broken IPv6 connectivity talking to 
your resolvers; maybe rogue RAs, a tunnel, VPN, etc.


The customers with this problem aren't reporting it because Happy 
Eyeballs, but the Google web-bug is detecting it.


We saw a reduction (and eventual end to) our experiences of this when we 
finished our native dual-stack deployment *and* when we blacklisted 
serving of  to some of our more troublesome netblocks - mainly 
remote access VPN users.


We monitor whether google are blacklisting us in our Nagios setup, so we 
can see if problems come back.


An alternative might be to steer different classes of users to different 
resolver query source IPs (either actual different resolvers, or using 
views  multiple IPs). Then, you can see which source IP and thus class 
of users is triggering it.


Good luck tracking it; it can be frustrating :o(

Cheers,
Phil

Re: Cost of IPv6 for IT operations team

[Phil Mayers]

On 11/04/15 10:27, Nick Hilliard wrote:


Uh, lemme just drop this in here: http://imgur.com/AYbpRG2


;o)



The problem with stage 4 is that it requires that the expertise garnered by
the initial deployment team is spread throughout the rest of the company,
ranging from product development to the FLS desk, right through to
customers.  This is a prolonged and time-consuming process, and
consequently expensive.  I'd love to see a larger scale discussion about
this, because it's one of the main blockages for ipv6 adoption and
discussion of live cases would help other organisations make the jump.



University environment here, fully dual-stacked everywhere.

Interestingly enough, we've not done a huge amount of training across 
the rest of the IT support function. It was always intended, but it's 
actually surprisingly rare to need to refer to an IP address - v4 or v6 
- when troubleshooting problems. MAC addresses are, generally, far more 
useful, and with v4/v6 ARP/ND tracking, give you the IPs.


We have spread some knowledge into our infrastructure applications 
team - exchange, webservers, etc. - and that's been fine.


We've had a lot less luck spreading the knowledge into the teams which 
deal with COTS application stacks e.g. Oracle. The problem there is the 
upstream support - if Oracle don't certify or deploy on IPv6, the 
people running that stack have no interesting in doing so.


The major thing is don't forget to configure the IPv6 vhost. But by 
dual-stacking everyones desktop, including the developers, that gets 
picked up early.


For supporting client connectivity (desktops, laptops) we've not found 
the protocol differences very relevant. MAC address on wired, 802.1x 
username on wireless or MAC if it's an auth-level problem.


We just haven't found it a problem. Maybe it's a University thing. I'm 
sure consumer ISPs have a much harder time.


Cheers,
Phil

Re: Cost of IPv6 for IT operations team

[Phil Mayers]

On 13/04/15 09:55, Benedikt Stockebrand wrote:


Which is a major effort in some environments because---contrary to what
Nick wrote---pretty much anyone involved needs to be familiarized with
IPv6.  The reason here is that if there is any problem once IPv6 is
enabled anywhere, then *all* people doing the troubleshooting need to
know how to exclude IPv6 from the list of possible problem causes.


See, this makes sense - it's a reasonable assertion. But it's the 
opposite of what we've found.


In all honesty, I doubt a large portion of the organisation understands 
how IPv*4* works. If you think about one of the many areas v6 differs - 
ARP/DHCP versus ND/SLAAC/DHCPv6 - how many people *really* understood 
the former that well?


IME, not many.

So I'm a bit conflicted. What you say makes sense, but contradicts my 
experience. Still, data != sumof(anecdote) and all ;o)



And as soon as you add a round or so of blame game, that tiny number of
people will also have to communicate towards management that no, no
matter what some other people claim, it isn't an IPv6 fault---which is
exceedingly difficult since this situation usually leads to management
turning towards the trouble ticket management system only to discover
that (a) pretty much anytime anything went wrong IPv6 was somehow
involved according to the paper trail and (b) time to fix has
significantly increased due to this additional showstopper step.


OTOH, we often find the network blamed for basically everything. So this 
is only a small extension ;o)



In an enterprise network that's actually a minor issue compared to
e.g. the business applications that need to be fixed up, or the overdue
OS updates that need to be deployed.


+1


While it's really convenient to use product refresh cycles to enable in
some cases, from what I've seen at least in enterprise environments this
frequently doesn't work as a general approach.


What problems have you seen? Insufficiently demanding RFP or lack of 
awareness of the need to put it in an RFP? Or something else?



In other words, just because someone claims some number measured in one
specific environment doesn't mean anything at all to yours.


Very strongly agreed.

At this stage of IT support as a profession - encompassing the 
technology, tools, management structures and so forth - it's clear to me 
there's huge variation, making any comparisons dangerously close to 
apples/oranges.


Cheers,
Phil

Re: Cost of IPv6 for IT operations team

[Phil Mayers]

On 26/03/15 09:04, BERENGUER Christophe wrote:

Hello everybody,


I work for a consulting firm.


For a client, I would like to estimate the work overload for IT
operations team to deploy IPv6 dual stack and for day to day operations.


On the internet, I have found an estimation around 20% of work overload
for the run phase. But if you have operational feedback it would be the
best!


I agree with others that this is a very hard thing to estimate.

I will say that we run our dual-stack network (fully deployed since ca. 
2012) with exactly the same staffing levels, and actually a slight 
reduction in our recurrent budget, as our older IPv4-only network.


I don't think our network is any less reliable, or suffers a higher 
level of incidents. This suggests to me that, in our case, IPv6 has 
added a very low operational cost. Our incidence of IPv6-related 
problems, particularly rogue RA from machines configured for connection 
sharing, has actually *decreased* substantially since we deployed native 
IPv6.


I don't believe the rollout cost was high. We used refresh cycles to 
upgrade to v6-capable gear, and rolled out slowly to grow our team 
knowledge. But we don't have detailed cost breakdowns.

Re: Why do we still need IPv4 when we are migrating to IPv6...

[Phil Mayers]

On 18/02/15 09:29, Anfinsen, Ragnar wrote:


A quick example; A good friend of mine is developing a smart
fireplace which can be controlled via API's. He do use a 3. party
development company to make the controller and API's. They did not
even think of IPv6 until I did my 5 minute speech about the


Don't get me started on SCADA systems.

Based on our own experience here, the companies that make this kind of 
equipment are, by and large, wilfully ignorant - and I choose those 
words after careful consideration - of the most basic aspects of networking.


It is frightening how badly designed some of this equipment is, and 
very, very scary how much of it wants to sit on public IP space to talk 
to the cloud part of the service these days.


At some point in the next decade, I anticipate enterprise IT departments 
doing internal charging for SCADA systems which need public IPv4 space; 
we have seen proposals for 500 SCADA nodes needing public IPs in a 
single building. New systems, new to market, designed by companies 
considered market leaders in the product space.


There is a woeful lack of ability in this bit of the industry. I'd love 
a big player to come in and blow the market sky high.


Cheers,
Phil

Re: Why do we still need IPv4 when we are migrating to IPv6...

[Phil Mayers]

On 13/02/15 11:26, Mikael Abrahamsson wrote:

On Fri, 13 Feb 2015, Thomas Schäfer wrote:


and the practice in Germany to blocking all IPv6-inbound traffic the
result is the problem for some gamers.


So I guess applications should use the same technique as one does to
traverse NAT44:s, ie both ends of the connection send packets to each
other to open their respective firewall.

I do agree that the firewall in question needs to not send rejects for
this traffic for this to work. I am happy this use-case was brought up,
because I hadn't heard and thought about this before. Personally I don't
want to silently drop packets, so I guess clients need to try a few
times and not listen to the (initial) ICMP messages until the hole is
open.


It all depends on the behaviour of the device(s)

It's perfectly possible for a CPE to send ICMP errors without those 
errors creating a NAT table entry and blocking the real (inside) host 
from using that 5-tuple.


In the situation I described yesterday, the CPE is Linux, and it could 
have done something like:


iptables -t raw -A OUTPUT -p icmp -j NOTRACK

Or it could not send errors for unknown UDP flows directed to high ports 
e.g.:


iptables -A INPUT -m state --state RELATED,ESTABLISHED -j PERMIT
iptables -A INPUT -p udp --dport 1024:65545 -j DROP

There's a bunch of different solutions.

None of this should be a problem for non-NATed IPv6. The absence of NAT 
will mean an ICMP error doesn't block a NAT translation - there's no 
such thing to block - so a CPE can send errors or not.


If you're NATing IPv6, well... you brought it on yourself ;o)

Cheers,
Phil

Re: Why do we still need IPv4 when we are migrating to IPv6...

[Phil Mayers]

On 13/02/15 13:27, Mikael Abrahamsson wrote:


Packet reaches HGW2, which has no flow state, and is dropped. ICMP error
message might be created.
In case of ICMP error message, U1 should ignore this.


That's an application-layer issue. It all depends on how they're talking 
to the socket API. They might not even see the ICMP error if they're 
just doing dumb send() calls.



U2 sends a packet from U2IP,U2PORT to U1IP,U1PORT.
HGW2 creates flow state.
Packet hits HGW1 which already has a flow state, and packet successfully
reaches U1.
U1 now can start sending packets to U2 as well and they've worked around
both of them having HGWs with stateful firewalls disallowing new
connections to them.

Right?


Yes.



The crucial step here seems to be the fact that initial packets might be
dropped and error messages be generated, but these should be ignored by
the application. Is this commonplace? Is it a problem at all?


As above, depends on how they're using the socket API. As a rule for UDP 
connections, you actually have to put *more* work in to see ICMP errors. 
It's certainly possible to ignore them.

Re: Why do we still need IPv4 when we are migrating to IPv6...

[Phil Mayers]

On 13/02/15 14:37, Thomas Schäfer wrote:

Why a discussion to drill the firewall with very tricky things?

(it's sound to me like the same sh... stun and other legacy ipv4 horrors.)


In my opinion the firewall should be configurable (unfortunately
DTAG-speedport-series, including the hybrid-modell dsl/lte can't) by
upnp or by the user.


That's fine, and I agree in theory.

But Sony and Microsoft aren't going to just assume or enforce that, and 
I don't blame them. They have to assume some proportion of devices will 
be behind a firewall or NAT, and will write the code accordingly.


Done correctly, it's very little additional burden over just sending 
straight UDP packets. There's really no reason for system/app vendors to 
*not* implement traversal, and it doesn't harm the network.


But you're right, this has gone off-topic. The point was that IPv6 makes 
this situation - person-to-person networking - better than in the NAT44 
world, and would improve e.g. internet gaming.

Re: SV: Why do we still need IPv4 when we are migrating to IPv6...

[Phil Mayers]

On 12/02/15 12:40, erik.tarald...@telenor.com wrote:

This might be so in Norway. In German customer portals the gamers mostly
demand ipv4 (public ipv4 address to their home) instead of DS-Lite. They
have already native IPv6 but avm was forced to allow teredo over DS
and DS-lite - because xbox has problems with native IPv6.

xbox is no good example for *wanting* IPv6.


Could you elaborate on the IPv6 issues for xbox?  I was under the impresion
that xbox works well with IPv6.


The Teredo implementation used for person2person connectivity in Xbox 
One does not have relays. That is, you can't talk Native IPv6 - XB1 Teredo.


The implication is that, unless all parties in an XB1 session have 
native IPv6, all parties will fall back to Teredo-over-IPv4. As such, 
you need working Teredo/IPv4 for XB1 today, as you're very likely to 
need to execute this fallback.


Given that Teredo relays were the unreliable bit, I can't fault this.

The XB1 Teredo stuff is actually quite a reasonable approach.

Cheers,
Phil

Re: Teredo sunset - did it happen?

[Phil Mayers]

On 17 November 2014 17:22:37 GMT+00:00, Michael Chang thenewm...@gmail.com 
wrote:
Presumably because the clients are unmanaged?

Correct. It's already disabled by group policy on our managed base.
-- 
Sent from my mobile device, please excuse brevity and typos

Teredo sunset - did it happen?

[Phil Mayers]

All,

ISTR that Teredo was going to be sunset, Microsoft having tested 
removing the DNS name teredo.ipv6.microsoft.com.


(Ignoring the Xbox One stuff here - just the windows desktop 
server/relay stuff)


However, my Windows 7 machine is still resolving that name and forming a 
Teredo address, and setting an IPv6 default route via that tunnel - I 
can see Teredo-encap'd router-solicit and router-advert messages being 
sent and received.


No traffic flows however - the Teredo direct connect tests are all 
failing (no reply to the ICMPv6 echo). So I've got a broken IPv6 tunnel :o/


Any ideas what's going on? Microsoft, anyone care to comment?

Cheers,
Phil

Re: Teredo sunset - did it happen?

[Phil Mayers]

On 17/11/2014 16:40, Jeroen Massar wrote:

On 2014-11-17 17:38, Phil Mayers wrote:

On 17/11/2014 16:23, Jeroen Massar wrote:


What are you trying to achieve by blocking that port?


I honestly don't know why you want to talk about other things, but I've
no interest in discussing them with you.


Then don't make statements that you are blocking them...


In the interests of the principle of charity - I am trying hard to 
assume good motives on your part - let me try again...


===

We've historically blocked Teredo, for probably erroneous reasons. I'd 
like to unblock it, to specifically let XBox One consoles use their new 
Teredo stuff.


At the same time, I'd like to avoid even the possibility of triggering a 
behaviour change on Microsoft Windows clients. I had thought Teredo was 
sunsetted, but examination of my Windows 7 PC suggests it is not, 
although it is broken.


To inform my decision about unblocking it, I'd like to ask a few questions.

Does anyone know why Teredo has not been sunsetted yet?

Does anyone know when Teredo will be sunsetted?

Does anyone know of a safe way to block Teredo from Microsoft Windows 
clients, but leave Teredo from XBox One unaffected? Jeroen, your 
suggestion of blocking the DNS name is a good one. Anyone any other 
ideas I should also consider.


===

Hopefully this is specific enough...

Re: Teredo sunset - did it happen?

[Phil Mayers]

On 17/11/2014 17:43, Darren Pilgrim wrote:


Any ideas what's going on? Microsoft, anyone care to comment?


Microsoft released an Windows Update for the prefix policy table.  The
update dropped Teredo's precedence to lower than IPv4.


Just to be clear - are you suggesting they did this instead of 
sunsetting Teredo altogether?


In any case, I was always under the impression this was the day-one 
experience - Teredo would only be used to talk to another Teredo DNS 
name or an IPv6-only name in the absence of native IPv6. Am I mistaken?

Clueless national monopoly providers

[Phil Mayers]

FFS

$ nc -6 -v www.bt.com 80
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 2a00:2381:::1:80.
GET / HTTP/1.0
Host: www.bt.com

Ncat: Connection reset by peer.


Maybe I should be glad BT haven't deployed any IPv6 to their residential 
customers; they'd only find some way to mess up a pretty straightforward 
task. Like keeping a link up for more than 24 hours.

Re: Clueless national monopoly providers

[Phil Mayers]

On 10/10/14 14:50, Bjoern A. Zeeb wrote:


% telnet -4 www.bt.com 80
Trying 62.239.186.73...
Connected to www.bt.com.
Escape character is '^]'.
GET /
Connection closed by foreign host.


Whatever load balancer that is, it needs an upgrade and understand g’ol HTTP 
0.9 as well in addition to IPv6 ;-)


In fairness, I've run into a lot of actual, real-life webserver code 
(usually Java Servlet based cough Oracle cough) that requires, at 
minimum, a Host: and Accept: header. Annoying, but I guess...


However as another person points out, this fails with real, browser, 
HTTP requests. The nc was just a convenient way of showing it ;o)

Re: wake on lan / wol with linux in IPv6-LAN (without IPv4)

[Phil Mayers]

On 22/09/14 08:42, Ignatios Souvatzis wrote:


But I imagine people might want to wake every host once a night
and run some backup or software update remotely; so unconcerned
machines would see, say, one or two packets times the number of
sleeping machines per night. How many hosts do you have per broadcast
domain? more than 2**8?


We have some IPv4 /21 with 2**10 hosts actually. They're managed 
machines and it's largely to work around the IPv6 address space wastage. 
It's marginally suboptimal from a size of failure domain PoV, but 
we've never actually had the sort of problems this can lead to in 
theory, and the alternative was worse.


Obviously single /64 IPv6 (bliss!)



There's another consideration: if you do have the need, due to huge
broadcast domains, nobody prevents you to make your machines
subscribe to a locally assigned multicast address and send your


Erm... while they're asleep? Doubtful that would work - most network 
stacks end up dropping then re-initialising the ethernet link when they 
go to sleep, so MLD snooping state is likely to be cleared, and absent 
something like AMT/vPro (which comes with it's own IPv6 hassles ;o) to 
join the group in the sleeping link-state...

Re: wake on lan / wol with linux in IPv6-LAN (without IPv4)

[Phil Mayers]

On 22/09/14 10:51, Phil Mayers wrote:

On 22/09/14 08:42, Ignatios Souvatzis wrote:


But I imagine people might want to wake every host once a night
and run some backup or software update remotely; so unconcerned
machines would see, say, one or two packets times the number of
sleeping machines per night. How many hosts do you have per broadcast
domain? more than 2**8?


We have some IPv4 /21 with 2**10 hosts actually


Just to clarify - nearly 2**11, so it isn't half a subnet wasted!

Re: Large IPv6 Multicast Domains

[Phil Mayers]

On 20/06/14 07:11, Mark Tinka wrote:

On Thursday, June 19, 2014 08:17:07 PM Stig Venaas wrote:


I'm
hoping SSM will be the way to go for interdomain in
general.


Agree - prefer SSM also.


Who doesn't - app coders AFAICT? ;o)

Re: Large IPv6 Multicast Domains

[Phil Mayers]

On 20/06/14 13:08, David Barak wrote:


There are specific use cases for ASM (in IPv4) in distributed
monitoring (many-few traffic flows)  It sure would be a shame for
that to go away...


Well, I doubt it will go away. Presumably embedded RP will serve those 
needs in the v6 world.


But IME transition from (*,g) to (s,g) and RP-tree flooding is where a 
lot of PIM-related issues occur, so I am totally on board with 
preferring SSM where possible.

Administrativa: auto-responder

[Phil Mayers]

Could the list admin please un-sub jstr...@cityoftaft.org, who is 
sending an I'm retiring auto-responder?


Cheers,
Phil

Re: Administrativa: auto-responder

[Phil Mayers]

On 04/06/2014 19:55, Phil Mayers wrote:

Could the list admin please un-sub jstr...@cityoftaft.org, who is
sending an I'm retiring auto-responder?


Oh FFS... and qual...@of2m.fr as well?

Has someone bulk-sub'ed a load of people mischeviously?

AMT/vPro MLD storms?

[Phil Mayers]

All,

In the last week or so, we've started to see a problem on newer PCs with 
the Intel AMT/vPro (a kind of inline out-of-band management controller, 
for those unfamiliar with it) which now supports IPv6... after a fashion.


The specific issues is that under certain as-yet unidentified 
conditions, two such machines which are asleep will start to emit MLD 
packets at a high rate - 1kpps. This eats a lot of CPU on the attached 
router (and can't be great for everything else, either).


The MLD packets must of course be coming from the AMT/vPro stack which 
shares the system MAC address (an unwise design decision IMO) and sort 
of shares it's IP stack. We've confirmed this by looking at the port 
speed, which is 10meg when the machine is asleep (versus 1gig when awake).


It seems that the AMT controllers goad each other into emitting the 
packets - if you take one offline, the other stops.


The MLD packets are of the form:

2c:44:fd:xx:xx:xx  33:33:00:01:00:03, ethertype IPv6 (0x86dd), length 
86: fe80::2e44:fdff:fexx:  ff02::1:3: HBH ICMP6, multicast listener 
reportmax resp delay: 0 addr: ff02::1:3, length 24


...and alternate from each machine; as above, as if each machine is 
induced to emit an MLD packet by seeing the other do it.


Note the v6 LL IP is a mutated form of EUI-64 (locally-assigned bit 
toggled?)


Has anyone seen anything like this?

Cheers,
Phil

Re: AMT/vPro MLD storms?

[Phil Mayers]

On 06/02/14 12:42, Sam Wilson wrote:


Note the v6 LL IP is a mutated form of EUI-64 (locally-assigned bit
toggled?)



Are you sure about that last?  Surely the U/L bit should be flipped


Oops. Quite right, well spotted.

Re: So, time for some real action?

[Phil Mayers]

On 06/02/2014 17:52, Andrew   Yourtchenko wrote:


Last time I checked, anyone with available days off can take them at
any time for any reason.


Most places aren't quite that generous; notice, simultaneous team member 
leave and exceptional circumstance clauses typically apply.


But I take your point; individuals are of course free, modulo such 
concerns, to take time off for their own reasons.


IMHO taking a days leave because IPv4 is still required is silly. But 
hey, who am I to say?



That's exactly the idea. It's explicitly *NOT* to break others'
networks nor to have the innocent users suffer.


Ok. But if you read the replies to the original email, it's clear a lot 
of people didn't get that. So there is a messaging problem here.



Having a defined day when others are doing the same thing makes it
easier to allocate the time for it, at least for some.


Shrug. If you say so.


If you are talking about the original wording on the AVAAZ - I'd be
very happy to hear better wording, feel free to unicast.


My problem is entirely with the work point. Denying yourself online 
shopping and facebook is just that - self-denial. Though a really brave 
option would be to do that *permanently*, and let the retailers know why 
you're *never* shopping with them until they're v6-ready.


Denying yourself the ability to work *in the field you're trying to 
affect change* seems futile.


It would be better to go to work, try and work with IPv4 disabled, make 
a note of everything that didn't work, then commit to fixing it all 
before the same time next year. That's both far harder, and far more 
productive, than throwing your hands in the air and saying nothing 
works without IPv4 - which is not a surprising conclusion ;o)



IMHO effort at this point would be best directed to the large, holdout
broadband providers in countries with low uptake (e.g. BT in the UK).



What would that effort consist of ?


That is an excellent question which I am not well equipped to answer. If 
there is anyone on the list with insight in the UK broadband market, and 
any workable suggestions and/or hopeful news, I'd love to hear it.


I note that BT have, recently, gone to the expense of deploying CGN but 
not IPv6 - which is not promising. Aalthough the newest CPE has IPv6 
stuff in the UI, currently all disabled, so maybe they'll turn it on 
later...

Re: RA DHCP problem...

[Phil Mayers]

On 30/12/2013 12:13, Mikael Abrahamsson wrote:


I am not asking these questions to be mean, I'm asking them to bring out
all the reasons so someone will document them so they can be presented
in a coherent consise manner (for instance an I-D). I know I have to do
this when I want things to change. Been there, done that.


Disclaimer: although I think defgw-in-DHCPv6 would be useful, we run 
SLAAC+RA without too many issues, and my gut feeling is that the IETF 
will never allow standardisation of, nor will vendors engage in 
deployment of, such a protocol in any timescale I care about. So this 
discussion is interesting to me for academic reasons only.


That said: I'll document a specific issue we face in a large-ish 
enterprise-style (UK University) network, which I think RA-less IPv6 
might alleviate/solve.


We run a layer2 edge, layer3 core, and use MAC-based or 802.1x-based 
VLAN assignment to put devices at the edge into different networks, 
which are mapped into different L3VPN. Our edge switches support 
something that's getting pretty common now - mutiple supplicant mode, 
where 1 untagged vlan can be present on a port. RX traffic is mapped to 
the right vlan based on source MAC, and TX traffic from all vlans is 
sent to the port, including broadcast/multicast.


(You can probably see where this is going)

This feature is pretty useful because it allows end users in, for 
example, student residences to use a dumb switch on their port, but 
still lets us map the devices into different VLANs, allowing us to catch 
new ones and force them to do a click-through registration, for example.


Other use-cases are VoIP+PC on the same port without LLDP/CDP/other 
voice VLAN hassles, or VM guests on separate VLANs to the VM host on 
unmanaged or semi-managed machines.


One problem we have with this setup: If two devices are on a port, in 
different IPv6-enabled VLANs, they both see both RAs, and IPv6 
connectivity breaks.


If however there were no RAs, that would not be a problem. Possibly 
DHCPv6 + defgw would solve that?


Now I can anticipate a number of objections to this, so let me try and 
cover them up front:


1. That's a terrible architecture it's (insecure | non-standard | blah)

To be blunt: shove it. That's not your call, it's mine. It works great 
for us (mutiple RAs aside) and has the cost/benefit tradeoff is smack 
bang in the middle of where we, as an organisation, want to be. We're 
aware of its limitations, and don't claim it as perfect. But I know lots 
of other places that run substantially similar setups.


2. The switch (could | should) intercept and unicast the RAs to the 
correct endpoint MAC


Maybe; that's a possible solution. In our case, the vendor is 
IPv6-ignorant, and I hold no hope of them ever doing it. But it's 
certainly an option. It does imply a busy CPU on the edge device as the 
number of VLANs/RAs-per-second goes up, and there might be other 
problems I can't think of right now (are there cases where a device will 
process an RA even if it's to a different destination MAC?)


3. 802.1x-REV will solve this as each endpoint will have separate 
MACSEC associations, and not see each others traffic


Yeeaah maybe, in about 15 years. Seems like it would be nice to 
solve it earlier than that... ;o)


===

Anyway, as I said above - I would have liked to have seen defgw in 
DHCPv6, but we run mostly OK on RA+SLAAC. Since people seem interested 
in having issues with RA documented, I thought I'd write this one up.


Thanks all for the interesting discussion (though I see lamentably 
little use of the principle of charity ;o)


Cheers,
Phil

Re: RA DHCP problem...

[Phil Mayers]

On 30/12/2013 15:13, Lorenzo Colitti wrote:


No, I mean - from a *security* perspective there's actually no security,
because if there existed a host implementation that always tried all
source addresses every time it connected, then that implementation would
always work with no issues, even if you tried to put it on a restricted
VLAN.


Er, no. Sorry, I don't understand why you'd think that. Perhaps I am 
misunderstanding you, or you me.



You could also fix this on the network side. You can even do this while
maintaining the architecture :-) - when we had this problem many years
ago (on wifi), the vendor fixed it by converting all RAs to unicast.


I did point this out in my original mail. As noted, our vendor is 
IPv6-ignorant, so on this current platform it's not going to happen, 
which is a shame as it's a moderately easy solution.



If you want to solve it using DHCP, then yes, clients that don't support
DHCP are out. But again, you can fix this in the network as well. From
an architectural perspective, what you have is a hack that happens to
work in IPv4 because nothing depends on true VLAN isolation. It doesn't
happen to work in IPv6.


Yes.

If DHCPv6 were usable, and could carry gw/prefix info, perhaps the hack 
would work again, and maybe it's a hack that's common and useful to a 
number of people, hence my email.


However, since that will never happen IMO, it is (to risk repeating 
myself) of solely academic interest to me.


I feel like we're going in circles a bit, so given the above I'm going 
to shut up now.



IMO this is the direction we should be going in. Not let's just use
DHCPv6, because it works so well in IPv4 (not).


Maybe. I guess we'll see in the medium term if those features become 
common enough (and operationally usable).

Re: RA DHCP problem...

[Phil Mayers]

On 30/12/2013 21:40, S.P.Zeidler wrote:

Thus wrote Phil Mayers (p.may...@imperial.ac.uk):


One problem we have with this setup: If two devices are on a port,
in different IPv6-enabled VLANs, they both see both RAs, and IPv6
connectivity breaks.


I assume you have considered fixing the route part by using fe80::1
(f.e.) as router address on all relevant VLANs?


In fact we use a consistent HSRPv2 config on all our subnets, so the 
gateway *is* always the same; not fe80::1 but fe80::5:73ff:fea0:1. So 
basically, yes we did this.



What reason spoke against it?


As above, none ;o)


You'll still have the using wrong prefix to source packets problem
though (which could be fixed using dhcpv6 if your equipment were inclined
to support it).


Sure. I guess I should re-check, now that we're on IOS 15SY; maybe they 
finally fixed the bug - I did eventually get them to acknowledge that 
looking up the DHCP server next-hop and getting an output interface of 
Null0 and a source IP of fe80::... was a bug, not a feature request :o/

Re: Over-utilisation of v6 neighbour slots

[Phil Mayers]

On 03/11/2013 16:30, Jared Mauch wrote:

I've noticed that my ipv6 is about 1ms faster than ipv4 consistently
in measurements. I doubt that is enough faster to make a difference
in most transactions so they would be equally preferred.


Interestingly, that last time I timed the IPv4 versys IPv6 latency - 
from myself to www.google.com - I saw a clear bias in favour of IPv4, of 
about 2ms.


Having just re-timed it, they're pretty much dead-on the same. 
Presumably my upstream and/or google have rejigged things in the meantime.

Re: Over-utilisation of v6 neighbour slots

[Phil Mayers]

On 21/10/13 20:35, Phil Mayers wrote:


Specifically, our Cisco 6500/sup720 ran out of IPv6 FIB slots, as
num_routes + num_neighs exceeded 32k (the default IPv4/IPv6 TCAM split
on this platform being 192k/32k).


I wanted to follow up on this. Some folks from Cisco kindly contacted me 
off-list, and correctly guessed that a large number of the IPv6 
neighbour entries were in state STALE, and pointed me to the 
relatively new:


  ipv6 nd cache expire seconds

...interface-level command. This wasn't in the IOS train we were running 
until relatively recently, so I hadn't seen it before.


Having applied this, we saw a sharp drop in v6 neighbour count, although 
it didn't seem to take effect on existing entries - I was able to force 
it by flapping the interface and refreshing all the neighbours.


The entries seem to expire after ipv6 nd cache expire + ipv6 nd 
reachable-time i.e. I see a max age in the neighbour table of 24 
minutes for parameter values of 1200 and 30 (in seconds and 
milliseconds) respectively.


There are also a bunch of newer per-interface ND commands (per-IF ND 
cache size limits, for example) that could help with resource 
exhaustion, so people on Cisco gear should take a look.

Re: Over-utilisation of v6 neighbour slots

[Phil Mayers]

On 10/24/2013 08:18 AM, Benedikt Stockebrand wrote:

In my opintion the problem here is not so much Apple, but Cisco.  While


Well, I think there's more than one problem.

Certainly fixed-size (and relatively small) FIBs in Cisco-land are a 
problem. On devices where the FIB is a relatively fast-but-inflexible 
architecture - like TCAM - good sizing decisions at design time need to 
be married with smart algorithms at runtime (i.e. partition TCAM 
dynamically not statically at reboot!). Sup720 doesn't do well in both 
categories!


It is only relatively recently that TCAM-based platforms have started to 
grow in terms of FIB size - sup2T still comes in the same sizes as 
sup720, but the new 6880 has bigger.


But even if you forget completely about the FIB-size issue, I *still* 
assert that Apple's v6 privacy address behaviour is idiotic. For those 
of us who log v6-MAC mappings into SQL, it balloons the logging 
requirements. It loads IPv6 FHS implementations. And it provides 
negligible - perhaps zero - improvement in privacy.


I've observed Apple devices powering up, generating a random IPv6 
address, NEVER USING IT, then powering it down and losing it, at 
intervals of tens of seconds. That's just asinine.


I assert that rolling the address on a timer, not on power/link 
activity, is the intent of the RFCs, and the desired behaviour, and that 
Apple are doing the wrong thing here.



I understand that CAM/TCAM is painfully expensive in hardware, in the
long run increasing its size is the way to go.  On the Cisco side, the


In the long run, a move to RAM-based trie lookups seems to be the way to 
go for FIBs, for the superior power use characteristics if nothing else.



quick workaround may be a reliable expiration mechanism.  On your side,
maybe some further segmentation can help to spread the load over
multiple routers (yes, I know that's frequently not an option on WiFi).


...as is the case here. That said, we are pondering moving the wireless 
routing off onto dedicated devices - anyone got any recommendations? ;o)


Cheers.
Phil

Re: Over-utilisation of v6 neighbour slots

[Phil Mayers]

On 22/10/13 10:18, Sam Wilson wrote:


On 22 Oct 2013, at 06:03, Eric Vyncke (evyncke) wrote:


But, the rapid rate of new RFC 4941 addresses for iOS has another
impact because network devices cannot anymore limit the number of
IPv6 addresses per MAC address in order to prevent a local DoS.

So, either you disable SLAAC and rely on stateful DHCPv6 (but then
Android is not happy) or use aggressive time to clean the ND
cache...


... with the attendant difficulty in tracing systems that might be
doing Bad Things.

We have a mixture of Sup2Ts and Sup720s and we don't (yet) have v6
enabled on most of them.  It's stuff like this that makes me think
it's *still* not time to offer a general v6 service.


I disagree - and since I'm the one who posted about the problem, I call 
dibs on getting to decide how serious it is ;o)


We offer a general IPv6 service, and we've had very few real problems. 
It is NOT as hard as people make out, and if you wait until every last 
problem is solved, you'll be waiting forever.


You'll also be missing out on the opportunity to learn about issues 
early and influence your vendors and your own future purchases in 
appropriate ways.


Hold off on IPv6 is something I would recommend to my competitors...

Over-utilisation of v6 neighbour slots

[Phil Mayers]

All,

We ran into an interesting issue on our wireless network today, caused 
mainly by the known behaviour of Apple clients in generating fresh 
privacy addresses every time there's a power sleep/wake state change 
(i.e. a lot) combined with a non-default NS/ND config on our side.


Specifically, our Cisco 6500/sup720 ran out of IPv6 FIB slots, as 
num_routes + num_neighs exceeded 32k (the default IPv4/IPv6 TCAM split 
on this platform being 192k/32k).


This was probably exacerbated by longer-than-default NS cache timeouts 
on our part, recommended in response to CPU issues we faced as discussed 
here:


http://www.gossamer-threads.com/lists/cisco/nsp/161344

To combat the issue I cranked the nd reachable-time down from 900 to 
300 seconds and re-carved the TCAM to give 64k IPv6 and reloaded, but it 
was all most troubling. To give an idea why, some stats:


10576 distinct MAC addresses with 1+ global IPv6 addresses
29610 IPv6 addresses (not including LL)
25.3% of hosts with = 4 v6 address
17299 IPv6 addresses consumed by that 25.3%

That is, ~25% of hosts consuming ~59% of the addresses in-use. OUIs for 
the MAC addresses in question were almost entirely Apple. Distribution was:


frequencyrel. cum.

 14651 43.98%   43.98% ***
 22084 19.70%   63.68% ***
 31164 11.01%   74.69% ***
 4 729  6.89%   81.58% **
 5 560  5.30%   86.88% *
 6 387  3.66%   90.54% *
 7 276  2.61%   93.14%
 8 206  1.95%   95.09%
 9 177  1.67%   96.77%
10 120  1.13%   97.90%
11  70  0.66%   98.56%
12  50  0.47%   99.04%
13  26  0.25%   99.28%
14  29  0.27%   99.56%
15  16  0.15%   99.71%
16  13  0.12%   99.83%
17   9  0.09%   99.91%
18   5  0.05%   99.96%
19   1  0.01%   99.97%
20   2  0.02%   99.99%
24   1  0.01%  100.00%


Some questions:

 1. Has anyone else run into this sort of thing (neighbour table 
exhaustion) and what kind of approach did you take to solving or 
ameliorating it? DHCPv6?


 2. Does anyone know if Apple (and other vendors) understand the 
negative consequences of their aggressive rotation of IPv6 privacy 
addresses, and are going to address it?


 3. Does anyone know if any equipment vendors have more intelligent 
strategies for handling this kind of situation - LRU expiry of v6 
neighbours at 90% util rather than self-destructing FIB overflow, for 
example ;o)


[We're aware the sup720 is old, but it seems like this could be an issue 
even for more recent devices at sufficient scale]


Cheers,
Phil

Re: Over-utilisation of v6 neighbour slots

[Phil Mayers]

On 21/10/2013 21:19, Cutler James R wrote:


4.  Does Apple's approach to IPv6 privacy addresses properly support
the intent of privacy addresses?

My tentative answer is, Yes, and we need to learn to cope.


The general approach perhaps, but the rollover timing is way, way too 
aggressive IMO. It should be on a timer, not driven by PHY wake events. 
Even 300 seconds would be an improvement over the behaviour we're seeing.


As to we need to learn to cope - lots of networks have huge amounts of 
capital investment which can't just be ripped out and replaced overnight 
because Apple have decided to be aggressive with address rollovers. If 
the main outcome is for FIB-pressured sites to disable IPv6 on OUIs 
registered to Apple, it's a retrograde step ;o)


Maybe we need a neigbour un-advert ICMPv6 message that the old 
addresses could be torn down with.

Re: The subnet-router anycast address

[Phil Mayers]

On 09/10/13 10:41, Harald Terkelsen wrote:

Hi!

Is anyone actually using the subnet-router anycast address in your network?


No.

Frankly I've never understood what purpose it served other than to confuse.


responds when IPv6 forwarding is enabled. On our wireless subnets, we
see lots of DAD requests for the subnet-router anycast coming from MAC
addresses registered to HTC. If the address is already in use, it looks
like the HTC-device do not do anything about it. If the address is not
in use, it will enable the subnet-router anycast address and starts
responding to NS requests for this address.


Great. More brokenware...

Re: IPv6 duplicate DAD packets from Android clients?

[Phil Mayers]

On 10/08/2013 01:02 AM, Erik Kline wrote:


It could be a bug, i.e. a client trying to do an NS for the router but
for some reason not using its link-local address (whacky race
condition where DAD for link-local hasn't completed?).



Maybe; we've got a lot of Android clients on the network, and I'd 
assumed any bug would be more widespread, but I guess given the 
diversity of hardware and OS fiddling that goes on, it might be feasible 
that only 2 of ~3000 hosts would be affected.


Maybe I'll try and identity the client OSes more accurately.

Re: teredo.ipv6.microsoft.com off?

[Phil Mayers]

On 07/18/2013 09:09 PM, Brian E Carpenter wrote:


Wait... I had the impression that iff there was no other IPv6 connectivity,
Teredo was used in older Windows because of the generic prefer IPv6 rule.
The default RFC 3484 table covers 6to4 but not Teredo.


AFAIK, every version of windows (i.e. Vista, 7, 8) that comes with 
Teredo also comes with a de-pref rule for it, not just recent versions.


Put another way, Teredo should never be preferred over IPv4, because all 
versions of Windows with Teredo use extended RFC 3484 rules.


Most of the Teredo activity we see is when IP addresses are used 
directly (i.e. no getaddrinfo). For example, BitTorrent connections 
where peers were looked up in DHT/PEX. In these cases, an IPv6 address 
will be connected to over Teredo if there's no other connectivity.

Re: teredo.ipv6.microsoft.com off?

[Phil Mayers]

On 17/07/13 21:09, Brian E Carpenter wrote:

On 17/07/2013 19:13, Ignatios Souvatzis wrote:
...


Let me ask one thing... a couple of years ago, when I read the
specification of Teredo, I was quite impressed by the details (If
you accept the premise that you have to work around being jailed
behind an IPv4 NAT) put into the protocol. One detail was that it
is supposed to be lowest priority and so go automatically away
(from the client end) as soon as some configued IPv6 is available
on the link.

Isn't that how it's implemented?


Yes, but the result is that the host tries to use Teredo preferentially
even if the IPv4 path is better; and if the Teredo path is broken


That is the opposite of how it's supposed to work. Teredo addresses 
should be de-pref'd below everything else, and would thus only be used 
for connection to IPv6-only hosts if the host lacked other IPv6 
connectivity.


As someone else has pointed out, maybe it gets used for IPv6 literals, 
but not hostnames - the RFC 3484 table on windows ensures this.

Re: same link-local address on multiple interface and OSPFv3

[Phil Mayers]

On 06/29/2013 09:31 PM, Brian E Carpenter wrote:


Of course not, but I was trying to see how deep in the product
design the issue might go. It sounds like dumb copying of the IPv4
logic


That seems like a pretty reasonable guess. OTOH, OSPFv3 is different 
enough that I expect the code was all or largely original, so it might 
be a layer 8 logic error ;o)

Re: Point-to-point /64

[Phil Mayers]

On 02/06/13 22:51, Brian E Carpenter wrote:

On 03/06/2013 08:49, Darren Pilgrim wrote:
...

I'm not sure about other switches, but for the Catalyst 3750/3750G, it
means some quirks with IPv6 ACLs.  The 3750/3750D can do ACLs on full
/128's, but only if the lower 64 bits are EUI64.


Huh? How can it possibly know that? (see draft-ietf-6man-ug)


It doesn't know that; it just ignores those bits, so unless the *are* 
EUI-64, your ACLs might mis-match. It uses those 16 bits for the port.


This is an issue on some higher-end platforms too (6500) but the ACL 
match mode is selectable there (google ipv6 ACL compression).

Re: IPV6 in the network core and MPLS

[Phil Mayers]

Right now on those platforms afaik mpls needs ipv4 in the core.

I have no idea if/when ldpv6 and the relevant bgp stuff will appear in ios/nxos 
- is it available on any platform (junos?) yet?

Jim Trotz jtr...@gmail.com wrote:

I have been trying to find out if we can use MPLS  LDP to setup MPLS
VPNS
within our all IPV6 network. Everything I read says no, LDP  MPLS
don't
support native IPV6, you have to use 6VPE and IPV4 in the core.

Is this true?  PS: We are an all Cisco IOS/NXOS based network.

--
Sent from my mobile device, please excuse brevity and typos.