Re: Google no longer returning AAAA records?
On 15/04/15 16:05, Brian Rak wrote: We noticed that we're no longer getting results back for google.com when we do queries from a few of our recursive servers (other ones are fine). A bit of searching revealed that a few of our servers are listed here http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_.txt (there are 4 listed for AS20473, which are the ones I'm referring to) However, I can't seem to find any information about why we'd be listed there, nor can I find anything telling me how to get delisted. Yes. They don't provide that info, nor do they provide a way to request a de-listing AFAIK. You'll be removed automatically when the problem goes away. There's a lot of discussion in the archives about this, but I believe that, as far as is known publicly: 1. There's a web-bug in the google search page 2. They correlate IPv6 failures of this against the DNS lookup 3. When the DNS source IP hits a certain threshold of correlatd failures, they stop serving records for a time (one week?). Subjectively it's irritating that Google don't provide info to operators as to the specifics of the cause, but look at it from their PoV - there's a lot of info, some of it potentially personal / data-protected, that they'd have to communicate securely to operators when they ask. It would be a lot of work for them and I'm somewhat sympathetic on that basis (although I wasn't when it was happening to us ;o) My guess: you've got some form of broken IPv6 connectivity talking to your resolvers; maybe rogue RAs, a tunnel, VPN, etc. The customers with this problem aren't reporting it because Happy Eyeballs, but the Google web-bug is detecting it. We saw a reduction (and eventual end to) our experiences of this when we finished our native dual-stack deployment *and* when we blacklisted serving of to some of our more troublesome netblocks - mainly remote access VPN users. We monitor whether google are blacklisting us in our Nagios setup, so we can see if problems come back. An alternative might be to steer different classes of users to different resolver query source IPs (either actual different resolvers, or using views multiple IPs). Then, you can see which source IP and thus class of users is triggering it. Good luck tracking it; it can be frustrating :o( Cheers, Phil
Google no longer returning AAAA records?
We noticed that we're no longer getting results back for google.com when we do queries from a few of our recursive servers (other ones are fine). A bit of searching revealed that a few of our servers are listed here http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_.txt (there are 4 listed for AS20473, which are the ones I'm referring to) However, I can't seem to find any information about why we'd be listed there, nor can I find anything telling me how to get delisted. I found an old mailing list post talking about contacting dns-admin@ and google-ipv6@, however dns-admin bounces telling me to go to google.com/support, and google-ipv6@ bounces with 'The group google-ipv6@ does not allow posting through email.' Is anyone here that can help me, or tell me who I should be talking to? Thanks, Brian Rak
Re: Google no longer returning AAAA records?
On 4/15/2015 11:28 AM, Phil Mayers wrote: On 15/04/15 16:05, Brian Rak wrote: We noticed that we're no longer getting results back for google.com when we do queries from a few of our recursive servers (other ones are fine). A bit of searching revealed that a few of our servers are listed here http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_.txt (there are 4 listed for AS20473, which are the ones I'm referring to) However, I can't seem to find any information about why we'd be listed there, nor can I find anything telling me how to get delisted. Yes. They don't provide that info, nor do they provide a way to request a de-listing AFAIK. You'll be removed automatically when the problem goes away. There's a lot of discussion in the archives about this, but I believe that, as far as is known publicly: 1. There's a web-bug in the google search page 2. They correlate IPv6 failures of this against the DNS lookup 3. When the DNS source IP hits a certain threshold of correlatd failures, they stop serving records for a time (one week?). Subjectively it's irritating that Google don't provide info to operators as to the specifics of the cause, but look at it from their PoV - there's a lot of info, some of it potentially personal / data-protected, that they'd have to communicate securely to operators when they ask. It would be a lot of work for them and I'm somewhat sympathetic on that basis (although I wasn't when it was happening to us ;o) My guess: you've got some form of broken IPv6 connectivity talking to your resolvers; maybe rogue RAs, a tunnel, VPN, etc. The customers with this problem aren't reporting it because Happy Eyeballs, but the Google web-bug is detecting it. We saw a reduction (and eventual end to) our experiences of this when we finished our native dual-stack deployment *and* when we blacklisted serving of to some of our more troublesome netblocks - mainly remote access VPN users. We monitor whether google are blacklisting us in our Nagios setup, so we can see if problems come back. An alternative might be to steer different classes of users to different resolver query source IPs (either actual different resolvers, or using views multiple IPs). Then, you can see which source IP and thus class of users is triggering it. Good luck tracking it; it can be frustrating :o( Cheers, Phil Well, we're a hosting provider and we have a very large number of users doing all sorts of crazy things. These particular resolvers are used by our low cost virtual server platform, so we see many users running a wide variety of proxy software. It's pretty difficult to prevent people from breaking their IPv6 connectivity when they have full control over the machines. There may be some things that we can do to reduce the number of broken IPv6 setups.. I just wasn't sure if we'd ever drop off that list automatically.
Re: Google no longer returning AAAA records?
I suggest checking if any of your affected users have broken 6to4 setups, and that you are applying the relevant mitigations in RFC 6343. MTU size issues and high latency have also both been mentioned as possible reasons for the mysterious blacklist. It has also been said that n...@google.com never replies. Regards Brian Carpenter On 16/04/2015 07:16, Brian Rak wrote: On 4/15/2015 11:28 AM, Phil Mayers wrote: On 15/04/15 16:05, Brian Rak wrote: We noticed that we're no longer getting results back for google.com when we do queries from a few of our recursive servers (other ones are fine). A bit of searching revealed that a few of our servers are listed here http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_.txt (there are 4 listed for AS20473, which are the ones I'm referring to) However, I can't seem to find any information about why we'd be listed there, nor can I find anything telling me how to get delisted. Yes. They don't provide that info, nor do they provide a way to request a de-listing AFAIK. You'll be removed automatically when the problem goes away. There's a lot of discussion in the archives about this, but I believe that, as far as is known publicly: 1. There's a web-bug in the google search page 2. They correlate IPv6 failures of this against the DNS lookup 3. When the DNS source IP hits a certain threshold of correlatd failures, they stop serving records for a time (one week?). Subjectively it's irritating that Google don't provide info to operators as to the specifics of the cause, but look at it from their PoV - there's a lot of info, some of it potentially personal / data-protected, that they'd have to communicate securely to operators when they ask. It would be a lot of work for them and I'm somewhat sympathetic on that basis (although I wasn't when it was happening to us ;o) My guess: you've got some form of broken IPv6 connectivity talking to your resolvers; maybe rogue RAs, a tunnel, VPN, etc. The customers with this problem aren't reporting it because Happy Eyeballs, but the Google web-bug is detecting it. We saw a reduction (and eventual end to) our experiences of this when we finished our native dual-stack deployment *and* when we blacklisted serving of to some of our more troublesome netblocks - mainly remote access VPN users. We monitor whether google are blacklisting us in our Nagios setup, so we can see if problems come back. An alternative might be to steer different classes of users to different resolver query source IPs (either actual different resolvers, or using views multiple IPs). Then, you can see which source IP and thus class of users is triggering it. Good luck tracking it; it can be frustrating :o( Cheers, Phil Well, we're a hosting provider and we have a very large number of users doing all sorts of crazy things. These particular resolvers are used by our low cost virtual server platform, so we see many users running a wide variety of proxy software. It's pretty difficult to prevent people from breaking their IPv6 connectivity when they have full control over the machines. There may be some things that we can do to reduce the number of broken IPv6 setups.. I just wasn't sure if we'd ever drop off that list automatically.
Re: Google no longer returning AAAA records?
And you are not alone... While my employer has deployed a lot of IPv6 internally (still not 100% though), some internal DNS servers are blacklisted by Google. Probably because a lot of our internal labs (which are also IPv6-enabled of course) are managed by the engineers using the lab, so, ending up in less than optimal IPv6 configuration in same case (when the lab has nothing to do with the IP layer = mine is optimized for IPv6 :-) ) -éric On 15/04/15 21:16, Brian Rak b...@choopa.com wrote: On 4/15/2015 11:28 AM, Phil Mayers wrote: On 15/04/15 16:05, Brian Rak wrote: We noticed that we're no longer getting results back for google.com when we do queries from a few of our recursive servers (other ones are fine). A bit of searching revealed that a few of our servers are listed here http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_.txt (there are 4 listed for AS20473, which are the ones I'm referring to) However, I can't seem to find any information about why we'd be listed there, nor can I find anything telling me how to get delisted. Yes. They don't provide that info, nor do they provide a way to request a de-listing AFAIK. You'll be removed automatically when the problem goes away. There's a lot of discussion in the archives about this, but I believe that, as far as is known publicly: 1. There's a web-bug in the google search page 2. They correlate IPv6 failures of this against the DNS lookup 3. When the DNS source IP hits a certain threshold of correlatd failures, they stop serving records for a time (one week?). Subjectively it's irritating that Google don't provide info to operators as to the specifics of the cause, but look at it from their PoV - there's a lot of info, some of it potentially personal / data-protected, that they'd have to communicate securely to operators when they ask. It would be a lot of work for them and I'm somewhat sympathetic on that basis (although I wasn't when it was happening to us ;o) My guess: you've got some form of broken IPv6 connectivity talking to your resolvers; maybe rogue RAs, a tunnel, VPN, etc. The customers with this problem aren't reporting it because Happy Eyeballs, but the Google web-bug is detecting it. We saw a reduction (and eventual end to) our experiences of this when we finished our native dual-stack deployment *and* when we blacklisted serving of to some of our more troublesome netblocks - mainly remote access VPN users. We monitor whether google are blacklisting us in our Nagios setup, so we can see if problems come back. An alternative might be to steer different classes of users to different resolver query source IPs (either actual different resolvers, or using views multiple IPs). Then, you can see which source IP and thus class of users is triggering it. Good luck tracking it; it can be frustrating :o( Cheers, Phil Well, we're a hosting provider and we have a very large number of users doing all sorts of crazy things. These particular resolvers are used by our low cost virtual server platform, so we see many users running a wide variety of proxy software. It's pretty difficult to prevent people from breaking their IPv6 connectivity when they have full control over the machines. There may be some things that we can do to reduce the number of broken IPv6 setups.. I just wasn't sure if we'd ever drop off that list automatically.
