Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 10:17, Roland Dobbins wrote:





Over on nznog, Cameron Bradley posited that this may be related to a 
TR-069/-064 Mirai variant, which makes use of a 'SetNTPServers' exploit. 
 Perhaps one of them is actually setting timeservers?  This SANS 
writeup details the SOAP strings:




---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 10:16, Roland Dobbins wrote:

> 



---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 10:09, Dan Drown wrote:

 This seems more like "someone pushed out bad firmware" rather than 
something malicious.


Everything old is new again . . .



---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Dan Drown]

Quoting Roland Dobbins :
Do you have flow telemetry, which provides a lot more information  
than basic pps/bps stats?


Sources are pretty widely spread out among cell networks/home  
internet, seem to be mostly US based.  I'm not seeing a large amount  
of traffic per single IP or single subnet.  This seems more like  
"someone pushed out bad firmware" rather than something malicious.


Are you seeing normal timesync queries, or lots of level-6/level-7  
admin command attempts?


SNTP Client timesync queries make up 91.3% of the traffic to my server.

The following NTP settings being most the popular (47% of all traffic  
to my server):


stratum=0, poll=4, precision=-6, root delay=1, root dispersion=1,  
reference timestamp=0, originator timestamp=0,

receive timestamp=0

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 5:45, Jose Gerardo Perales Soto wrote:

We've recently experienced a traffic increase on the NTP queries to 
NTP pool project (pool.ntp.org) servers.


Do you have flow telemetry, which provides a lot more information than 
basic pps/bps stats?


Are you seeing normal timesync queries, or lots of level-6/level-7 admin 
command attempts?


---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Kraig Beahn]

How much of a traffic increase?

On Dec 15, 2016 5:46 PM, "Jose Gerardo Perales Soto" <
gerardo.pera...@axtel.com.mx> wrote:

> Hi,
>
> We've recently experienced a traffic increase on the NTP queries to NTP
> pool project (pool.ntp.org) servers. One theory is that some service
> provider NTP infraestructure failed approximately 2 days ago and traffic is
> now being redirected to servers belonging to the NTP pool project.
>
> Does anyone from the service provider community have any comments?
>
> Gerardo Perales
>
>
> 
>
> NOTA: La información de este correo es de propiedad exclusiva y
> confidencial. Este mensaje es sólo para el destinatario señalado, si usted
> no lo es, destrúyalo de inmediato. Ninguna información aquí contenida debe
> ser entendida como dada o avalada por AXTEL, S.A.B. de C.V, sus
> subsidiarias o sus empleados, salvo cuando ello expresamente se indique. Es
> responsabilidad de quien recibe este correo de asegurarse que esté libre de
> virus, por lo tanto ni AXTEL, S.A.B. de C.V, sus subsidiarias ni sus
> empleados aceptan responsabilidad alguna.
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by AXTEL, S.A.B. de
> C.V, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither AXTEL, S.A.B. de C.V, its subsidiaries nor their
> employees accept any responsibility.
>

Re: ChangeIP.com has been down for 20+ hours

[Javier J]

Anyone have a contact there?


They probably could have used a hot standby of their DB.

On Wed, Dec 14, 2016 at 9:24 PM, Jay Farrell via NANOG 
wrote:

> See their twitter: https://twitter.com/changeipcom
>
> ChangeIP.com ‏@ChangeIPcom  Dec 13
>
> DNS Service functions restored, website, dynamic dns, and control panel
> functions remain offline as we continue DB restore process.
>
> On Mon, Dec 12, 2016 at 11:35 AM, Brian J. Dent 
> wrote:
>
> >
>

Re: Recent NTP pool traffic increase

[joel jaeggli]

On 12/15/16 3:07 PM, Dan Drown wrote:
> Quoting Jose Gerardo Perales Soto :
>> We've recently experienced a traffic increase on the NTP queries to
>> NTP pool project (pool.ntp.org) servers. One theory is that some
>> service provider NTP infraestructure failed approximately 2 days ago
>> and traffic is now being redirected to servers belonging to the NTP
>> pool project.
>>
>> Does anyone from the service provider community have any comments?
>
> To add some more numbers to this, I'm seeing 4x the usual NTP traffic
> to my server in pool.ntp.org, starting Dec 13.
>
> Top source ASNs by % of NTP traffic seen by my server (I don't have
> pre-Dec 13 traffic by ASN handy)
>
> sprint 4.0%
> verizon-wireless 3.4%
> tmobile 2.9%
> att-wireless 2.8%
> comcast 2.1%
> orange 1.8%
> sky 1.6%
> twc 1.0%
> att 1.0%
> swisscom 0.9%
> saudinet 0.8%
> virgin 0.6%
> opaltelecom 0.5%
> qwest 0.5%
> eli 0.2%
> verizon 0.2%
>
> Possibly related is the new iOS release.  Does the new iOS generate
> more NTP traffic?  Can anyone measure that?

IOS uses time.appple.com which is widely available.





signature.asc
Description: OpenPGP digital signature

Re: Recent NTP pool traffic increase

[Dan Drown]

Quoting Jose Gerardo Perales Soto :
We've recently experienced a traffic increase on the NTP queries to  
NTP pool project (pool.ntp.org) servers. One theory is that some  
service provider NTP infraestructure failed approximately 2 days ago  
and traffic is now being redirected to servers belonging to the NTP  
pool project.


Does anyone from the service provider community have any comments?


To add some more numbers to this, I'm seeing 4x the usual NTP traffic  
to my server in pool.ntp.org, starting Dec 13.


Top source ASNs by % of NTP traffic seen by my server (I don't have  
pre-Dec 13 traffic by ASN handy)


sprint 4.0%
verizon-wireless 3.4%
tmobile 2.9%
att-wireless 2.8%
comcast 2.1%
orange 1.8%
sky 1.6%
twc 1.0%
att 1.0%
swisscom 0.9%
saudinet 0.8%
virgin 0.6%
opaltelecom 0.5%
qwest 0.5%
eli 0.2%
verizon 0.2%

Possibly related is the new iOS release.  Does the new iOS generate  
more NTP traffic?  Can anyone measure that?

Re: Recent NTP pool traffic increase

[Blake Hudson]

I would think if a service provider failed, the stats would bear that 
out. For example, if one of the top ISPs in the world was forwarding 
requests, then you would likely see an increase in the number of queries 
generated from IP addresses registered to that organization. A similar 
effect could occur if a large ISP recently started distributing NTP 
servers as part of their DHCP options when they had not previously. If 
historical query data is not available, the current data could be used 
to make an educated guess and follow up on the likely data trails as 
currently visible.


I would also not rule out the possibility that a Netgear, DLink, 
T-mobile or some other vendor or distributor of access gear pushed out a 
firmware update which enabled NTP when it previously was disabled or 
otherwise changed a device's NTP settings or behavior.


--Blake

Jose Gerardo Perales Soto wrote on 12/15/2016 4:45 PM:

Hi,

We've recently experienced a traffic increase on the NTP queries to NTP pool 
project (pool.ntp.org) servers. One theory is that some service provider NTP 
infraestructure failed approximately 2 days ago and traffic is now being 
redirected to servers belonging to the NTP pool project.

Does anyone from the service provider community have any comments?

Gerardo Perales

Recent NTP pool traffic increase

[Jose Gerardo Perales Soto]

Hi,

We've recently experienced a traffic increase on the NTP queries to NTP pool 
project (pool.ntp.org) servers. One theory is that some service provider NTP 
infraestructure failed approximately 2 days ago and traffic is now being 
redirected to servers belonging to the NTP pool project.

Does anyone from the service provider community have any comments?

Gerardo Perales




NOTA: La información de este correo es de propiedad exclusiva y confidencial. 
Este mensaje es sólo para el destinatario señalado, si usted no lo es, 
destrúyalo de inmediato. Ninguna información aquí contenida debe ser entendida 
como dada o avalada por AXTEL, S.A.B. de C.V, sus subsidiarias o sus empleados, 
salvo cuando ello expresamente se indique. Es responsabilidad de quien recibe 
este correo de asegurarse que esté libre de virus, por lo tanto ni AXTEL, 
S.A.B. de C.V, sus subsidiarias ni sus empleados aceptan responsabilidad alguna.
NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by AXTEL, S.A.B. de C.V, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither AXTEL, S.A.B. de C.V, its subsidiaries nor their employees 
accept any responsibility.

Re: Rogers Peering Request

[jim deleskie]

Will reach out to some folks I know there. PM me Network, AS etc.

On Thu, Dec 15, 2016 at 3:33 PM, Ryan Gard  wrote:

> Looking for a Rogers contact to get things moving on a peering request.
> Been trying to shout into their ear for well over a month, and haven't
> heard anything back. Further, PeeringDB information seems egregiously
> outdated as the URLs listed no longer are serviceable.
>
> Hoping this is the last ditch effort to wake somebody up in the red tower.
>
> Thanks!
>
> --
> Ryan Gard
>

Re: Cogent NOC

[Randy]

Hi All,

Final update from Cogent -- glad they have finally acknowledged -- but 
no ETA, just great:


After further investigation, we have identified an issue of congestion 
on our core device. At this time we are scheduling a maintenance to 
alleviate the congestion which in turn will fix the packetloss seen 
across the Ashburn area.


The maintenance has not yet been scheduled but we will inform you once 
we have a set date.


---
~Randy

Rogers Peering Request

[Ryan Gard]

Looking for a Rogers contact to get things moving on a peering request.
Been trying to shout into their ear for well over a month, and haven't
heard anything back. Further, PeeringDB information seems egregiously
outdated as the URLs listed no longer are serviceable.

Hoping this is the last ditch effort to wake somebody up in the red tower.

Thanks!

-- 
Ryan Gard

Re: BCP38 and Red Hat

[Christopher Morrow]

On Thu, Dec 15, 2016 at 9:48 AM, Stephen Satchell  wrote:

> https://bugzilla.redhat.com/show_bug.cgi?id=1370963
>
> Just a reminder that I have a feature request outstanding with Red Hat
> to add support for BCP38, as well as measures for certain protocol-based
> amplification reflection attacks.  My intent for making the suggestion
> is to stiffen firewalld(8) in Red Hat Enterprise and clones,
> particularly when an RHEL-based box is used as an edge router or
> firewall box.
>
> I've looked at firewalld, and it would be easy to add *some* of BCP38
> into it rather quickly...assuming that the developers step up to the
> plate.  There are parts of BCP38 that won't be so easy to do, given the
> architecture of the package.
>
> In my spare time, by the way, I'm working on a BCP-compilant firewall
> generator for IPTABLES.  Spare time?  Well, that *is* a bit of a laugh...
>

Given some quick time with definition making:
  https://github.com/google/capirca

does this pretty easily, for example:
def/NETWORK.net - content:
  MYNETS = 192.0.24.0/24
  MYWEB = 192.0.24.2/32
  STEPHEN_HOME = 198.16.0.23/32

def/SERVICES.svc - content:
  HTTP = tcp/80
  HTTPS = tcp/443
  SQUID = tcp/3128
  APACHE_PROXY = tcp/8080
  PROXY = SQUID APACHE_PROXY

office/pol/fw.pol - content
  header {
comment:: "My firewall policy"
target:: iptables OUTPUT DROP nostate
  }
  term permit-web-stephen {
comment:: "Permit stephen to my web, really FROM my web to stephen"
destination-address:: STEPHEN_HOME
source-address:: MYWEB
protocol:: tcp
destination-port:: HTTP HTTPS PROXY
action:: permit
  }
  term bcp-38-only {
comment:: "Permit only mynets outbound"
source-address:: MYNETS
action:: accept
  }
  term default-deny {
comment:: "All other traffic dies"
action:: deny
  }


run the acl generation (aclgen.py) and ... out pops iptables to do what you
want.
a simple matter of script/software makes this even simple for iptables
operators across many flavors of topology.

-chris
(note: I am not just a user of this solution I'm also a contributor)

BCP38 and Red Hat

[Stephen Satchell]

https://bugzilla.redhat.com/show_bug.cgi?id=1370963

Just a reminder that I have a feature request outstanding with Red Hat
to add support for BCP38, as well as measures for certain protocol-based
amplification reflection attacks.  My intent for making the suggestion
is to stiffen firewalld(8) in Red Hat Enterprise and clones,
particularly when an RHEL-based box is used as an edge router or
firewall box.

I've looked at firewalld, and it would be easy to add *some* of BCP38
into it rather quickly...assuming that the developers step up to the
plate.  There are parts of BCP38 that won't be so easy to do, given the
architecture of the package.

In my spare time, by the way, I'm working on a BCP-compilant firewall
generator for IPTABLES.  Spare time?  Well, that *is* a bit of a laugh...